Re: [Bug uprobes/11672] utrace_report_syscall_exit crash

[Roland McGrath <roland at redhat dot com>]

>       (__stp_utrace_attach): Loop on -ERESTARTSYS after utrace_barrier.
>       (__stp_utrace_task_finder_target_quiesce): Likewise.

That is a busy-wait loop, because -ERESTARTSYS means "signal is pending"
and that stays true without any blocking until you let it get back to user
mode (or almost, i.e. signal handling).  I don't really understand the
context of where these functions get called.  I'm guessing it is in some
control thread belonging to stapio or something like that.  In that case,
the signal pending is a SIGINT killing stapio or something like that I
suppose.  Is that the case?

This is a relatively "safe" busy-wait.  utrace_barrier will return 0 when
the tracee is all clear, not short-circuit because of signal_pending()
unless it actually has to block.  It's waiting for the tracee on the other
CPU to complete your callback, which should be pretty quick.  But it's
still a busy-wait that suddenly chews CPU in a spurt, and all quite fragile.

> On irc roland said that newer utrace have a new enginer callback
> "release" that is called when the engine (and its handlers) are totally
> done. We could use that instead of trying to track things ourselves
> before unloading the module. 

It's called on the last utrace_engine_put(), intended as what would release
whatever resources ->data points to.  Aside from your own refs, the "last"
engine ref is held by "attachedness", so it goes exactly when UTRACE_DETACH
(or implicit detach on reap) is complete--i.e., always after the very last
report_* callback that might ever be made to that engine.  So it makes
perfect sense as the place that "releases the hold" on stap module shutdown.

> But it isn't available in older utrace implementations.

I don't think anyone really cares about the intermediate utrace versions,
only the recent Fedora kernels (that have the current code), and RHEL5.  In
RHEL5 utrace, the API (and implementation) about this stuff is pretty
drastically different (no utrace_barrier).  I certainly don't really recall
all the racy corner details of that version, but I don't think it offers
any kind of synchronization option you can rely on short of quiescing the
thread before detach.

The only clearly safe alternative I see there is some giant synchronization
like stop_machine_run, off hand.  I'm trying to imagine a changed scenario
where you use module_get/module_put to prevent the module removal until the
detach is safely complete.  But I'm not sure how to do that without some
extra helper module that holds the callback routines so that they can use
module_put without undercutting their own code mapping.  And if you had
that, you probably wouldn't need to keep the script module refs at all in
the first place.  I think the forcible quiesce before detach is probably
the way to go.  RHEL5's utrace_detach has those signal-interruption side
effects anyway.  Then you just have to check for the sudden-death races
that utrace_detach's return value will alert you to.


Thanks,
Roland