User-space probes: Plan B+

[Jim Keniston]

Here's where we stand on user-space probes (uprobes).  The intent of
uprobes is to enable application developers to create low-overhead,
dynamic instrumentation for their apps, with uprobes-based
instrumentation interoperating usefully, as needed, with kprobes-based
instrumentation.  Comments are welcome.

Recent History
--------------
Last spring, Prasanna Panchamukhi offered up a kernel-only approach,
where instrumentation would be coded as a kernel module, a la kprobes.
This performed well (e.g. 1 usec per probepoint hit on my Pentium M),
but we got bad reviews on such things as the kernel-only approach
and the per-executable tracing (e.g., hooking read_page(s)).

I tried an approach based on ptrace, with no kernel enhancements, but
it lacked certain necessary features (e.g., #2-5 below), probe overhead
was 12-15x worse than Prasanna's approach, and I couldn't get it to
work when probing multiple processes.  (Frank Eigler independently
suggested this approach and termed it "Plan B from outer space.")

While I was stumped trying to make Plan B work, Roland McGrath made
utrace available to us.  We looked this over as we found the time,
and it looked promising.

There has been much debate within the kprobes teams about the proper
programming model to support.  Discussions at OLS didn't yield many
new ideas, let alone consensus.

The Current Approach: Overview
------------------------------
The approach we are now coding can be summarized as follows.  (Okay,
it's not much like Plan B, but B+ sounds better than C.)

a. A system-call API that is an alternative to ptrace, provides
better support for probepoints and return probes, and exploits all
the process-lifetime events made accessible by utrace.

b. The "tracer" process detects events (e.g., probe hits) by polling
rather than catching SIGCHLD signals.

c. Hooks to allow kernel-mode instrumentation to cooperate with
user-mode "tracer" processes.

Here are the requirements we will satisfy with this approach.

0. Per-process (not per-executable) tracing.

1. Instrumentation can be coded entirely as a user-space app...

2. ... but in situations where performance is critical, uprobes can
run a named kernel handler without waking up the tracer process.

3. A user-mode tracer can invoke a previously registered kernel-mode
handler, so we have simple and efficient communication between user-
and kernel-mode instrumentation.

4. Multiple tracer processes can trace the same tracee.

5. As needed, we can "pre-define" a set of useful kernel handlers.

6. Uprobes can be easily extended (exploiting utrace) to support
notifying the tracer of non-probepoint events in the probee,
such as signals and system calls.

7. The user API should be easier to use than the ptrace API.

8. Handlers run in process context -- the tracee's context (see
requirement 2) or the tracer's context while the tracee is stopped
(see requirement 3).

A typical tracer app would do the following:

- Call uprobe_register() to establish a probepoint and be notified
(or run a kernel handler) when the probepoint is hit.

- Call uprobe_poll() repeatedly to poll for, and handle, events.
(A tracing app would have to spawn multiple threads to trace
multiple processes.)

- Whenever appropriate, call uprobe_run_khandler() to interoperate
with kernel-side instrumentation.

- Call uprobe_unregister() to cancel uprobes.

Apart from implementing kernel-side support for uprobes, the only
addition to the kernel API is a register_khandler() function that takes
a name, handler, and access-permission info.  (The handler takes,
as optional args, pointers to a uprobe object and an arbitrary,
user-defined data area.)

User API
--------
A summary of the user-side API is attached.

Jim


-------------- next part --------------
Terminology
-----------
tracee = the task being probed/traced
tracer = the task that probes/traces the tracee

Each tracer can trace at most one tracee.  To monitor multiple tracees
concurrently, spawn multiple tracer threads.

Multiple tracers can trace the same tracee.

User API Data Structures
------------------------
enum uprobe_event {
	UPEV_BKPT,	// probepoint
	UPEV_FUNCRET,	// return probe
	UPEV_EXIT,
	/*
	 * TODO: Add other utrace-supported events:
	 * CLONE, VFORK_DONE, EXEC, DEATH, SYSCALL_ENTRY, SYSCALL_EXIT,
	 * SIGNAL, SIGNAL_*.
	 */
};

struct uprobe {
	unsigned long		id;	// user-specified ID
	enum uprobe_event	event;	// event type
	pid_t			who;	// pid
	unsigned long		where;	// virtual address for UPEV_BKPT
					// or UPEV_FUNCRET
	const char		*khandler;	// If non-null, run khandler
					// without notifying tracer.
};

User API Overview
-----------------
int uprobe_register(const struct uprobe *u);
Register interest in a specified type of event.  This is how probepoints
are registered.

int uprobe_unregister(pid_d pid, unsigned long id);
Unregister interest in a specified probepoint or other event, or all
events for specified tracee.

int uprobe_poll(struct uprobe *u, int timeout);
Block until any of the previously registered events happens in the tracee
(u->who).

int uprobe_detach(pid_t pid);
Unregister interest in all events associated with the specifed tracee.

int uprobe_run_khandler(const char *khandler, struct uprobe *u, void *data);
Run a kernel-mode handler, possibly exchanging data between user and
kernel space.

API Specs
---------
int uprobe_register(const struct uprobe *u);
--------------------------------------------
The tracer process registers an interest in the specified event.
If event == UPEV_BKPT, we set a probepoint for process who at virtual
address where.  Caller sets id (which may be a pointer or integer)
for future reference.

When the specified event happens, uprobes does exactly one of the
following:

a) If khandler is specified, uprobes calls that kernel handler,
passing it a pointer to the kernel equivalent of the uprobe object.
uprobes then continues execution of process who (e.g., in the case
of a breakpoint, doing the necessary single-step and such).

khandler must have been previously registered by a call to the (new)
kernel function register_khandler(), described elsewhere (soon).

b) If khandler is null, and the tracer process is running uprobe_poll()
for process who, uprobe_poll() returns.

c) Otherwise, uprobes just continues execution of process who.
Note that uprobe_poll() will not catch events previously specified
by uprobe_register() that happen before the uprobe_poll() call.

If there are multiple tracer processes for the same tracee, their
ID "namespaces" (i.e., numberspaces) are considered distinct.

int uprobe_unregister(pid_t pid, unsigned long id);
----------------------------------------------
Unregisters the tracer's interest in the event with the specified
id on the specified tracee.  We need to support the tracer calling
uprobe_unregister() for the event he's currently handling.

uprobe_unregister(pid, UPID_ALL) unregisters all of this tracer's
uprobes for the specified tracee.

int uprobe_poll(struct uprobe *u, int timeout);
-----------------------------------------------
If process u->who is stopped, allow it to continue.* Then sleep until
an event of interest happens in process u->who, or until timeout ms
have passed, whichever comes first.  On timeout, return 0.  If an
event happens, uprobes returns 1 and fills in the uprobe object with
the values that were specified when that uprobe was registered.

*If a tracer registers n uprobes for the same tracee and address,
he'll have to call uprobe_poll() n times to be notified of all
of them.  The tracee won't continue until all n notifications have
been delivered (or cancelled by uprobe_unregister() or uprobe_detach()).


int uprobe_run_khandler(const char *khandler, struct uprobe *u, void *data);
----------------------------------------------------------------------------
Run the kernel-mode handler whose name is khandler, passing it the
pointers u and data.  The named khandler must have been previously
registered by a call to the (new) kernel function register_khandler()
(see above).  The contents (if any) of *u and *data before the call
are up to the caller; their contents on return are up to the khandler.

It's up to the tracer and the khandler to guarantee that the process,
if any, specified by u->who is in a state where the khandler can
act on it.  Typically, the tracer will call uprobe_run_khandler() after
uprobe_poll() returns, indicating that an event of interest has
occurred.