After writing enough analysis scripts for yourself, you may become known as an expert to your colleagues, who will want to use your scripts. Systemtap makes it possible to share in a controlled manner; to build libraries of scripts that build on each other. In fact, all of the functions (pid(), etc.) used in the scripts above come from tapset scripts like that. A “tapset” is just a script that designed for reuse by installation into a special directory.
Systemtap attempts to resolve references to global symbols (probes, functions, variables) that are not defined within the script by a systematic search through the tapset library for scripts that define those symbols. Tapset scripts are installed under the default directory named /usr/share/systemtap/tapset. A user may give additional directories with the -I DIR option. Systemtap searches these directories for script (.stp) files.
The search process includes subdirectories that are specialized for a particular kernel version and/or architecture, and ones that name only larger kernel families. Naturally, the search is ordered from specific to general, as shown in Figure 7.
# stap -p1 -vv -e ’probe begin { }’ > /dev/null
Created temporary directory "/tmp/staplnEBh7"
Searched ’/usr/share/systemtap/tapset/2.6.15/i686/*.stp’, match count 0
Searched ’/usr/share/systemtap/tapset/2.6.15/*.stp’, match count 0
Searched ’/usr/share/systemtap/tapset/2.6/i686/*.stp’, match count 0
Searched ’/usr/share/systemtap/tapset/2.6/*.stp’, match count 0
Searched ’/usr/share/systemtap/tapset/i686/*.stp’, match count 1
Searched ’/usr/share/systemtap/tapset/*.stp’, match count 12
Pass 1: parsed user script and 13 library script(s) in 350usr/10sys/375real ms.
Running rm -rf /tmp/staplnEBh7
When a script file is found that defines one of the undefined symbols, that entire file is added to the probing session being analyzed. This search is repeated until no more references can become satisfied. Systemtap signals an error if any are still unresolved.
This mechanism enables several programming idioms. First, it allows some global symbols to be defined only for applicable kernel version/architecture pairs, and cause an error if their use is attempted on an inapplicable host. Similarly, the same symbol can be defined differently depending on kernels, in much the same way that different kernel include/asm/ARCH/ files contain macros that provide a porting layer.
Another use is to separate the default parameters of a tapset routine from its implementation. For example, consider a tapset that defines code for relating elapsed time intervals to process scheduling activities. The data collection code can be generic with respect to which time unit (jiffies, wall-clock seconds, cycle counts) it can use. It should have a default, but should not require additional run-time checks to let a user choose another. Figure 8 shows a way.
# cat tapset/time-common.stp
global __time_vars
function timer_begin (name) { __time_vars[name] = __time_value () }
function timer_end (name) { return __time_value() - __time_vars[name] }
# cat tapset/time-default.stp
function __time_value () { return gettimeofday_us () }
# cat tapset-time-user.stp
probe begin
{
timer_begin ("bench")
for (i=0; i<100; i++) ;
printf ("%d cycles\n", timer_end ("bench"))
exit ()
}
function __time_value () { return get_ticks () } # override for greater precision
A tapset that exports only data may be as useful as ones that exports functions or probe point aliases (see below). Such global data can be computed and kept up-to-date using probes internal to the tapset. Any outside reference to the global variable would incidentally activate all the required probes.
Probe point aliases allow creation of new probe points from existing ones. This is useful if the new probe points are named to provide a higher level of abstraction. For example, the system-calls tapset defines probe point aliases of the form syscall.open etc., in terms of lower level ones like kernel.function("sys_open"). Even if some future kernel renames sys_open, the aliased name can remain valid.
A probe point alias definition looks like a normal probe. Both start with the keyword probe and have a probe handler statement block at the end. But where a normal probe just lists its probe points, an alias creates a new name using the assignment (=) operator. Another probe that names the new probe point will create an actual probe, with the handler of the alias prepended.
This prepending behavior serves several purposes. It allows the alias definition to “preprocess” the context of the probe before passing control to the user-specified handler. This has several possible uses:
| if ($flag1 != $flag2) next | skip probe unless given condition is met |
| name = "foo" | supply probe-describing values |
| var = $var | extract target variable to plain local variable |
Figure 9 demonstrates a probe point alias definition as well as its use. It demonstrates how a single probe point alias can expand to multiple probe points, even to other aliases. It also includes probe point wildcarding. These functions are designed to compose sensibly.
# cat probe-alias.stp
probe syscallgroup.io = syscall.open, syscall.close,
syscall.read, syscall.write
{ groupname = "io" }
probe syscallgroup.process = syscall.fork, syscall.execve
{ groupname = "process" }
probe syscallgroup.*
{ groups [execname() . "/" . groupname] ++ }
probe end
{
foreach (eg+ in groups)
printf ("%s: %d\n", eg, groups[eg])
}
global groups
# stap probe-alias.stp
05-wait_for_sys/io: 19
10-udev.hotplug/io: 17
20-hal.hotplug/io: 12
X/io: 73
apcsmart/io: 59
[...]
make/io: 515
make/process: 16
[...]
xfce-mcs-manage/io: 3
xfdesktop/io: 5
[...]
xmms/io: 7070
zsh/io: 78
zsh/process: 5
Sometimes, a tapset needs provide data values from the kernel that cannot be extracted using ordinary target variables ($var). This may be because the values are in complicated data structures, may require lock awareness, or are defined by layers of macros. Systemtap provides an “escape hatch” to go beyond what the language can safely offer. In certain contexts, you may embed plain raw C in tapsets, exchanging power for the safety guarantees listed in section 3.6. End-user scripts may not include embedded C code, unless systemtap is run with the -g (“guru” mode) option. Tapset scripts get guru mode privileges automatically.
Embedded C can be the body of a script function. Instead enclosing the function body statements in { and }, use %{ and %}. Any enclosed C code is literally transcribed into the kernel module: it is up to you to make it safe and correct. In order to take parameters and return a value, macros STAP_ARG_* and STAP_RETVALUE are made available. The familiar data-gathering functions pid(), execname(), and their neighbours are all embedded C functions. Figure 10 contains another example.
Since systemtap cannot examine the C code to infer these types, an optional5 annotation syntax is available to assist the type inference process. Simply suffix parameter names and/or the function name with :string or :long to designate the string or numeric type. In addition, the script may include a %{ %} block at the outermost level of the script, in order to transcribe declarative code like #include <linux/foo.h>. These enable the embedded C functions to refer to general kernel types.
There are a number of safety-related constraints that should be observed by developers of embedded C code.
Do not dereference pointers that are not known or testable valid.
Do not call any kernel routine that may cause a sleep or fault.
Consider possible undesirable recursion, where your embedded C function calls a routine that may be the subject of a probe. If that probe handler calls your embedded C function, you may suffer infinite regress. Similar problems may arise with respect to non-reentrant locks.
If locking of a data structure is necessary, use a trylock type call to attempt to take the lock. If that fails, give up, do not block.
# cat embedded-C.stp
%{
#include <linux/sched.h>
#include <linux/list.h>
%}
function task_execname_by_pid:string (pid:long) %{
struct task_struct *p;
struct list_head *_p, *_n;
list_for_each_safe(_p, _n, ¤t->tasks) {
p = list_entry(_p, struct task_struct, tasks);
if (p->pid == (int)STAP_ARG_pid)
snprintf(STAP_RETVALUE, MAXSTRINGLEN, "%s", p->comm);
}
%}
probe begin
{
printf("%s(%d)\n", task_execname_by_pid(target()), target())
exit()
}
# pgrep emacs
16641
# stap -g embedded-C.stp -x 16641
emacs(16641)
Using the tapset search mechanism just described, potentially many script files can become selected for inclusion in a single session. This raises the problem of name collisions, where different tapsets accidentally use the same names for functions/globals. This can result in errors at translate or run time.
To control this problem, systemtap tapset developers are advised to follow naming conventions. Here is some of the guidance.
Pick a unique name for your tapset, and substitute it for TAPSET below.
Separate identifiers meant to be used by tapset users from those that are internal implementation artifacts.
Document the first set in the appropriate man pages.
Prefix the names of external identifiers with TAPSET_ if there is any likelihood of collision with other tapsets or end-user scripts.
Prefix any probe point aliases with an appropriate prefix.
Prefix the names of internal identifiers with __TAPSET_.
Write a tapset that implements deferred and “cancelable” logging. Export a function that enqueues a text string (into some private array), returning an id token. Include a timer-based probe that periodically flushes the array to the standard log output. Export another function that, if the entry was not already flushed, allows a text string to be cancelled from the queue. One might speculate that similar functions and tapsets exist.
Create a “relative timestamp” tapset with functions return all the same values as the ones in the timestamp tapset, except that they are made relative to the start time of the script.
Create a tapset that exports a global array that contains a mapping of recently seen process ID numbers to process names. Intercept key system calls (execve?) to update the list incrementally.
Send your tapset ideas to the mailing list!