3 - The systemtap client and compile server are now available.
4 These allow you to compile a systemtap module on a host other than
5 the one which it will be run, providing the client and server
6 are compatible. Other than using a server for passes 1 through
7 4, the client behaves like the 'stap' front end itself. This
8 means, among other things, that the client will automatically
9 load the resulting module on the local host unless -p[1234]
12 The client/server now use SSL for network connection security and
15 See stap-server(8) for more details.
17 - The stap "-r DIR" option may be used to identify a hand-made kernel
18 build directory. The tool determines the appropriate release string
19 automatically from the directory.
21 - Serious problems associated with user-space probing in shared libraries
22 was corrected, making it now possible to experiment with probe shared
23 libraries. Assuming dwarf debugging information is installed, use this
24 twist on the normal syntax:
26 probe process("/lib64/libc-2.8.so").function("....") { ... }
28 This would probe all threads that call into that library. Running
29 "stap -c CMD" or "stap -x PID" naturally restricts this to the target
30 command+descendants only. $$vars etc. may be used.
32 - For scripts that sometimes terminate with excessive "skipped" probes,
33 rerunning the script with "-t" (timing) will print more details about
36 - Symbol tables and unwind (backtracing) data support were formerly
37 compiled in for all probed modules as identified by the script
38 (kernel; module("name"); process("file")) plus those listed by the
39 stap "-d BINARY" option. Now, this data is included only if the systemtap
40 script uses tapset functions like probefunc() or backtrace() that require
41 such information. This shrinks the probe modules considerably for the rest.
43 - Per-pass verbosity control is available with the new "--vp {N}+" option.
44 "stap --vp 040" adds 4 units of -v verbosity only to pass 2. This is useful
45 for diagnosing errors from one pass without excessive verbosity from others.
47 - Most probe handlers now run with interrupts enabled, for improved
48 system responsiveness and less probing overhead. This may result
49 in more skipped probes, for example if a reentrant probe handler
50 is attempted from within an interrupt handler. It may also make the
51 systemtap overload detection facility more likely to be triggered, as
52 interrupt handlers' run time would be included in the self-assessed
53 overhead of running probe handlers.
55 * What's new in version 0.8
57 - Cache limiting is now available. If the compiled module cache size is
58 over a limit specified in the $SYSTEMTAP_DIR/cache/cache_mb_limit file,
59 some old cache entries will be unlinked. See man stap(1) for more.
61 - Error and warning messages are now followed by source context displaying
62 the erroneous line/s and a handy '^' in the following line pointing to the
65 - A bug reporting tool "stap-report" is now available which will quickly
66 retrieve much of the information requested here:
67 http://sourceware.org/systemtap/wiki/HowToReportBugs
69 - The translator can resolve members of anonymous structs / unions:
70 given struct { int foo; struct { int bar; }; } *p;
71 this now works: $p->bar
73 - The stap "-F" flag activates "flight recorder" mode, which consists of
74 translating the given script as usual, but implicitly launching it into
75 the background with staprun's existing "-L" (launch) option. A user
76 can later reattach to the module with "staprun -A MODULENAME".
78 - Additional context variables are available on user-space syscall probes.
79 - $argN ($arg1, $arg2, ... $arg6) in process(PATH_OR_PID).syscall
80 gives you the argument of the system call.
81 - $return in process(PATH_OR_PID).syscall.return gives you the return
82 value of the system call.
84 - Target process mode (stap -c CMD or -x PID) now implicitly restricts all
85 "process.*" probes to the given child process. (It does not affect
86 kernel.* or other probe types.) The CMD string is normally run directly,
87 rather than via a /bin/sh -c subshell, since then utrace/uprobe probes
88 receive a fairly "clean" event stream. If metacharacters like
89 redirection operators were present in CMD, then "sh -c CMD" is still
90 used, and utrace/uprobe probes will receive events from the shell.
92 % stap -e 'probe process.syscall, process.end {
93 printf("%s %d %s\n", execname(), pid(), pp())}'\
95 ls 2323 process.syscall
96 ls 2323 process.syscall
99 - Probe listing mode is improved: "-L" lists available script-level variables
101 % stap -L 'syscall.*open*'
102 syscall.mq_open name:string name_uaddr:long filename:string mode:long u_attr_uaddr:long oflag:long argstr:string
103 syscall.open name:string filename:string flags:long mode:long argstr:string
104 syscall.openat name:string filename:string flags:long mode:long argstr:string
106 - All user-space-related probes support $PATH-resolved executable
109 probe process("ls").syscall {}
110 probe process("./a.out").syscall {}
112 work now, instead of just
114 probe process("/bin/ls").syscall {}
115 probe process("/my/directory/a.out").syscall {}
117 - Prototype symbolic user-space probing support:
119 # stap -e 'probe process("ls").function("*").call {
120 log (probefunc()." ".$$parms)
125 - debugging information for the named program
126 - a version of utrace in the kernel that is compatible with the "uprobes"
127 kernel module prototype. This includes RHEL5 and older Fedora, but not
128 yet current lkml-track utrace; a "pass 4a"-time build failure means
129 your system cannot use this yet.
131 - Global variables which are written to but never read are now
132 automatically displayed when the session does a shutdown. For example:
135 probe timer.profile {running_tasks[pid(),tid()] = execname()}
136 probe timer.ms(8000) {exit()}
138 - A formatted string representation of the variables, parameters, or local
139 variables at a probe point is now supported via the special $$vars,
140 $$parms, and $$locals context variables, which expand to a string
141 containing a list "var1=0xdead var2=0xbeef var3=?". (Here, var3 exists
142 but is for some reason unavailable.) In return probes only, $$return
143 expands to an empty string for a void function, or "return=0xf00".
146 * What's new in version 0.7
148 - .statement("func@file:*") and .statement("func@file:M-N") probes are now
149 supported to allow matching a range of lines in a function. This allows
150 tracing the execution of a function.
152 - Scripts relying on probe point wildcards like "syscall.*" that expand
153 to distinct kprobes are processed significantly faster than before.
155 - The vector of script command line arguments is available in a
156 tapset-provided global array argv[]. It is indexed 1 ... argc,
157 another global. This can substitute for of preprocessor
158 directives @NNN that fail at parse time if there are not
161 printf("argv: %s %s %s", argv[1], argv[2], argv[3])
163 - .statement("func@file+line") probes are now supported to allow a
164 match relative to the entry of the function incremented by line
165 number. This allows using the same systemtap script if the rest
166 of the file.c source only changes slightly.
168 - A probe listing mode is available.
177 - More user-space probe types are added:
179 probe process(PID).begin { }
180 probe process("PATH").begin { }
181 probe process(PID).thread.begin { }
182 probe process("PATH").thread.begin { }
183 probe process(PID).end { }
184 probe process("PATH").end { }
185 probe process(PID).thread.end { }
186 probe process("PATH").thread.end { }
187 probe process(PID).syscall { }
188 probe process("PATH").syscall { }
189 probe process(PID).syscall.return { }
190 probe process("PATH").syscall.return { }
192 - Globals now accept ; terminators
195 global little[10], big[5];
197 * What's new in version 0.6
199 - A copy of the systemtap tutorial and language reference guide
202 - There is a new format specifier, %m, for the printf family of
203 functions. It functions like %s, except that it does not stop when
204 a nul ('\0') byte is encountered. The number of bytes output is
205 determined by the precision specifier. The default precision is 1.
208 printf ("%m", "My String") // prints one character: M
209 printf ("%.5", myString) // prints 5 bytes beginning at the start
212 - The %b format specifier for the printf family of functions has been enhanced
215 1) When the width and precision are both unspecified, the default is %8.8b.
216 2) When only one of the width or precision is specified, the other defaults
217 to the same value. For example, %4b == %.4b == %4.4b
218 3) Nul ('\0') bytes are used for field width padding. For example,
220 printf ("%b", 0x1111deadbeef2222) // prints all eight bytes
221 printf ("%4.2b", 0xdeadbeef) // prints \0\0\xbe\xef
223 - Dynamic width and precision are now supported for all printf family format
224 specifiers. For example:
228 printf ("%*.*b", four, two, 0xdeadbbeef) // prints \0\0\xbe\xef
229 printf ("%*d", four, two) // prints <space><space><space>2
231 - Preprocessor conditional expressions can now include wildcard style
232 matches on kernel versions.
233 %( kernel_vr != "*xen" %? foo %: bar %)
235 - Prototype support for user-space probing is showing some progress.
236 No symbolic notations are supported yet (so no probing by function names,
237 file names, process names, and no access to $context variables), but at
238 least it's something:
240 probe process(PID).statement(ADDRESS).absolute { }
242 This will set a uprobe on the given process-id and given virtual address.
243 The proble handler runs in kernel-space as usual, and can generally use
244 existing tapset functions.
246 - Crash utility can retrieve systemtap's relay buffer from a kernel dump
247 image by using staplog which is a crash extension module. To use this
248 feature, type commands as below from crash(8)'s command line:
250 crash> extend staplog.so
251 crash> help systemtaplog
253 Then, you can see more precise help message.
255 - You can share a relay buffer amoung several scripts and merge outputs from
256 several scripts by using "-DRELAY_HOST" and "-DRELAY_GUEST" options.
260 % stap -ve 'probe begin{}' -o merged.out -DRELAY_HOST &
261 # wait until starting the host.
262 % stap -ve 'probe begin{print("hello ");exit()}' -DRELAY_GUEST
263 % stap -ve 'probe begin{print("world\n");exit()}' -DRELAY_GUEST
265 Then, you'll see "hello world" in merged.out.
267 - You can add a conditional statement for each probe point or aliase, which
268 is evaluated when the probe point is hit. If the condition is false, the
269 whole probe body(including aliases) is skipped. For example:
272 probe syscall.* if (switch) { ... }
273 probe procfs.write {switch = strtol($value,10)} /* enable/disable ctrl */
275 - Systemtap will warn you if your script contains unused variables or
276 functions. This is helpful in case of misspelled variables. If it
277 doth protest too much, turn it off with "stap -w ...".
279 - You can add error-handling probes to a script, which are run if a
280 script was stopped due to errors. In such a case, "end" probes are
281 not run, but "error" ones are.
283 probe error { println ("oops, errors encountered; here's a report anyway")
284 foreach (coin in mint) { println (coin) } }
286 - In a related twist, one may list probe points in order of preference,
287 and mark any of them as "sufficient" beyond just "optional". Probe
288 point sequence expansion stops if a sufficient-marked probe point has a hit.
289 This is useful for probes on functions that may be in a module (CONFIG_FOO=m)
290 or may have been compiled into the kernel (CONFIG_FOO=y), but we don't know
293 probe module("sd").function("sd_init_command") ? ,
294 kernel.function("sd_init_command") ? { ... }
296 which might match neither, now one can write this:
298 probe module("sd").function("sd_init_command") ! , /* <-- note excl. mark */
299 kernel.function("sd_init_command") { ... }
301 - New security model. To install a systemtap kernel module, a user
302 must be one of the following: the root user; a member of the
303 'stapdev' group; or a member of the 'stapusr' group. Members of the
304 stapusr group can only use modules located in the
305 /lib/modules/VERSION/systemtap directory (where VERSION is the
306 output of "uname -r").
308 - .statement("...@file:line") probes now apply heuristics to allow an
309 approximate match for the line number. This works similarly to gdb,
310 where a breakpoint placed on an empty source line is automatically
311 moved to the next statement. A silly bug that made many $target
312 variables inaccessible to .statement() probes was also fixed.
314 - LKET has been retired. Please let us know on <systemtap@sourceware.org>
315 if you have been a user of the tapset/tools, so we can help you find
318 - New families of printing functions println() and printd() have been added.
319 println() is like print() but adds a newline at the end;
320 printd() is like a sequence of print()s, with a specified field delimiter.
322 * What's new since version 0.5.14?
324 - The way in which command line arguments for scripts are substituted has
325 changed. Previously, $1 etc. would interpret the corresponding command
326 line argument as an numeric literal, and @1 as a string literal. Now,
327 the command line arguments are pasted uninterpreted wherever $1 etc.
328 appears at the beginning of a token. @1 is similar, but is quoted as
329 a string. This change does not modify old scripts, but has the effect
330 of permitting substitution of arbitrary token sequences.
332 # This worked before, and still does:
333 % stap -e 'probe timer.s($1) {}' 5
334 # Now this also works:
335 % stap -e 'probe syscall.$1 {log(@1)}' open
336 # This won't crash, just signal a recursion error:
338 # As before, $1... is recognized only at the beginning of a token
339 % stap -e 'probe begin {foo$1=5}'
341 * What's new since version 0.5.13?
343 - The way in which systemtap resolves function/inline probes has changed:
344 .function(...) - now refers to all functions, inlined or not
345 .inline(...) - is deprecated, use instead:
346 .function(...).inline - filters function() to only inlined instances
347 .function(...).call - filters function() to only non-inlined instances
348 .function(...).return - as before, but now pairs best with .function().call
349 .statement() is unchanged.
351 * What's new since version 0.5.12?
353 - When running in -p4 (compile-only) mode, the compiled .ko file name
354 is printed on standard output.
356 - An array element with a null value such as zero or an empty string
357 is now preserved, and will show up in a "foreach" loop or "in" test.
358 To delete such an element, the scripts needs to use an explicit
359 "delete array[idx]" statement rather than something like "array[idx]=0".
361 - The new "-P" option controls whether prologue searching heuristics
362 will be activated for function probes. This was needed to get correct
363 debugging information (dwarf location list) data for $target variables.
364 Modern compilers (gcc 4.1+) tend not to need this heuristic, so it is
365 no longer default. A new configure flag (--enable-prologues) restores
366 it as a default setting, and is appropriate for older compilers (gcc 3.*).
368 - Each systemtap module prints a one-line message to the kernel informational
369 log when it starts. This line identifies the translator version, base
370 address of the probe module, a broken-down memory consumption estimate, and
371 the total number of probes. This is meant as a debugging / auditing aid.
373 - Begin/end probes are run with interrupts enabled (but with
374 preemption disabled). This will allow begin/end probes to be
375 longer, to support generating longer reports.
377 - The numeric forms of kernel.statement() and kernel.function() probe points
378 are now interpreted as relocatable values - treated as relative to the
379 _stext symbol in that kernel binary. Since some modern kernel images
380 are relocated to a different virtual address at startup, such addresses
381 may shift up or down when actually inserted into a running kernel.
383 kernel.statement(0xdeadbeef): validated, interpreted relative to _stext,
384 may map to 0xceadbeef at run time.
386 In order to specify unrelocated addresses, use the new ".absolute"
387 probe point suffix for such numeric addresses. These are only
388 allowed in guru mode, and provide access to no $target variables.
389 They don't use debugging information at all, actually.
391 kernel.statement(0xfeedface).absolute: raw, unvalidated, guru mode only
393 * What's new since version 0.5.10?
395 - Offline processing of debugging information, enabling general
396 cross-compilation of probe scripts to remote hosts, without
397 requiring identical module/memory layout. This slows down
398 compilation/translation somewhat.
400 - Kernel symbol table data is loaded by staprun at startup time
401 rather than compiled into the module.
403 - Support the "limit" keyword for foreach iterations:
404 foreach ([x,y] in ary limit 5) { ... }
405 This implicitly exits after the fifth iteration. It also enables
406 more efficient key/value sorting.
408 - Support the "maxactive" keyword for return probes:
409 probe kernel.function("sdfsdf").maxactive(848) { ... }
410 This allows up to 848 concurrently outstanding entries to
411 the sdfsdf function before one returns. The default maxactive
412 number is smaller, and can result in missed return probes.
414 - Support accessing of saved function arguments from within
415 return probes. These values are saved by a synthesized
416 function-entry probe.
418 - Add substantial version/architecture checking in compiled probes to
419 assert correct installation of debugging information and correct
420 execution on a compatible kernel.
422 - Add probe-time checking for sufficient free stack space when probe
423 handlers are invoked, as a safety improvement.
425 - Add an optional numeric parameter for begin/end probe specifications,
426 to order their execution.
427 probe begin(10) { } /* comes after */ probe begin(-10) {}
429 - Add an optional array size declaration, which is handy for very small
431 global little[5], big[20000]
433 - Include some example scripts along with the documentation.
435 - Change the start-time allocation of probe memory to avoid causing OOM
436 situations, and to abort cleanly if free kernel memory is short.
438 - Automatically use the kernel DWARF unwinder, if present, for stack
441 - Many minor bug fixes, performance, tapset, and error message