3 - The <sys/sdt.h> user-space markers no longer default to an implicit
4 MARKER_NAME_ENABLED() semaphore check for each marker. To check for
5 enabled markers use a .d declaration file, then:
6 if (MARKER_NAME_ENABLED()) MARKER_NAME()
8 - Hyphenated <sys/sdt.h> marker names such as process(...).mark("foo-bar")
9 are now accepted in scripts. They are mapped to the double-underscore
12 - Prototype support for debuginfo-less <sys/sdt.h> user-space markers
13 is included for some architectures (i686 and x86-64 initially).
14 Compile your sdt.h-instrumented program with -DSTAP_SDT_V2 to
15 activate this mode. This implementation also supports preserving
16 the "provider" name associated with a marker:
17 probe process("foo").provider("bar").mark("baz") to match
18 STAP_PROBE<n>(bar, baz <...>)
20 - Embedded-C may be used within expressions as values, when in guru mode:
21 num = %{ LINUX_VERSION_CODE %} // int64_t
22 name = %{ /* string */ THIS_MODULE->name %} // const char*
23 printf ("%s %x\n", name, num)
24 The usual /* pure */, /* unprivileged */, and /* guru */ markers may be used
25 as with embedded-C functions.
27 - By default the systemtap-runtime RPM builds now include a shared
28 library, staplog.so, that allows crash to extract systemtap data from
31 - Iterating with "foreach" can now explicitly save the value for the loop.
32 foreach(v = [i,j] in array)
33 printf("array[%d,%s] = %d\n", i, j, v /* array[i,j] */)
35 - The new "--ldd" option automatically adds any additional shared
36 libraries needed by probed or -d-listed userspace binaries to the -d
37 list, to enable symbolic backtracing through them. Similarly, the
38 new "--all-modules" option automatically adds any currently loaded
39 kernel modules (listed in /proc/modules) to the -d list.
41 - A new family of set_kernel_* functions make it easier for gurus to write
42 new values at arbitrary memory addresses.
44 - Probe wildcards can now use '**' to cross the '.' separator.
49 * What's new in version 1.2
51 - Prototype support for "perf events", where the kernel supports the
52 2.6.33 in-kernel API. Probe points may refer to low-level
53 perf_event_attr type/config numbers, or to a number of aliases
54 defined in the new perf.stp tapset:
55 probe perf.sw.cpu_clock, perf.type(0).config(4) { }
57 - Type-casting can now use multiple headers to resolve codependencies.
58 @cast(task, "task_struct",
59 "kernel<linux/sched.h><linux/fs_struct.h>")->fs->umask
61 - Tapset-related man pages have been renamed. 'man -k 3stap' should show
62 the installed list, which due to prefixing should no longer collide over
63 ordinary system functions.
65 - User space marker arguments no longer use volatile if the version of gcc,
66 which must be at least 4.5.0, supports richer DWARF debuginfo. Use cflags
67 -DSTAP_SDT_VOLATILE=volatile or -DSTAP_SDT_VOLATILE= when building
68 the sys/sdt.h application to override this one way or another.
70 - A new construct for error handling is available. It is similar to c++
71 exception catching, using try and catch as new keywords. Within a handler
72 or function, the following is valid and may be nested:
73 try { /* arbitrary statements */ }
74 catch (er) { /* e.g. println("caught error ", er) */ }
76 - A new command line flag '-W' forces systemtap to abort translation of
77 a script if any warnings are produced. It is similar to gcc's -Werror.
78 (If '-w' is also supplied to suppress warnings, it wins.)
80 - A new predicate @defined is available for testing whether a
81 particular $variable/expression is resolvable at translate time:
82 probe foo { if (@defined($bar)) log ("$bar is available here") }
84 - Adjacent string literals are glued together, making this
86 probe process("/usr" @1 "/bin").function("*") { ... }
88 - In order to limit potential impact from future security problems,
89 the stap-server process does not permit its being launched as root.
91 - On recent kernels, for some architectures/configurations, hardware
92 breakpoint probes are supported. The probe point syntax is:
94 probe kernel.data(ADDRESS).write
95 probe kernel.data(ADDRESS).length(LEN).write
96 probe kernel.data("SYMBOL_NAME").write
98 * What's new in version 1.1
100 - New tracepoint based tapset for memory subsystem.
102 - The loading of signed modules by staprun is no longer allowed for
103 ordinary, unprivileged users. This means that only root, members of
104 the group 'stapdev' and members of the group 'stapusr' can load
105 systemtap modules using staprun, stap or stap-client. The minimum
106 privilege required to run arbitrary --unprivileged scripts is now
107 'stapusr' membership.
109 - The stap-server initscript is available. This initscript allows you
110 to start systemtap compile servers as a system service and to manage
111 these servers as a group or individually. The stap-server initscript
112 is installed by the systemtap-server rpm. The build directory for
113 the uprobes module (/usr/share/systemtap/runtime/uprobes) is made
114 writable by the 'stap-server' group. All of the files generated when
115 building the uprobes module, including the digital signature, are
116 also writable by members of stap-server.
118 See initscript/README.stap-server for details.
120 - Some of the compile server client, server and certificate management
121 tools have been moved from $bindir to $libexecdir/systemtap.
122 You should use the new stap-server script or the stap-server initscript
123 for server management where possible. The stap-server script provides the same
124 functionality as the stap-server initscript except that the servers are
125 run by the invoking user by default as opposed to servers started by the
126 stap-server initscript which are run by the user stap-server
127 by default. See stap-server(8) for more information.
129 You may continue to use these tools by adding $libexecdir/systemtap to
130 your path. You would need to do this, for example, if you are not root,
131 you want to start a compile server and you are not running systemtap from a
132 private installation. In this case you still need to use stap-start-server.
134 - Any diagnostic output line that starts with "ERROR", as in
135 error("foo"), will promote a "Pass 5: run failed", and the return
138 - Systemtap now warns about global variables being referenced from other
139 script files. This aims to protect against unintended local-vs-global
140 namespace collisions such as:
142 % cat some_tapset.stp
143 probe baz.one = bar { foo = $foo; bar = $bar }
144 % cat end_user_script.stp
145 global foo # intended to be private variable
146 probe timer.s(1) { foo ++ }
147 probe baz.* { println(foo, pp()) }
148 % stap end_user_script.stp
149 WARNING: cross-file global variable reference to foo from some_tapset.stp
151 - Preprocessor conditional for kernel configuration testing:
152 %( CONFIG_foo == "y" %? ... %)
154 - ftrace(msg:string) tapset function to send strings to the system-wide
155 ftrace ring-buffer (if any).
157 - Better support for richer DWARF debuginfo output from GCC 4.5
158 (variable tracking assignments). Kernel modules are now always resolved
159 against all their dependencies to find any info referring to missing
160 symbols. DW_AT_const_value is now supported when no DW_AT_location
163 * What's new in verson 1.0
165 - process().mark() probes now use an enabling semaphore to reduce the
166 computation overhead of dormant probes.
168 - The function spec for dwarf probes now supports C++ scopes, so you can
169 limit the probes to specific namespaces or classes. Multiple scopes
170 can be specified, and they will be matched progressively outward.
171 probe process("foo").function("std::vector<*>::*") { }
172 probe process("foo").function("::global_function") { }
174 - It is now possible to cross-compile systemtap scripts for foreign
175 architectures, using the new '-a ARCH' and '-B OPT=VALUE' flags.
176 For example, put arm-linux-gcc etc. into your $PATH, and point
177 systemtap at the target kernel build tree with:
178 stap -a arm -B CROSS_COMPILE=arm-linux- -r /build/tree [...]
179 The -B option is passed to kbuild make. -r identifies the already
180 configured/built kernel tree and -a its architecture (kbuild ARCH=...).
181 Systemtap will infer -p4.
183 - Cross compilation using the systemtap client and server
184 - stap-start-server now accepts the -r, -R, -I, -B and -a options in
185 order to start a cross compiling server. The server will correctly
186 advertise itself with respect to the kernel release and architecture
187 that it compiles for.
188 - When specified on stap-client, the -r and -a options will be
189 considered when searching for a suitable server.
191 - When using the systemtap client and server udp port 5353 must be open
192 in your firewall in order for the client to find servers using
193 avahi-browse. Also the systemtap server will choose a random port in
194 the range 1024-63999 for accepting ssl connections.
196 - Support for unprivileged users:
197 ***********************************************************************
198 * WARNING!!!!!!!!!! *
199 * This feature is EXPERIMENTAL at this time and should be used with *
200 * care. This feature allows systemtap kernel modules to be loaded by *
201 * unprivileged users. The user interface and restrictions will change *
202 * as this feature evolves. *
203 ***********************************************************************
204 - Systemtap modules generated from scripts which use a restricted
205 subset of the features available may be loaded by staprun for
206 unprivileged users. Previously, staprun would load modules only for
207 root or for members of the groups stapdev and stapusr.
208 - Using the --unprivileged option on stap enables translation-time
209 checking for use by unprivileged users (see restrictions below).
210 - All modules deemed suitable for use by unprivileged users will be
211 signed by the systemtap server when --unprivileged is specified on
212 stap-client. See module signing in release 0.9.8 and stap-server in
214 - Modules signed by trusted signers (servers) and verified by staprun
215 will be loaded by staprun regardless of the user's privilege level.
216 - The system administrator asserts the trustworthiness of a signer
217 (server) by running stap-authorize-signing-cert <cert-file> as root,
218 where the <cert-file> can be found in
219 ~<user>/.systemtap/ssl/server/stap.cert for servers started by
220 ordinary users and in $sysconfdir/systemtap/ssl/server/stap.cert for
221 servers started by root.
222 - Restrictions are intentionally strict at this time and may be
223 relaxed in the future:
224 - probe points are restricted to:
225 begin, begin(n), end, end(n), error, error(n), never,
226 timer.{jiffies,s,sec,ms,msec,us,usec,ns,nsec}(n)*, timer.hz(n),
227 process.* (for processes owned by the user).
228 - use of embedded C code is not allowed.
229 - use of tapset functions is restricted.
230 - some tapset functions may not be used at all. A message will be
231 generated at module compilation time.
232 - some actions by allowed tapset functions may only be performed
233 in the context of the user's own process. A runtime fault will
234 occur in these situations, for example, direct memory access.
235 - The is_myproc() tapset function has been provided so that
236 tapset writers for unprivileged users can check that the
237 context is of the users own process before attempting these
239 - accessing the kernel memory space is not allowed.
240 - The following command line options may not be used by stap-client
242 - The following environment variables are ignored by stap-client:
243 SYSTEMTAP_RUNTIME, SYSTEMTAP_TAPSET, SYSTEMTAP_DEBUGINFO_PATH
244 - nss and nss-tools are required to use this feature.
246 - Support output file switching by SIGUSR2. Users can command running
247 stapio to switch output file by sending SIGUSR2.
249 - Memory consumption for scripts involving many uprobes has been
250 dramatically reduced.
252 - The preprocessor now supports || and && in the conditions.
253 e.g. %( arch == "x86_64" || arch == "ia64" %: ... %)
255 - The systemtap notion of "architecture" now matches the kernel's, rather
256 than that of "uname -m". This means that 32-bit i386 family are all
257 known as "i386" rather than "i386" or "i686"; "ppc64" as "powerpc";
258 "s390x" as "s390", and so on. This is consistent between the new
259 "-a ARCH" flag and the script-level %( arch ... %) conditional.
261 - It is now possible to define multiple probe aliases with the same name.
262 A probe will expand to all matching aliases.
265 probe foo { } # expands twice, once to bar and once to baz
267 - A new experimental transport mechanism, using ftrace's ring_buffer,
268 has been added. This may become the default transport mechanism in
269 future versions of systemtap. To test this new transport mechanism,
270 define 'STP_USE_RING_BUFFER'.
272 - Support for recognizing DW_OP_{stack,implicit}_value DWARF expressions
273 as emitted by GCC 4.5.
275 * What's new in version 0.9.9
277 - Systemwide kernel .function.return (kretprobe) maxactive defaults may
278 be overridden with the -DKRETACTIVE=nnn parameter.
280 - Translation pass 2 is significantly faster by avoiding unnecessary
281 searching through a kernel build/module directory tree.
283 - When compiled against elfutils 0.142 systemtap now handles the new
284 DW_OP_call_frame_CFA generated by by GCC.
286 - uprobes and ustack() are more robust when used on applications that
287 depend on prelinked/separate debuginfo shared libraries.
289 - User space PROBE marks are not always found with or without separate
290 debuginfo. The .probes section itself is now always put in the main
291 elf file and marked as allocated. When building pic code the section
292 is marked writable. The selinux memory check problems seen with
293 programs using STAP_PROBES is fixed.
295 - statement() probes can now override "address not at start of statement"
296 errors in guru mode. They also provide alternative addresses to use
299 - The stapgraph application can generate graphs of data and events
300 emitted by systemtap scripts in real time. Run "stapgraph
301 testsuite/systemtap.examples/general/grapher.stp" for an example of
302 graphing the system load average and keyboard events.
304 - Dwarf probes now show parameters and local variables in the verbose
307 - Symbol aliases are now resolved to their canonical dwarf names. For
308 example, probing "malloc" in libc resolves to "__libc_malloc".
310 - The syntax for dereferencing $target variables and @cast() gained new
312 - Array indexes can now be arbitrary numeric expressions.
313 - Array subscripts are now supported on pointer types.
314 - An '&' operator before a @cast or $target returns the address of the
315 final component, especially useful for nested structures.
317 - For reading all probe variables, kernel.mark now supports $$vars and
318 $$parms, and process.syscall now supports $$vars.
320 - The SNMP tapset provides probes and functions for many network
321 statistics. See stapprobes.snmp(3stap) for more details.
323 - The dentry tapset provides functions to map kernel VFS directory entries
324 to file or full path names: d_path(), d_name() and reverse_path_walk().
326 - SystemTap now has userspace markers in its own binaries, and the stap
327 tapset provides the available probepoints and local variables.
329 - Miscellaneous new tapset functions:
330 - pgrp() returns the process group ID of the current process
331 - str_replace() performs string replacement
333 * What's new in version 0.9.8
335 - Miscellaneous new tapset functions:
336 - sid() returns the session ID of the current process
337 - stringat() indexes a single character from a string.
339 - Using %M in print formats for hex dumps can now print entire buffers,
340 instead of just small numbers.
342 - Dwarfless syscalls: The nd_syscalls tapset is now available to probe
343 system calls without requiring kernel debugging information. All of
344 the same probepoints in the normal syscalls tapset are available with
345 an "nd_" prefix, e.g. syscall.open becomes nd_syscall.open. Most
346 syscall arguments are also available by name in nd_syscalls.
348 - Module signing: If the appropriate nss libraries are available on your
349 system, stap-server will sign each compiled module using a self-generated
350 certificate. This is the first step toward extending authority to
351 load certain modules to unprivileged users. For now, if the system
352 administrator adds a certificate to a database of trusted signers
353 (stap-authorize-signing-cert), modules signed using that certificate
354 will be verified by staprun against tampering. Otherwise, you should
355 notice no difference in the operation of stap or staprun.
357 * What's new in version 0.9.7
359 - @cast can now determine its type information using an explicit header
360 specification. For example:
361 @cast(tv, "timeval", "<sys/time.h>")->tv_sec
362 @cast(task, "task_struct", "kernel<linux/sched.h>")->tgid
364 - The overlapping process.* tapsets are now separated. Those probe points
365 documented in stapprobes(3stap) remain the same. Those that were formerly
366 in stapprobes.process(3stap) have been renamed to kprocess, to reflect
367 their kernel perspective on processes.
369 - The --skip-badvars option now also suppresses run-time error
370 messages that would otherwise result from erroneous memory accesses.
371 Such accesses can originate from $context expressions fueled by
372 erroneous debug data, or by kernel_{long,string,...}() tapset calls.
374 - New probes kprobe.function(FUNCTION) and kprobe.function(FUNCTION).return
375 for dwarfless probing. These postpone function address resolution to
376 run-time and use the kprobe symbol-resolution mechanism.
377 Probing of absolute statements can be done using the
378 kprobe.statement(ADDRESS).absolute construct.
380 - EXPERIMENTAL support for user process unwinding. A new collection of
381 tapset functions have been added to handle user space backtraces from
382 probe points that support them (currently process and timer probes -
383 for timer probes test whether or not in user space first with the
384 already existing user_mode() function). The new tapset functions are:
385 uaddr - User space address of current running task.
386 usymname - Return the symbol of an address in the current task.
387 usymdata - Return the symbol and module offset of an address.
388 print_ustack - Print out stack for the current task from string.
389 print_ubacktrace - Print stack back trace for current task.
390 ubacktrace - Hex backtrace of current task stack.
391 Please read http://sourceware.org/ml/systemtap/2009-q2/msg00364.html
392 on the current restrictions and possible changes in the future and
393 give feedback if you want to influence future developments.
395 * What's new in version 0.9.5
397 - New probes process().insn and process().insn.block that allows
398 inspection of the process after each instruction or block of
399 instructions executed. So to count the total number of instructions
400 a process executes during a run do something like:
401 $ stap -e 'global steps; probe process("/bin/ls").insn {steps++}
402 probe end {printf("Total instructions: %d\n", steps);}' \
404 This feature can slow down execution of a process somewhat.
406 - Systemtap probes and function man pages extracted from the tapsets
407 are now available under 3stap. To show the page for probe vm.pagefault
408 or the stap function pexecname do:
409 $ man 3stap vm.pagefault
410 $ man 3stap pexecname
412 - Kernel tracepoints are now supported for probing predefined kernel
413 events without any debuginfo. Tracepoints incur less overhead than
414 kprobes, and context parameters are available with full type
415 information. Any kernel 2.6.28 and later should have defined
416 tracepoints. Try the following to see what's available:
417 $ stap -L 'kernel.trace("*")'
419 - Typecasting with @cast now supports modules search paths, which is
420 useful in case there are multiple places where the type definition
421 may be found. For example:
422 @cast(sdev, "scsi_device", "kernel:scsi_mod")->sdev_state
424 - On-file flight recorder is supported. It allows stap to record huge
425 trace log on the disk and to run in background.
426 Passing -F option with -o option runs stap in background mode. In this
427 mode, staprun is detached from console, and stap itself shows staprun's
429 Specifying the max size and the max number of log files are also available
430 by passing -S option. This option has one or two arguments seperated by
431 a comma. The first argument is the max size of a log file in MB. If the
432 size of a log file exceeds it, stap switches to the next log file
433 automatically. The second is how many files are kept on the disk. If the
434 number of log files exceeds it, the oldest log file is removed
435 automatically. The second argument can be omitted.
437 For example, this will record output on log files each of them is smaller
438 than 1024MB and keep last 3 logs, in background.
439 % stap -F -o /tmp/staplog -S 1024,3 script.stp
441 - In guru mode (-g), the kernel probing blacklist is disabled, leaving
442 only a subset - the kernel's own internal kprobe blacklist - to attempt
443 to filter out areas unsafe to probe. The differences may be enough to
444 probe more interrupt handlers.
446 - Variables unavailable in current context may be skipped by setting a
447 session level flag with command line option --skip-badvars now available.
448 This replaces any dwarf $variable expressions that could not be resolved
449 with literal numeric zeros, along with a warning message.
451 - Both kernel markers and kernel tracepoint support argument listing
452 through stap -L 'kernel.mark("*")' or stap -L 'kernel.trace("*")'
454 - Users can use -DINTERRUPTIBLE=0 to prevent interrupt reentrancy in
455 their script, at the cost of a bit more overhead to toggle the
458 - Added reentrancy debugging. If stap is run with the arguments
459 "-t -DDEBUG_REENTRANCY", additional warnings will be printed for
460 every reentrancy event, including the probe points of the
461 resident and interloper probes.
463 - Default to --disable-pie for configure.
464 Use --enable-pie to turn it back on.
466 - Improved sdt.h compatibility and test suite for static dtrace
467 compatible user space markers.
469 - Some architectures now use syscall wrappers (HAVE_SYSCALL_WRAPPERS).
470 The syscall tapset has been enhanced to take care of the syscall
471 wrappers in this release.
473 - Security fix for CVE-2009-0784: stapusr module-path checking race.
475 * What's new in version 0.9
477 - Typecasting is now supported using the @cast operator. A script can
478 define a pointer type for a "long" value, and then access type members
479 using the same syntax as with $target variables. For example, this will
480 retrieve the parent pid from a kernel task_struct:
481 @cast(pointer, "task_struct", "kernel")->parent->pid
483 - process().mark() probes are now possible to trace static user space
484 markers put in programs with the STAP_PROBE macro using the new
485 sys/sdt.h include file. This also provides dtrace compatible markers
486 through DTRACE_PROBE and an associated python 'dtrace' script that
487 can be used in builds based on dtrace that need dtrace -h or -G
490 - For those that really want to run stap from the build tree there is
491 now the 'run-stap' script in the top-level build directory that sets
492 up the SYSTEMTAP_TAPSET, SYSTEMTAP_RUNTIME, SYSTEMTAP_STAPRUN, and
493 SYSTEMTAP_STAPIO environment variables (installing systemtap, in a
494 local prefix, is still recommended for common use).
496 - Systemtap now comes with a new Beginners Guide that walks the user
497 through their first steps setting up stap, understanding how it all
498 works, introduces some useful scripts and describes some common
499 pitfalls. It isn't created by default since it needs a Publican
500 setup, but full build instructions can be found in the wiki:
501 http://sourceware.org/systemtap/wiki/PublicanQuikHowto
502 An online version can be found at:
503 http://sourceware.org/systemtap/SystemTap_Beginners_Guide.pdf
505 - Standard tapsets included with Systemtap were modified to include
506 extractable documentation information based on the kernel-doc
507 infrastructure. When configured --enabled-docs a HTML and PDF
508 version of the Tapset Reference Manual is produced explaining probes
509 defined in each tapset.
511 - The systemtap client and compile server are now available.
512 These allow you to compile a systemtap module on a host other than
513 the one which it will be run, providing the client and server
514 are compatible. Other than using a server for passes 1 through
515 4, the client behaves like the 'stap' front end itself. This
516 means, among other things, that the client will automatically
517 load the resulting module on the local host unless -p[1234]
518 was specified. See stap-server(8) for more details.
519 The client/server now use SSL for network connection security and
522 The systemtap client and server are prototypes only. Interfaces, options
523 and usage may change at any time.
525 - function("func").label("label") probes are now supported to allow matching
526 the label of a function.
528 - Systemtap initscript is available. This initscript allows you to run
529 systemtap scripts as system services (in flight recorder mode) and
530 control those scripts individually.
531 See README.systemtap for details.
533 - The stap "-r DIR" option may be used to identify a hand-made kernel
534 build directory. The tool determines the appropriate release string
535 automatically from the directory.
537 - Serious problems associated with user-space probing in shared libraries
538 were corrected, making it now possible to experiment with probe shared
539 libraries. Assuming dwarf debugging information is installed, use this
540 twist on the normal syntax:
542 probe process("/lib64/libc-2.8.so").function("....") { ... }
544 This would probe all threads that call into that library. Running
545 "stap -c CMD" or "stap -x PID" naturally restricts this to the target
546 command+descendants only. $$vars etc. may be used.
548 - For scripts that sometimes terminate with excessive "skipped" probes,
549 rerunning the script with "-t" (timing) will print more details about
550 the skippage reasons.
552 - Symbol tables and unwind (backtracing) data support were formerly
553 compiled in for all probed modules as identified by the script
554 (kernel; module("name"); process("file")) plus those listed by the
555 stap "-d BINARY" option. Now, this data is included only if the systemtap
556 script uses tapset functions like probefunc() or backtrace() that require
557 such information. This shrinks the probe modules considerably for the rest.
559 - Per-pass verbosity control is available with the new "--vp {N}+" option.
560 "stap --vp 040" adds 4 units of -v verbosity only to pass 2. This is useful
561 for diagnosing errors from one pass without excessive verbosity from others.
563 - Most probe handlers now run with interrupts enabled, for improved
564 system responsiveness and less probing overhead. This may result
565 in more skipped probes, for example if a reentrant probe handler
566 is attempted from within an interrupt handler. It may also make the
567 systemtap overload detection facility more likely to be triggered, as
568 interrupt handlers' run time would be included in the self-assessed
569 overhead of running probe handlers.
571 * What's new in version 0.8
573 - Cache limiting is now available. If the compiled module cache size is
574 over a limit specified in the $SYSTEMTAP_DIR/cache/cache_mb_limit file,
575 some old cache entries will be unlinked. See man stap(1) for more.
577 - Error and warning messages are now followed by source context displaying
578 the erroneous line/s and a handy '^' in the following line pointing to the
581 - A bug reporting tool "stap-report" is now available which will quickly
582 retrieve much of the information requested here:
583 http://sourceware.org/systemtap/wiki/HowToReportBugs
585 - The translator can resolve members of anonymous structs / unions:
586 given struct { int foo; struct { int bar; }; } *p;
587 this now works: $p->bar
589 - The stap "-F" flag activates "flight recorder" mode, which consists of
590 translating the given script as usual, but implicitly launching it into
591 the background with staprun's existing "-L" (launch) option. A user
592 can later reattach to the module with "staprun -A MODULENAME".
594 - Additional context variables are available on user-space syscall probes.
595 - $argN ($arg1, $arg2, ... $arg6) in process(PATH_OR_PID).syscall
596 gives you the argument of the system call.
597 - $return in process(PATH_OR_PID).syscall.return gives you the return
598 value of the system call.
600 - Target process mode (stap -c CMD or -x PID) now implicitly restricts all
601 "process.*" probes to the given child process. (It does not affect
602 kernel.* or other probe types.) The CMD string is normally run directly,
603 rather than via a /bin/sh -c subshell, since then utrace/uprobe probes
604 receive a fairly "clean" event stream. If metacharacters like
605 redirection operators were present in CMD, then "sh -c CMD" is still
606 used, and utrace/uprobe probes will receive events from the shell.
608 % stap -e 'probe process.syscall, process.end {
609 printf("%s %d %s\n", execname(), pid(), pp())}'\
611 ls 2323 process.syscall
612 ls 2323 process.syscall
615 - Probe listing mode is improved: "-L" lists available script-level variables
617 % stap -L 'syscall.*open*'
618 syscall.mq_open name:string name_uaddr:long filename:string mode:long u_attr_uaddr:long oflag:long argstr:string
619 syscall.open name:string filename:string flags:long mode:long argstr:string
620 syscall.openat name:string filename:string flags:long mode:long argstr:string
622 - All user-space-related probes support $PATH-resolved executable
625 probe process("ls").syscall {}
626 probe process("./a.out").syscall {}
628 work now, instead of just
630 probe process("/bin/ls").syscall {}
631 probe process("/my/directory/a.out").syscall {}
633 - Prototype symbolic user-space probing support:
635 # stap -e 'probe process("ls").function("*").call {
636 log (probefunc()." ".$$parms)
641 - debugging information for the named program
642 - a version of utrace in the kernel that is compatible with the "uprobes"
643 kernel module prototype. This includes RHEL5 and older Fedora, but not
644 yet current lkml-track utrace; a "pass 4a"-time build failure means
645 your system cannot use this yet.
647 - Global variables which are written to but never read are now
648 automatically displayed when the session does a shutdown. For example:
651 probe timer.profile {running_tasks[pid(),tid()] = execname()}
652 probe timer.ms(8000) {exit()}
654 - A formatted string representation of the variables, parameters, or local
655 variables at a probe point is now supported via the special $$vars,
656 $$parms, and $$locals context variables, which expand to a string
657 containing a list "var1=0xdead var2=0xbeef var3=?". (Here, var3 exists
658 but is for some reason unavailable.) In return probes only, $$return
659 expands to an empty string for a void function, or "return=0xf00".
662 * What's new in version 0.7
664 - .statement("func@file:*") and .statement("func@file:M-N") probes are now
665 supported to allow matching a range of lines in a function. This allows
666 tracing the execution of a function.
668 - Scripts relying on probe point wildcards like "syscall.*" that expand
669 to distinct kprobes are processed significantly faster than before.
671 - The vector of script command line arguments is available in a
672 tapset-provided global array argv[]. It is indexed 1 ... argc,
673 another global. This can substitute for of preprocessor
674 directives @NNN that fail at parse time if there are not
677 printf("argv: %s %s %s", argv[1], argv[2], argv[3])
679 - .statement("func@file+line") probes are now supported to allow a
680 match relative to the entry of the function incremented by line
681 number. This allows using the same systemtap script if the rest
682 of the file.c source only changes slightly.
684 - A probe listing mode is available.
693 - More user-space probe types are added:
695 probe process(PID).begin { }
696 probe process("PATH").begin { }
697 probe process(PID).thread.begin { }
698 probe process("PATH").thread.begin { }
699 probe process(PID).end { }
700 probe process("PATH").end { }
701 probe process(PID).thread.end { }
702 probe process("PATH").thread.end { }
703 probe process(PID).syscall { }
704 probe process("PATH").syscall { }
705 probe process(PID).syscall.return { }
706 probe process("PATH").syscall.return { }
708 - Globals now accept ; terminators
711 global little[10], big[5];
713 * What's new in version 0.6
715 - A copy of the systemtap tutorial and language reference guide
718 - There is a new format specifier, %m, for the printf family of
719 functions. It functions like %s, except that it does not stop when
720 a nul ('\0') byte is encountered. The number of bytes output is
721 determined by the precision specifier. The default precision is 1.
724 printf ("%m", "My String") // prints one character: M
725 printf ("%.5", myString) // prints 5 bytes beginning at the start
728 - The %b format specifier for the printf family of functions has been enhanced
731 1) When the width and precision are both unspecified, the default is %8.8b.
732 2) When only one of the width or precision is specified, the other defaults
733 to the same value. For example, %4b == %.4b == %4.4b
734 3) Nul ('\0') bytes are used for field width padding. For example,
736 printf ("%b", 0x1111deadbeef2222) // prints all eight bytes
737 printf ("%4.2b", 0xdeadbeef) // prints \0\0\xbe\xef
739 - Dynamic width and precision are now supported for all printf family format
740 specifiers. For example:
744 printf ("%*.*b", four, two, 0xdeadbbeef) // prints \0\0\xbe\xef
745 printf ("%*d", four, two) // prints <space><space><space>2
747 - Preprocessor conditional expressions can now include wildcard style
748 matches on kernel versions.
749 %( kernel_vr != "*xen" %? foo %: bar %)
751 - Prototype support for user-space probing is showing some progress.
752 No symbolic notations are supported yet (so no probing by function names,
753 file names, process names, and no access to $context variables), but at
754 least it's something:
756 probe process(PID).statement(ADDRESS).absolute { }
758 This will set a uprobe on the given process-id and given virtual address.
759 The proble handler runs in kernel-space as usual, and can generally use
760 existing tapset functions.
762 - Crash utility can retrieve systemtap's relay buffer from a kernel dump
763 image by using staplog which is a crash extension module. To use this
764 feature, type commands as below from crash(8)'s command line:
766 crash> extend staplog.so
767 crash> help systemtaplog
769 Then, you can see more precise help message.
771 - You can share a relay buffer amoung several scripts and merge outputs from
772 several scripts by using "-DRELAY_HOST" and "-DRELAY_GUEST" options.
776 % stap -ve 'probe begin{}' -o merged.out -DRELAY_HOST &
777 # wait until starting the host.
778 % stap -ve 'probe begin{print("hello ");exit()}' -DRELAY_GUEST
779 % stap -ve 'probe begin{print("world\n");exit()}' -DRELAY_GUEST
781 Then, you'll see "hello world" in merged.out.
783 - You can add a conditional statement for each probe point or aliase, which
784 is evaluated when the probe point is hit. If the condition is false, the
785 whole probe body(including aliases) is skipped. For example:
788 probe syscall.* if (switch) { ... }
789 probe procfs.write {switch = strtol($value,10)} /* enable/disable ctrl */
791 - Systemtap will warn you if your script contains unused variables or
792 functions. This is helpful in case of misspelled variables. If it
793 doth protest too much, turn it off with "stap -w ...".
795 - You can add error-handling probes to a script, which are run if a
796 script was stopped due to errors. In such a case, "end" probes are
797 not run, but "error" ones are.
799 probe error { println ("oops, errors encountered; here's a report anyway")
800 foreach (coin in mint) { println (coin) } }
802 - In a related twist, one may list probe points in order of preference,
803 and mark any of them as "sufficient" beyond just "optional". Probe
804 point sequence expansion stops if a sufficient-marked probe point has a hit.
805 This is useful for probes on functions that may be in a module (CONFIG_FOO=m)
806 or may have been compiled into the kernel (CONFIG_FOO=y), but we don't know
809 probe module("sd").function("sd_init_command") ? ,
810 kernel.function("sd_init_command") ? { ... }
812 which might match neither, now one can write this:
814 probe module("sd").function("sd_init_command") ! , /* <-- note excl. mark */
815 kernel.function("sd_init_command") { ... }
817 - New security model. To install a systemtap kernel module, a user
818 must be one of the following: the root user; a member of the
819 'stapdev' group; or a member of the 'stapusr' group. Members of the
820 stapusr group can only use modules located in the
821 /lib/modules/VERSION/systemtap directory (where VERSION is the
822 output of "uname -r").
824 - .statement("...@file:line") probes now apply heuristics to allow an
825 approximate match for the line number. This works similarly to gdb,
826 where a breakpoint placed on an empty source line is automatically
827 moved to the next statement. A silly bug that made many $target
828 variables inaccessible to .statement() probes was also fixed.
830 - LKET has been retired. Please let us know on <systemtap@sourceware.org>
831 if you have been a user of the tapset/tools, so we can help you find
834 - New families of printing functions println() and printd() have been added.
835 println() is like print() but adds a newline at the end;
836 printd() is like a sequence of print()s, with a specified field delimiter.
838 * What's new since version 0.5.14?
840 - The way in which command line arguments for scripts are substituted has
841 changed. Previously, $1 etc. would interpret the corresponding command
842 line argument as an numeric literal, and @1 as a string literal. Now,
843 the command line arguments are pasted uninterpreted wherever $1 etc.
844 appears at the beginning of a token. @1 is similar, but is quoted as
845 a string. This change does not modify old scripts, but has the effect
846 of permitting substitution of arbitrary token sequences.
848 # This worked before, and still does:
849 % stap -e 'probe timer.s($1) {}' 5
850 # Now this also works:
851 % stap -e 'probe syscall.$1 {log(@1)}' open
852 # This won't crash, just signal a recursion error:
854 # As before, $1... is recognized only at the beginning of a token
855 % stap -e 'probe begin {foo$1=5}'
857 * What's new since version 0.5.13?
859 - The way in which systemtap resolves function/inline probes has changed:
860 .function(...) - now refers to all functions, inlined or not
861 .inline(...) - is deprecated, use instead:
862 .function(...).inline - filters function() to only inlined instances
863 .function(...).call - filters function() to only non-inlined instances
864 .function(...).return - as before, but now pairs best with .function().call
865 .statement() is unchanged.
867 * What's new since version 0.5.12?
869 - When running in -p4 (compile-only) mode, the compiled .ko file name
870 is printed on standard output.
872 - An array element with a null value such as zero or an empty string
873 is now preserved, and will show up in a "foreach" loop or "in" test.
874 To delete such an element, the scripts needs to use an explicit
875 "delete array[idx]" statement rather than something like "array[idx]=0".
877 - The new "-P" option controls whether prologue searching heuristics
878 will be activated for function probes. This was needed to get correct
879 debugging information (dwarf location list) data for $target variables.
880 Modern compilers (gcc 4.1+) tend not to need this heuristic, so it is
881 no longer default. A new configure flag (--enable-prologues) restores
882 it as a default setting, and is appropriate for older compilers (gcc 3.*).
884 - Each systemtap module prints a one-line message to the kernel informational
885 log when it starts. This line identifies the translator version, base
886 address of the probe module, a broken-down memory consumption estimate, and
887 the total number of probes. This is meant as a debugging / auditing aid.
889 - Begin/end probes are run with interrupts enabled (but with
890 preemption disabled). This will allow begin/end probes to be
891 longer, to support generating longer reports.
893 - The numeric forms of kernel.statement() and kernel.function() probe points
894 are now interpreted as relocatable values - treated as relative to the
895 _stext symbol in that kernel binary. Since some modern kernel images
896 are relocated to a different virtual address at startup, such addresses
897 may shift up or down when actually inserted into a running kernel.
899 kernel.statement(0xdeadbeef): validated, interpreted relative to _stext,
900 may map to 0xceadbeef at run time.
902 In order to specify unrelocated addresses, use the new ".absolute"
903 probe point suffix for such numeric addresses. These are only
904 allowed in guru mode, and provide access to no $target variables.
905 They don't use debugging information at all, actually.
907 kernel.statement(0xfeedface).absolute: raw, unvalidated, guru mode only
909 * What's new since version 0.5.10?
911 - Offline processing of debugging information, enabling general
912 cross-compilation of probe scripts to remote hosts, without
913 requiring identical module/memory layout. This slows down
914 compilation/translation somewhat.
916 - Kernel symbol table data is loaded by staprun at startup time
917 rather than compiled into the module.
919 - Support the "limit" keyword for foreach iterations:
920 foreach ([x,y] in ary limit 5) { ... }
921 This implicitly exits after the fifth iteration. It also enables
922 more efficient key/value sorting.
924 - Support the "maxactive" keyword for return probes:
925 probe kernel.function("sdfsdf").maxactive(848) { ... }
926 This allows up to 848 concurrently outstanding entries to
927 the sdfsdf function before one returns. The default maxactive
928 number is smaller, and can result in missed return probes.
930 - Support accessing of saved function arguments from within
931 return probes. These values are saved by a synthesized
932 function-entry probe.
934 - Add substantial version/architecture checking in compiled probes to
935 assert correct installation of debugging information and correct
936 execution on a compatible kernel.
938 - Add probe-time checking for sufficient free stack space when probe
939 handlers are invoked, as a safety improvement.
941 - Add an optional numeric parameter for begin/end probe specifications,
942 to order their execution.
943 probe begin(10) { } /* comes after */ probe begin(-10) {}
945 - Add an optional array size declaration, which is handy for very small
947 global little[5], big[20000]
949 - Include some example scripts along with the documentation.
951 - Change the start-time allocation of probe memory to avoid causing OOM
952 situations, and to abort cleanly if free kernel memory is short.
954 - Automatically use the kernel DWARF unwinder, if present, for stack
957 - Many minor bug fixes, performance, tapset, and error message