1 * What's new in version 4.1
3 - Runtime/tapsets were ported to include up to kernel version PRERELEASE
7 * What's new in version 4.0, 2018-10-13
9 - Runtime/tapsets were ported to include up to kernel version 4.19-rc
11 - A new network service, stap-exporter, is included. It glues
12 systemtap and the web. It allows a prometheus (or compatible
13 systems such as pcp) to consume metrics exported by systemtap
14 scripts. Some tapset macros/functions are available to make it
15 easier to write such scripts. See the stap-exporter(8) man page and
18 - When a systemtap module is loaded, the name of the original stap script
19 is now printed to dmesg by the kernel runtime.
21 - On some Fedora kernels, the information necessary to automatically
22 engage in SecureBoot module signing is hidden from systemtap.
23 Setting the $SYSTEMTAP_SIGN environment variable forces it on.
24 A running stap-server instance will also be needed.
26 - Embedded-C functions marked /* guru */ may now be invoked from other
27 tapset probes / functions, while still being invalid for normal call
28 from an unprivileged user script.
30 - The syscall tapset is now updated to work on kernel 4.17+.
31 Additionally, the tapset now includes an automatic fallback alias to
32 the sys_enter / sys_exit kernel tracepoints, if no other
33 kprobe-based mechanism is found. These changes have brought
34 unavoidable consequences. Raw $target variables for the syscall
35 arguments and return probes (e.g. @entry($fd), $return, returnval())
36 may not longer be relied upon. Instead, use the variables defined by
37 the tapset aliases. For example:
39 % stap -L syscall.read
40 syscall.read name:string fd:long buf_uaddr:long count:long argstr:string
41 % stap -L syscall.read.return
42 syscall.read.return name:string retval:long retstr:string
44 to see the available variables for that syscall. See
45 [man stapprobes] for further details. returnval() in particular is
46 being deprecated soon; use retval in syscall.*.return probes instead.
48 - New script language operators @kderef/@uderef and @kregister/@uregister
50 @kderef/@uderef (size,address) can be used to dereference integers and
51 @kregister/@uregister (dwarf#) can be used to access register values.
53 - A systemd service file has been added for systemtap.service (which
54 runs a configurable set of scripts automatically on system
55 startup). The existing /etc/init.d/systemtap init script has been
56 moved to a new utility command 'systemtap-service' which preserves
57 functionality such as configuring onboot systemtap scripts via
58 dracut. See systemtap-service(8) for details.
60 - The eBPF backend's string support has been improved. Strings
61 can now be stored in variables, passed as function arguments,
62 and stored as array keys and values.
64 - The 3rd operand of the ternary operator '?:' in the script language
65 now binds tighter than the binary assignment operators like '=' and
66 '+=', just like the C language. The original operator precedence can
67 be restored by the '--compatible 3.3' option.
69 - The script language now supports the use of bare 'return' statements
70 (without any return values) inside functions which do not return any
71 values. A trailing semicolon is recommended for such return
72 statements to avoid any potential ambiguity. The parser treats a
73 following semicolon (';') or a closing curly bracket ('}') as a
74 terminator for such bare return statements.
76 - Parentheses after unary '&' with a target-symbol expression is
77 now accepted in the script language.
79 - Tapset functions register() and u_register() now support 8-bit
80 x86 register names "ah", "al", "bh", "bl", "ch", "cl", "dh", and
81 "dl" on both x86_64 and i386. And 16-bit x86 registers are now
82 truly read as 16-bit integers instead of as 32-bit ones.
84 - The experimental ftrace ring buffer mechanism (STP_USE_RING_BUFFER)
85 has been deprecated and may be removed in future versions.
87 * What's new in version 3.3, 2018-06-08
89 - A new "stap --example FOO.stp" mode searches the example scripts
90 distributed with systemtap for a file named FOO.stp, so its whole
91 path does not need to be typed in.
93 - Systemtap's runtime has learned to deal with several of the
94 collateral damage from kernel hardening after meltdown/spectre,
95 including more pointer hiding and relocation. The kptr_restrict
96 procfs flag is forced on if running on a new enough kernel.
98 - The "stap --sysroot /PATH" option has received a revamp, so it
99 works much better against cross-compiled environments.
101 - The eBPF backend has learned to perform loops - at least in the
102 userspace "begin/end" probe contexts, so one can iterate across BPF
103 arrays for reporting. (The linux kernel eBPF interpreter precludes
104 loops and string processing.) It can also handle much larger probe
105 handler bodies, with a smarter register spiller/allocator.
107 - The eBPF backend now supports uprobes, perf counter, timer, and
110 - An rpm macro %_systemtap_tapsetdir is now defined, to make it
111 easier for third party packages to add .stp files into the standard
114 - Several low level locking-related fixes were added to the runtime
115 that used uprobes/tracepoint apis, in order to work more reliably on
116 rt kernels and on high-cpu-count machines.
118 - Runtime/tapsets were ported to include up to kernel version 4.16.
119 (The syscall tapsets are broken on kernel 4.17-rc, and will be fixed
120 in a next release coming soon; PR23160.)
122 - Add new built-in tapset function abort() which is similar to exit(),
123 but it aborts the current probe handler (and any function calls in
124 it) immediately instead of waiting for its completion. Probe handlers
125 already running on *other* CPU cores, however, will still continue
126 to their completion. Unlike error(), abort() cannot be caught by
127 try {...} catch {...}. Similar to exit(), abort() yeilds the zero
128 process exit code. It works with both the kernel and dyninst runtimes.
129 This function can be disabled by the '--compatible 3.3' option.
131 * What's new in version 3.2, 2017-10-18
133 - SystemTap now includes an extended Berkeley Packet Filter (eBPF)
134 backend. This experimental backend does not use kernel modules
135 and instead produces eBPF programs that are verified by the kernel
136 and executed by an in-kernel virtual machine. Select this backend
137 with the new stap option '--runtime=bpf'. For example:
139 stap --runtime=bpf -e \
140 'probe kernel.function("sys_open") { printf("hi from stapbpf!\n") }'
142 Please see the stapbpf(8) man page for more information.
144 - The regular expression engine now supports extraction of the matched
145 string and subexpressions using the matched() tapset function:
147 if ("regexculpicator" =~ "reg(ex.*p).*r") log(matched(1))
150 - The translator produces better diagnostics for common/annoying case
151 of missing debuginfo that blocks use of context $variables.
153 - "stap -k" build trees in $TMPDIR now also include a preprocessed .i form
154 of the generated module .c code, for problem diagnostics purposes.
156 - The syscall.execve probes now provide a decoded env_str string vector,
157 just like the argument vector. Because of this, the unused
158 __count_envp() and __count_compat_evenp() functions have been
161 - The task_exe_file() Function has been deprecated and replaced by the
162 current_exe_file() function.
164 - A new probe alias input.char allows scripts to access input from stdin
167 * What's new in version 3.1, 2017-02-17
169 - Systemtap now needs C++11 to build.
171 - Syscall and nd_syscall tapsets have been merged in a way that either
172 dwarf-based, or non-dwarf probe gets automatically used based on
173 debuginfo availability (e.g. probe syscall.open).
175 To force use the dwarf based probe, a dw_syscall has been introduced
176 (e.g. probe dw_syscall.open) and the non-dwarf syscall probes were
177 left untouched (e.g. nd_syscall.open).
179 - The syscall tapset files have been reorganized in a way that original
180 big tapset files carrying many syscall probes were split into smaller
181 'sysc_' prefixed tapset files. This should reduce the syscall tapset
184 - The powerpc variant of syscall.compat_sysctl got deprecated on favor of
185 syscall.sysctl32. This aligns the syscall to its respective nd_syscall and
186 to ia64/s390/x86_64 variants too.
188 - The syscall.compat_pselect7a (this was actually a typo, but still available
189 for compatibility purposes with --compatible 1.3) has beed deprecated.
191 - The 'description_auddr' convenience variable of syscall.add_key has been
194 - Support has been added for probing python 2 and 3 functions using a
195 custom python helper module. Python function probes can target
196 function entry, returns, or specific line numbers.
198 probe python2.module("myscript").function("foo")
201 To run with the custom python helper module, you'd use python's '-m'
202 option like the following:
204 stap myscript.stp -c "python -m HelperSDT myscript.py"
206 - Java method probes now convert all types of java parameters to
207 strings using the java toString() method before passing them to
208 systemtap probes; new argN variables copy them into string
209 variables. Previously, only numeric types were passed, and only by
210 casting to integers. The previous behaviour is available with
213 3.1: probe java(...).class(...).method(...) { printf("%s", arg1) }
214 3.0: probe java(...).class(...).method(...) { printf("%d", $arg1) }
216 - An older defensive measure to suppress kernel kprobes optimizations
217 since the 3.x era has been disabled for recent kernels. This improves
218 the performance of kernel function probes. In case of related problems,
219 please report and work around with:
220 # echo 0 > /proc/sys/debug/kprobes-optimization
222 - Context variables in .return probes should be accessed with @entry($var)
223 rather than $var, to make it clear that entry-time snapshots are being
224 used. The latter construct now generates a warning. Availability testing
225 with either @defined(@entry($var)) or @defined($var) works.
227 - Tapsets containing process probes may now be placed in the special
228 $prefix/share/systemtap/tapset/PATH/ directory to have their process parameter
229 prefixed with the location of the tapset. For example,
231 process("foo").function("NAME") expands to process("/usr/bin/foo").function("NAME")
232 when placed in $prefix/share/systemtap/tapset/PATH/usr/bin/
234 This is intended to help write more reusable tapsets for userspace binaries.
236 - The implementation of "var <<< X" for each aggregate variable is now
237 specially compiled to compute only the script-requested @op(var) values,
238 not all potential ones. This speeds up the <<< operations.
240 - Systemtap now warns if script arguments given on the command line are unused,
241 instead of mentioned by the script with $n/@n.
243 - Netfilter tapsets now provide variables data_hex and data_str to display packet
244 contents in hexadecimal and ASCII respectively.
246 - Translator now accepts new @const() operator for convenient expressing
247 constants in tapset code, or guru-mode scripts. See stap(1) for details.
249 - New -T option allows the script to be terminated after a specified number
250 of seconds. This is a shortcut for adding the probe, timer {exit()}.
252 - New installcheck-parallel testsuite feature allows running the tests in
253 parallel in order to save time. See testsuite/README for details.
255 - New tapset functions set_user_string(), set_user_string_n(), set_user_long()
256 set_user_int(), set_user_short(), set_user_char() and set_user_pointer() to
257 write a value of specified type directly to a user space address.
259 - New tapset functions user_buffer_quoted(), user_buffer_quoted_error(),
260 kernel_buffer_quoted(), and kernel_buffer_quoted_error() to print a
261 buffer of an exact length. These functions can handle '\0' characters
264 - New statistics @variance() operator using the Welford's online algorithm
265 for per-cpu computation, and the Total Variance formula authored by
266 Niranjan Kamat and Arnab Nandi from the Ohio State University for the
267 cross-cpu aggregation.
269 - New command within interactive mode, sample. Allows you to search through
270 all included example scripts to load for further editing or running. Sample
271 and example scripts have been moved to /usr/share/systemtap/examples. A
272 symlink in the former location under $docdir links to it.
274 * What's new in version 3.0, 2016-03-27
276 - The new experimental "interactive" mode, specified by "stap -i",
277 drops you into a command-line prompt where you can build up a script,
278 run it, edit it, run it again, etc. Type "help" for a list of commands.
280 - New experimental --monitor[=INTERVAL] option similar to unix "top". This
281 allows users to see statistics about the running module(uptime, module name,
282 invoker uid, memory sizes, global variables, and the current probe list
283 along with their statistics).
285 An interface is also provided to allow control over the running
286 module(resetting global variables, sorting the list of probes, deactivating
287 and reactivating probes).
289 - The performance of associative arrays have been dramatically
290 improved, especially for densely filled tables and for multiple
291 indexes. The hash tables behind these arrays is now sized as a
292 function of the array maximum size with an optional MAPHASHBIAS
293 space/time tradeoff knob.
295 - Add macros @prints to print a scalar aggregate variable, @prints[1-9]
296 to print an array aggregate (of given index-arity), formatted similarly
297 to the automatic printing of written-only global variables.
300 probe oneshot { a <<< 1; b[tid()] <<< 2 }
301 probe end { @prints(a); @prints1(b) }
303 - Functions may now be overloaded during module runtime using the "next"
304 statement in script functions and STAP_NEXT macro for embedded-C functions.
305 They may also be overloaded by number of parameters during compile time.
309 function f() { if (condition) next; print("first function") }
310 function f() %{ STAP_NEXT; print("second function") %}
311 function f() { print("third function") }
313 For the given functions above, a functioncall f(), will execute the body of the
314 third function if condition evaluates to true and print "third function".
315 Note that the second function is unconditionally nexted.
317 Parameter overloading:
318 function g() { print("first function") }
319 function g(x) { print("second function") }
320 g() -> "first function"
321 g(1) -> "second function"
323 Note that runtime overloading does not occur in the above example as the number
324 of parameters of the functions differ. The use of a next statement inside a function
325 while no more overloads remain will trigger a runtime exception. The function
326 candidates are selected at compile time and is determined by the number of arguments
327 provided for the functioncall.
329 - Add Czech version of manual pages.
331 - The stap compile server will log the stap client's options that are passed
332 to the server. The options that get logged on the server will include the
333 script name or the -e script, depending on which is used by the client.
335 - Embedded-C functions and blocks may now access script level global variables
336 using the STAP_GLOBAL_GET_* and STAP_GLOBAL_SET_* macros.
338 To read or write the script global var, the /* pragma:read:var */ or
339 /* pragma:write:var */ marker must be placed in the embedded-C function or block.
340 The written type must match the type inferred at script level.
343 STAP_GLOBAL_SET_var(STAP_GLOBAL_GET_var()+1) -> increments script global var by 1
344 STAP_GLOBAL_SET_var("hello")
347 STAP_GLOBAL_GET_var(index-1, ..., index-n)
348 STAP_GLOBAL_SET_var(index-1, ..., index-n, new value)
350 - Probe point brace expansion is now supported to improve brevity in
351 specifying probe points. For example,
353 process.{function("a"), function("b").{call,return}}
354 => process.function("a"), process.function("b").call, process.function("b").return
356 process.{function("*").callees,plt}?
357 => process.function("*").callees?, process.plt?
359 {kernel,module("nfs")}.function("nfs*")!
360 => kernel.function("nfs*")!, module("nfs").function("nfs*")!
362 - Profiling timers at arbitrary frequencies are now provided and perf probes
363 now support a frequency field as an alternative to sampling counts.
365 probe timer.profile.freq.hz(N)
366 probe perf.type(N).config(M).hz(X)
368 The specified frequency is only accurate up to around 100hz. You may
369 need to provide a higher value to achieve the desired rate.
371 - Added support for private global variables and private functions. The scope
372 of these is limited to the tapset file they are defined in (PR19136).
374 - New tapset function string_quoted() to quote and \-escape general strings.
375 String $context variables that are pretty-printed are now processed with
376 such a quotation engine, falling back to a 0x%x (hex pointer) on errors.
378 - Functions get_mmap_args() and get_32mmap_args() got deprecated.
380 * What's new in version 2.9, 2015-10-08
382 - SystemTap now uses symbols from /proc/kallsyms when kernel debuginfo is not
385 - New --prologue-searching[=WHEN] option has been added to stap with '-P' being
386 its short counterpart. Using --prologue-searching=never turns prologue
387 searching deliberately off working around issue of int_arg() returning wrong
388 value when a 32-bit userspace binary having debug info is being probed with
389 active prologue searching (PR18649).
391 - The powerpc variant of nd_syscall.compat_sysctl got deprecated on favor of
392 nd_syscall.sysctl32. This aligns the nd_syscall to its respective syscall and
393 to ia64/s390/x86_64 variants too.
395 - New tapset function assert(expression, msg) has been added.
397 - Embedded-C functions may now use the new STAP_PRINTF(fmt, ...)
400 - New tapset functions fullname_struct_path and fullname_struct_nameidata
401 resolve full path names from internal kernel struct pointers.
403 - New tapset functions arch_bytes() and uarch_bytes() to obtain address size
404 for kernel and user space respectively.
406 - New tapset function switch_file() allows control over rotation
409 - The [nd_]syscall tapset got autodocumented. Related paragraph got added to PDF
410 and HTML tapset reference. Also a new tapset::syscall 3stap man page got added.
412 - Embedded-C functions with parameter arity-0 can now be marked with
413 the /* stable */ /* pure */ pragmas, if (roughly speaking) the
414 function is side-effect-free and idempotent. The translator may
415 execute these speculatively and have their results memoized. This
416 lets probes with multiple calls to such functions run faster.
418 Context variable ($foo) getter functions (in non-guru mode), and
419 numerous tapset functions are now marked as /* stable */ /* pure */.
420 Several example scripts have been modified to eschew explicit
423 - Callee probe points now support '.return' and '.call' suffix.
426 process("proc").function("foo").callee("bar").return
428 will fire upon returning from bar when called by foo.
430 process("proc").function("foo").callee("bar").call
432 will only fire for non-inlined callees.
434 - The following tapset variables and functions are deprecated in
436 - The '__int32_compat' library macro got deprecated in favor of
437 new '__compat_long' library macro.
438 - The 'uargs' convenience variable of the 'seccomp' syscall probe
439 got deprecated in favor of new 'uargs_uaddr' variable.
441 - SystemTap has reduced its memory consumption by using interned_strings (a
442 wrapper for boost::string_ref) in place of std::string instances. The change
443 is to reduce the number of duplicate strings created by replacing them with
444 interned_strings which act like pointers to existing strings.
446 For the implementation of interned_string, see stringtable.h
448 * What's new in version 2.8, 2015-06-17
450 - SystemTap has improved support for probing golang programs. Work has been
451 done to be able to handle DWARF information, reporting file names, line
452 numbers, and column numbers, and tolerance of odd characters in symbol names.
454 - The function::*, probe::* and new macro::* man pages cross-references the
455 enclosing tapset::* man page. For example:
457 function::pn(3stap) mentions tapset::pn(3stap) in the SEE ALSO section
459 - New stapref(1) man page provides a reference for the scripting language. The
460 stapref page contains an overview of the features available in the language,
461 such as keywords, data types, operators and more.
463 - The @task macro performs the very common @cast to a task_struct.
465 The embedded-C bodies of task_current() and pid2task() are now wrapped
466 by @task, which gives them a debuginfo type on the return value. With
467 autocast type propagation, this removes the need for any explicit @cast
470 Other places which take untyped task pointers as parameters, for
471 instance, now use @task as well to simplify their code.
473 - New namespace-aware tapset functions [task_]ns_*() and ia new option
474 --target-namespaces=PID to denote a target set of namespaces corresponding to
475 the PID's namespaces. The namespace-aware tapsets will return values
476 relative to the target namespaces if specified, or the stap process' namespaces.
478 - Netfilter probes now attempt to decode Spanning Tree Protocol packets
479 into local variables: probe netfilter.bridge.*, br_* variables,
480 stp_dump.stp sample script.
482 - Colorization of error string tokens is made more robust, especially
483 in presence of $N/@N substitution.
485 - The following tapset variables and functions are deprecated in
487 - The 'hostname_uaddr' variable in the syscall.setdomainname and
488 nd_syscall.setdomainname probe aliases have been deprecated in
489 favor of the new 'domainname_uaddr' variable.
490 - The 'fd' and 'fd_str' variables in the syscall.execveat and
491 nd_syscall.execveat probe aliases have been deprecated in favor of
492 the new 'dirfd' and 'dirfd_str' variables.
494 * What's new in version 2.7, 2015-02-18
496 - Some systemtap sample scripts are now identified with the "_best" keyword,
497 because they are generally useful or educational. They are now promoted
498 within the generated index files.
500 - Passing strings to and from functions has become faster due to optimization
501 (passing some strings by reference instead of by value/copy). It may
502 be disabled by using the unoptimize flag (-u).
504 To make embedded-C functions eligible for the same optimization, use the pragma
505 /* unmodified-fnargs */ to indicate that the function body will not modify
506 the function arguments. Remember to use MAXSTRINGLEN for string length,
507 rather than sizeof(string_arg) (which might now be a pointer).
509 - SystemTap now allows .function probes to be specified by their full function
510 name, file, and declaration line number. Use the .statement probe to probe a
511 specific line number.
513 - Tracepoint probes can now also be specified by the target subsystem. For
514 example, the following are all supported:
516 probe kernel.trace("sched:sched_switch") --> probe sched_switch found in the
518 probe kernel.trace("sched:*") --> probe all tracepoints in sched subsystem
520 As a result, tapset functions such as pn() will now return a different string
521 than before. To retain the previous behaviour, use '--compatible=2.6'.
523 - The following functions are deprecated in release 2.7:
524 - _adjtx_mode_str(), _statfs_f_type_str(), _waitid_opt_str(),
525 _internal_wait_opt_str(), and _epoll_events_str().
527 - New tapset functions [u]symfileline(), [u]symfile() and [u]symline() will
528 return a string containing the specified portion of the filename:linenumber
529 match from a given address.
531 Using these functions may result in large generated modules from stored
532 address->file:line information.
534 * What's new in version 2.6, 2014-09-05
536 - SystemTap now supports on-the-fly arming/disarming of certain probe types:
537 kprobes, uprobes, and timer.*s(NUM) probes. For example, this probe
539 probe kernel.function("vfs_read") if (i > 4) { ... }
541 will automatically register/unregister the associated kprobe on vfs_read
542 whenever the value of the condition changes (as some probe handler
543 modifies 'i'). This allows us to avoid probe overhead when we're not
544 interested. If the arming capability is not relevant/useful, nest the
545 condition in the normal probe handler:
547 probe kernel.function("vfs_read") { if (i > 4) { ... } }
549 - statement("*@file:NNN").nearest probes now available to let systemtap
550 translate probe to nearest probe-able line to one given if necessary
552 - process("PATH").library("PATH").plt("NAME").return probes are now supported.
554 - SystemTap now supports SDT probes with operands that refer to symbols.
556 - While in listing mode (-l/-L), probes printed are now more consistent
559 - Statement probes now support enumerated linenos to probe discontiguous
560 linenos using the form:
562 process.statement("foo@file.c:3,5-7,9")
564 - Statement counting is now suppressed in the generated c code for probes that
565 are non-recursive and loop-free. Statement counting can be turned back on in
566 unoptimize mode (-u).
568 - SystemTap now asserts that the PID provided for a process probe corresponds
569 to a running process.
571 - DWARF process probes can be bound to a specific process using the form:
573 process(PID).function("*")
575 - SystemTap now accepts additional scripts through the new -E SCRIPT option.
576 There still needs to be a main script specified through -e or file in order
577 to provide an additional script. This makes it feasible to have scripts in
578 the $HOME/.systemtap/rc file. For example:
580 -E 'probe begin, end, error { log("systemtap script " . pn()) }'
581 -E 'probe timer.s(30) { error ("timeout") }
583 The -E SCRIPT option can also be used in listing mode (-l/-L), such that
584 probe points for the additional scripts will not listed, but other parts of
585 the script are still available, such as macros or aliases.
587 - SystemTap now supports array slicing within foreach loop conditions, delete
588 statements and membership tests. Wildcards are represented by "*". Examples
589 of the expressions are:
591 foreach ([a,b,c] in val[*,2,*])
595 - Integer expressions which are derived from DWARF values, like context $vars,
596 @cast, and @var, will now carry that type information into subsequent reads.
597 Such expressions can now use "->" and "[]" operators, as can local variables
598 which were assigned such values.
600 foo = $param->foo; printf("x:%d y:%d\n", foo->x, foo->y)
601 printf("my value is %d\n", ($type == 42 ? $foo : $bar)->value)
602 printf("my parent pid is %d\n", task_parent(task_current())->tgid)
604 * What's new in version 2.5, 2014-04-30
606 - Systemtap now supports backtracing through its own, invoking module.
608 - Java probes now support backtracing using the print_java_backtrace()
609 and sprint_java_backtrace() functions.
611 - Statement probes (e.g. process.statement) are now faster to resolve,
612 more precise, and work better with inlined functions.
614 - New switches have been added to help inspect the contents of installed
617 stap --dump-functions --> list all library functions and their args
618 stap --dump-probe-aliases --> list all library probe aliases
620 - The heuristic algorithms used to search for function-prologue
621 endings were improved, to cover more optimization (or
622 lack-of-optimization, or incorrect-debuginfo) cases. These
623 heuristics are necessary to find $context parameters for some
624 function-call/entry probes. We recommend programs be built with
625 CFLAGS+=-grecord-gcc-switches to feed information to the heuristics.
627 - The stap --use-server option now more correctly supports address:port
628 type parametrization, for manual use in the absence of avahi.
630 - A new probe alias "oneshot" allows a single quick script fragment to run,
633 - The argv tapset now merges translate-time and run-time positional
634 arguments, so all of these work:
636 stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' hello world
638 stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' \
639 -G argv_1=hello -G argv_2=world
641 staprun hello.ko argv_1=hello argv_2=world
643 - SystemTap now falls back on the symbol table for probing
644 functions in processes if the debuginfo is not available.
646 - SystemTap now supports a %( guru_mode == 0 /* or 1 */ %)
647 conditional for making dual-use scripts.
649 - SystemTap now supports UEFI/SecureBoot systems, via
650 machine-owner-keys maintained by a trusted stap-server on the
651 network. (Key enrollment requires a one-time reboot and BIOS
653 https://sourceware.org/systemtap/wiki/SecureBoot
655 - SystemTap now reports more accurate and succinct errors on type
658 - Embedded-C functions may use STAP_RETURN(value) instead of the
659 more wordy STAP_RETVALUE assignment followed by a "goto out".
660 The macro supports numeric or string values as appropriate.
661 STAP_ERROR(...) is available to return with a (catchable) error.
663 - Some struct-sockaddr fields are now individually decoded for
664 socket-related syscalls:
665 probe syscall.connect { println (uaddr_af, ":", uaddr_ip) }
667 - The documentation for the SystemTap initscript service and the
668 SystemTap compile-server service have been completely converted from
669 README files to man pages (see systemtap(8) and stap-server(8)).
671 - SystemTap is now capable of inserting modules early during the boot
672 process on dracut-based systems. See the 'onboot' command in
673 systemtap(8) for more information.
675 - DWARF probes can now use the '.callee[s]' variants, which allow more
676 precise function probing. For example, the probe point
678 process("myproc").function("foo").callee("bar")
680 will fire upon entering bar() from foo(). A '.callees' probe will
681 instead place probes on all callees of foo().
682 Note that this also means that probe point wildcards should be used
683 with more care. For example, use signal.*.return rather than
684 signal.*.*, which would also match '.callees'. See stapprobes(3stap)
685 for more info. This feature requires at least GCC 4.7.
687 - A few new functions in the task_time tapsets, as well as a new tapset
688 function task_ancestry(), which prints out the parentage of a process.
690 - The kprocess.exec probe has been updated to use syscall.execve, which
691 allows access to the new process' arguments (through the new 'argstr'
692 or 'args' variables) as well as giving better support across kernel
693 versions. Note also that the 'filename' variable now holds the
694 filename (quoted), or the address (unquoted) if it couldn't be
697 - The [s]println() function can now be called without any arguments to
698 simply print a newline.
700 - Suggestions are now provided when markers could not be resolved. For
701 example, process("stap").mark("benchmart") will suggest 'benchmark'.
703 - SystemTap colors can now be turned off by simply setting
704 SYSTEMTAP_COLORS to be empty, rather than having to make it invalid.
706 - There is a new context tapset function, pnlabel(), which returns the
707 name of the label which fired.
709 - The following tapset variables and functions are deprecated in
711 - The 'clone_flags', 'stack_start', 'stack_size',
712 'parent_tid_uaddr', and 'child_tid_uaddr' variables in the
713 'syscall.fork' and 'nd_syscall.fork' probe aliases.
714 - The '_sendflags_str()' and '_recvflags_str()' functions have been
715 deprecated in favor of the new '_msg_flags_str()' function.
716 - The 'flags' and 'flags_str' variables in the 'syscall.accept' and
717 'nd_syscall.accept' probe alias.
718 - The 'first', 'second', and 'uptr_uaddr' variables in the
719 'syscall.compat_sys_shmctl', and 'nd_syscall.compat_sys_shmctl'
720 probe aliases have been deprecated in favor of the new 'shmid',
721 'cmd', and 'buf_uaddr' variables.
723 * What's new in version 2.4, 2013-11-06
725 - Better suggestions are given in many of the semantic errors in which
726 alternatives are provided. Additionally, suggestions are now provided
727 when plt and trace probes could not be resolved. For example,
728 kernel.trace("sched_siwtch") will suggest 'sched_switch'.
730 - SystemTap is now smarter about error reporting. Errors from the same
731 source are considered duplicates and suppressed. A message is
732 displayed on exit if any errors/warnings were suppressed.
734 - Statistics aggregate typed objects are now implemented locklessly,
735 if the translator finds that they are only ever read (using the
736 foreach / @count / etc. constructs) in a probe-begin/end/error.
738 - SystemTap now supports probing inside virtual machines using the
739 libvirt and unix schemes, e.g.
741 stap -ve 'probe timer.s(1) { printf("hello!\n") }' \
742 --remote=libvirt://MyVirtualMachine
744 Virtual machines managed by libvirt can be prepared using stapvirt.
745 See stapvirt(1) and the --remote option in stap(1) for more details.
747 - Systemtap now checks for and uses (when available) the .gnu_debugdata
748 section which contains a subset of debuginfo, useful for backtraces
751 - SystemTap map variables are now allocated with vmalloc() instead of
752 with kmalloc(), which should cause memory to be less fragmented.
754 - Although SystemTap itself requires elfutils 0.148+, staprun only
755 requires elfutils 0.142+, which could be useful with the
756 '--disable-translator' configure switch.
758 - Under FIPS mode (/proc/sys/crypto/fips_enabled=1), staprun will
759 refuse to load systemtap modules (since these are not normally
760 signed with the kernel's build-time keys). This protection may
761 be suppressed with the $STAP_FIPS_OVERRIDE environment variable.
763 - The stap-server client & server code now enable all SSL/TLS
764 ciphers rather than just the "export" subset.
766 - For systems with in-kernel utrace, 'process.end' and 'thread.end'
767 probes will hit before the target's parent process is notified of
768 the target's death. This matches the behavior of newer kernels
769 without in-kernel utrace.
771 * What's new in version 2.3, 2013-07-25
773 - More context-accessing functions throw systemtap exceptions upon a
774 failure, whereas in previous versions they might return non-error
775 sentinel values like "" or "<unknown>". Use try { } / catch { }
776 around these, or new wrapper functions such as user_string_{n_}quoted()
777 that internally absorb exceptions.
779 - java("org.my.MyApp") probes are now restricted to pre-existing jvm pid's with
780 a listing in jps -l output to avoid recursive calls
782 - The tapset [nd_]syscall.semop parameter tsops_uaddr is renamed sops_uaddr for
783 consistency with [nd_]syscall.semtimedop.
785 - The udp.stp tapset adds some ip-address/port variables.
787 - A new guru-mode-only tapset function raise() is available to send signals
790 - Support for the standard Posix ERE named character classes has been
791 added to the regexp engine, e.g. [:digit:], [:alpha:], ...
793 - A substantial internal overhaul of the regexp engine has resulted in
794 correct behaviour on further obscure edge cases. The regexp engine
795 now implements the ERE standard and correctly passes the testsuite
796 for the glibc regexp engine (minus portions corresponding to
797 unimplemented features -- i.e. subexpression capture and reuse).
799 - Alternative functions are now suggested when function probes could not be
800 resolved. For example, kernel.function("vfs_reads") will suggest vfs_read.
801 Other probes for which suggestions are made are module.function,
802 process.function, and process.library.function.
804 - Has life been a bit bland lately? Want to spice things up? Why not write a
805 few faulty probes and feast your eyes upon the myriad of colours adorning
806 your terminal as SystemTap softly whispers in your ear... 'parse error'.
807 Search for '--color' in 'man stap' for more info.
809 - The following tapset functions are deprecated in release 2.3:
810 'stap_NFS_CLIENT', '__getfh_inode', '_success_check',
811 '_sock_prot_num', '_sock_fam_num', '_sock_state_num',
812 '_sock_type_num', and '_sock_flags_num'.
814 * What's new in version 2.2.1, 2013-05-16
815 * What's new in version 2.2, 2013-05-14
817 - Experimental support has been added for probing Java methods using
818 Byteman 2.0 as a backend. Java method probes can target method entries,
819 returns, or specific statements in the method as specified by line number.
821 probe java("org.my.MyApp").class("^java.lang.Object").method("foo(int)")
824 See java/README for information on how to set up Java/Byteman
825 functionality. Set env STAPBM_VERBOSE=yes for more tracing.
827 - The stap -l output and pn() tapset function's return value may be slightly
828 different for complicated web of wildcarded/aliased probes.
830 - The dyninst backend has improved in several aspects:
832 - Setting custom values for global variables is now supported, both
833 with -G when compiling a script, and from the stapdyn command line
834 when loading a precompiled module.
836 - A high-performance shared-memory-based transport is used for
839 - A systemd service file and tmpfile have been added to allow
840 systemtap-server to be managed natively by systemd.
842 - Due to the removal of register_timer_hook in recent kernels, the
843 behaviour of timer.profile has been changed slightly. This probe is
844 now an alias which uses the old mechanism where possible, but falls
845 back to perf.sw.cpu_clock or another mechanism when the kernel timer
846 hook is not available.
848 To require the kernel timer hook mechanism in your script, use
849 timer.profile.tick instead of timer.profile.
851 - The following tapset variables are deprecated in release 2.2:
852 - The 'origin' variables in the 'generic.fop.llseek',
853 'generic.fop.llseek.return', and 'nfs.fop.llseek' probes. The
854 'origin' variable has been replaced by the 'whence' variable.
855 - The 'page_index' variable in the 'vfs.block_sync_page' and
856 'vfs.buffer_migrate_page' probe aliases.
857 - The 'write_from' and 'write_upto' variables in the
858 '_vfs.block_prepare_write' and '_vfs.block_prepare_write.return'
860 - The 'regs' variable in the 'syscall.sigaltstack',
861 'nd_syscall.sigaltstack', 'syscall.fork', and 'nd_syscall.fork'
863 - The 'first', 'second', 'third', and 'uptr_uaddr' variables in the
864 'syscall.compat_sys_shmat' and 'nd_syscall.compat_sys_shmat' probe
867 - The following tapset functions are deprecated in release 2.2:
868 'ppos_pos', '_dev_minor', and '_dev_major'
870 - The folowing tapset functions used to return error strings instead
871 of raising an error. The original behavior is deprecated in release
874 'ctime', 'probemod', 'modname'
876 * What's new in version 2.1, 2013-02-13
878 - EMACS and VIM editor modes for systemtap source files are included / updated.
880 - The translator now eliminates duplicate tapset files between its
881 preferred directory (as configured during the build with --prefix=/
882 or specified with the -I /path option), and files it may find under
883 $XDG_DATA_DIRS. This should eliminate a class of conflicts between
884 parallel system- and hand-built systemtap installations.
886 - The translator accepts a --suppress-time-limits option, which defeats
887 time-related constraints, to allows probe handlers to run for indefinite
888 periods. It requires the guru mode (-g) flag to work. Add the earlier
889 --suppress-handler-errors flag for a gung-ho "just-keep-going" attitude.
891 - Perf event probes may now be read on demand. The counter probe is
892 defined using the counter-name part:
893 probe perf.type(0).config(0).counter("NAME"). The counter is
894 read in a user space probe using @perf("NAME"), e.g.
895 process("PROCESS").statement("func@file") {stat <<< @perf("NAME")}
897 - Perf event probes may now be bound to a specific task using the
898 process-name part: probe perf.type(0).config(0).process("NAME") { }
899 If the probed process name is not specified, then it is inferred
900 from the -c CMD argument.
902 - Some error messages and warnings now refer to additional information
903 that is found in man pages. These are generally named
904 error::FOO or warning::BAR (in the 7stap man page section)
909 - The dyninst backend has improved in several aspects:
910 - The runtime now allows much more concurrency when probing multithreaded
911 processes, and will also follow probes across forks.
912 - Several new probe types are now supported, including timers, function
913 return, and process.begin/end and process.thread.begin/end.
914 - Semaphores for SDT probes are now set properly.
915 - Attaching to existing processes with -x PID now works.
917 - The foreach looping construct can now sort aggregate arrays by the user's
918 choice of aggregating function. Previously, @count was implied. e.g.:
919 foreach ([x,y] in array @sum +) { println(@sum(array[x,y])) }
921 - Proof of concept support for regular expression matching has been added:
922 if ("aqqqqqb" =~ "q*b") { ... }
923 if ("abc" !~ "q*b") { ... }
925 The eventual aim is to support roughly the same functionality as
926 the POSIX Extended Regular Expressions implemented by glibc.
927 Currently missing features include extraction of the matched string
928 and subexpressions, and named character classes ([:alpha:], [:digit:], &c).
930 Special thanks go to the re2c project, whose public domain code this
931 functionality has been based on. For more info on re2c, see:
932 http://sourceforge.net/projects/re2c/
934 - The folowing tapset variables are deprecated in release 2.1 and will
935 be removed in release 2.2:
936 - The 'send2queue' variable in the 'signal.send' probe.
937 - The 'oldset_addr' and 'regs' variables in the 'signal.handle' probe.
939 - The following tapset probes are deprecated in release 2.1 and will
940 be removed in release 2.2:
942 - signal.handle.return
944 * What's new in version 2.0, 2012-10-09
946 - Systemtap includes a new prototype backend, which uses Dyninst to instrument
947 a user's own processes at runtime. This backend does not use kernel modules,
948 and does not require root privileges, but is restricted with respect to the
949 kinds of probes and other constructs that a script may use.
951 Users from source should configure --with-dyninst and install a
952 fresh dyninst snapshot such as that in Fedora rawhide. It may be
953 necessary to disable conflicting selinux checks; systemtap will advise.
955 Select this new backend with the new stap option --runtime=dyninst
956 and a -c target process, along with normal options. (-x target
957 processes are not supported in this prototype version.) For example:
959 stap --runtime=dyninst -c 'stap -l begin' \
960 -e 'probe process.function("main") { println("hi from dyninst!") }'
962 - To aid diagnosis, when a kernel panic occurs systemtap now uses
963 the panic_notifier_list facility to dump a summary of its trace
964 buffers to the serial console.
966 - The systemtap preprocessor now has a simple macro facility as follows:
968 @define add(a,b) %( ((@a)+(@b)) %)
969 @define probegin(x) %(
975 @probegin( foo = @add(40, 2); print(foo) )
977 Macros defined in the user script and regular tapset .stp files are
978 local to the file. To get around this, the tapset library can define
979 globally visible 'library macros' inside .stpm files. (A .stpm file
980 must contain a series of @define directives and nothing else.)
982 The status of the feature is experimental; semantics of macroexpansion
983 may change (unlikely) or expand in the future.
985 - Systemtap probe aliases may be used with additional suffixes
986 attached. The suffixes are passed on to the underlying probe
987 point(s) as shown below:
989 probe foo = bar, baz { }
990 probe foo.subfoo.option("gronk") { }
991 // expands to: bar.subfoo.option("gronk"), baz.subfoo.option("gronk")
993 In practical terms, this allows us to specify additional options to
994 certain tapset probe aliases, by writing e.g.
995 probe syscall.open.return.maxactive(5) { ... }
997 - To support the possibility of separate kernel and dyninst backends,
998 the tapsets have been reorganized into separate folders according to
999 backend. Thus kernel-specific tapsets are located under linux/, the
1000 dyninst-specific ones under dyninst/
1002 - The backtrace/unwind tapsets have been expanded to allow random
1003 access to individual elements of the backtrace. (A caching mechanism
1004 ensures that the backtrace computation run at most once for each
1005 time a probe fires, regardless of how many times or what order the
1006 query functions are called in.) New tapset functions are:
1007 stack/ustack - return n'th element of backtrace
1008 callers/ucallers - return first n elements of backtrace
1009 print_syms/print_usyms - print full information on a list of symbols
1010 sprint_syms/sprint_usyms - as above, but return info as a string
1012 The following existing functions have been superseded by print_syms()
1013 et al.; new scripts are recommended to avoid using them:
1019 - The probefunc() tapset function is now myproc-unprivileged, and can
1020 now be used in unprivileged scripts for such things as profiling in
1021 userspace programs. For instance, try running
1022 systemtap.examples/general/para-callgraph.stp in unprivileged mode
1023 with a stapusr-permitted probe. The previous implementation of
1024 probefunc() is available with "stap --compatible=1.8".
1026 - Preprocessor conditional to vary code based on script privilege level:
1027 unprivileged -- %( systemtap_privilege == "stapusr" %? ... %)
1028 privileged -- %( systemtap_privilege != "stapusr" %? ... %)
1029 or, alternately %( systemtap_privilege == "stapsys"
1030 || systemtap_privilege == "stapdev" %? ... %)
1032 - To ease migration to the embedded-C locals syntax introduced in 1.8
1033 (namely, STAP_ARG_* and STAP_RETVALUE), the old syntax can now be
1034 re-enabled on a per-function basis using the /* unmangled */ pragma:
1036 function add_foo:long(a:long, b:long) %{ /* unmangled */
1037 THIS->__retvalue = THIS->a + STAP_ARG_b;
1040 Note that both the old and the new syntax may be used in an
1041 /* unmangled */ function. Functions not marked /* unmangled */
1042 can only use the new syntax.
1044 - Adjacent string literals are now glued together irrespective of
1045 intervening whitespace or comments:
1046 "foo " "bar" --> "foo bar"
1047 "foo " /* comment */ "bar" --> "foo bar"
1048 Previously, the first pair of literals would be glued correctly,
1049 while the second would cause a syntax error.
1051 * What's new in version 1.8, 2012-06-17
1053 - staprun accepts a -T timeout option to allow less frequent wake-ups
1054 to poll for low-throughput output from scripts.
1056 - When invoked by systemtap, the kbuild $PATH environment is sanitized
1057 (prefixed with /usr/bin:/bin:) in an attempt to exclude compilers
1058 other than the one the kernel was presumed built with.
1060 - Printf formats can now use "%#c" to escape non-printing characters.
1062 - Pretty-printed bitfields use integers and chars use escaped formatting
1065 - The systemtap compile-server and client now support IPv6 networks.
1066 - IPv6 addresses may now be specified on the --use-server option and will
1067 be displayed by --list-servers, if the avahi-daemon service is running and
1069 - Automatic server selection will automatically choose IPv4 or IPv6 servers
1070 according to the normal server selection criteria when avahi-daemon is
1071 running. One is not preferred over the other.
1072 - The compile-server will automatically listen on IPv6 addresses, if
1074 - To enable IPv6 in avahi-daemon, ensure that /etc/avahi/avahi-daemon.conf
1075 contains an active "use-ipv6=yes" line. After adding this line run
1076 "service avahi-daemon restart" to activate IPv6 support.
1077 - See man stap(1) for details on how to use IPv6 addresses with the
1078 --use-server option.
1080 - Support for DWARF4 .debug_types sections (for executables and shared
1081 libraries compiled with recent GCC's -gdwarf-4 / -fdebug-types-section).
1082 PR12997. SystemTap now requires elfutils 0.148+, full .debug_types support
1083 depends on elfutils 0.154+.
1085 - Systemtap modules are somewhat smaller & faster to compile. Their
1086 debuginfo is now suppressed by default; use -B CONFIG_DEBUG_INFO=y to
1089 - @var now an alternative language syntax for accessing DWARF variables
1090 in uprobe and kprobe handlers (process, kernel, module). @var("somevar")
1091 can be used where $somevar can be used. The @var syntax also makes it
1092 possible to access non-local, global compile unit (CU) variables by
1093 specifying the CU source file as follows @var("somevar@some/src/file.c").
1094 This will provide the target variable value of global "somevar" as defined
1095 in the source file "some/src/file.c". The @var syntax combines with all
1096 normal features of DWARF target variables like @defined(), @entry(),
1097 [N] array indexing, field access through ->, taking the address with
1098 the & prefix and shallow or deep pretty printing with a $ or $$ suffix.
1100 - Stap now has resource limit options:
1106 All resource limiting has been moved from the compile server to stap
1107 itself. When running the server as "stap-server", default resource
1108 limit values are specified in ~stap-server/.systemtap/rc.
1110 - Bug CVE-2012-0875 (kernel panic when processing malformed DWARF unwind data)
1113 - The systemtap compile-server now supports multiple concurrent connections.
1114 Specify the desired maximum number of concurrent connections with
1115 the new stap-server/stap-serverd --max-threads option. Specify a
1116 value of '0' to tell the server not to spawn any new threads (handle
1117 all connections serially in the main thread). The default value is
1118 the number of processor cores on the host.
1120 - The following tapset functions are deprecated in release 1.8 and will be
1121 removed in release 1.9:
1124 - SystemTap now mangles local variables to avoid collisions with C
1125 headers included by tapsets. This required a change in how
1126 embedded-C functions access local parameters and the return value slot.
1128 Instead of THIS->foo in an embedded-C function, please use the newly
1129 defined macro STAP_ARG_foo (substitute the actual name of the
1130 argument for 'foo'); instead of THIS->__retvalue, use the newly
1131 defined STAP_RETVALUE. All of the tapsets and test cases have been
1132 adapted to use this new notation.
1134 If you need to run code which uses the old THIS-> notation, run stap
1135 with the --compatible=1.7 option.
1137 - There is updated support for user-space probing against kernels >=
1138 3.5, which have no utrace but do have the newer inode-uprobes work
1139 by Srikar Dronamraju and colleagues. For kernels < 3.5, the
1140 following 3 sets of kernel patches would need to be backported to
1141 your kernel to use this preliminary user-space probing support:
1143 - inode-uprobes patches:
1144 - 2b144498350860b6ee9dc57ff27a93ad488de5dc
1145 - 7b2d81d48a2d8e37efb6ce7b4d5ef58822b30d89
1146 - a5f4374a9610fd7286c2164d4e680436727eff71
1147 - 04a3d984d32e47983770d314cdb4e4d8f38fccb7
1148 - 96379f60075c75b261328aa7830ef8aa158247ac
1149 - 3ff54efdfaace9e9b2b7c1959a865be6b91de96c
1150 - 35aa621b5ab9d08767f7bc8d209b696df281d715
1151 - 900771a483ef28915a48066d7895d8252315607a
1152 - e3343e6a2819ff5d0dfc4bb5c9fb7f9a4d04da73
1153 - exec tracepoint kernel patch:
1154 - 4ff16c25e2cc48cbe6956e356c38a25ac063a64d
1155 - task_work_add kernel patches:
1156 - e73f8959af0439d114847eab5a8a5ce48f1217c4
1157 - 4d1d61a6b203d957777d73fcebf19d90b038b5b2
1158 - 413cd3d9abeaef590e5ce00564f7a443165db238
1159 - dea649b8ac1861107c5d91e1a71121434fc64193
1160 - f23ca335462e3c84f13270b9e65f83936068ec2c
1162 * What's new in version 1.7, 2012-02-01
1164 - Map inserting and deleting is now significantly faster due to
1165 improved hashing and larger hash tables. The hashes are also
1166 now randomized to provide better protection against deliberate
1169 - Formatted printing is faster by compiling the formatting directives
1170 to C code rather than interpreting at run time.
1172 - Systemtap loads extra command line options from $SYSTEMTAP_DIR/rc
1173 ($HOME/.systemtap/rc by default) before the normal argc/argv. This
1174 may be useful to activate site options such as --use-server or
1175 --download-debuginfo or --modinfo.
1177 - The stap-server has seen many improvements, and is no longer considered
1180 - The stap-server service (initscript) now supports four new options:
1185 These allow the specification of macro definitions to be passed to stap
1186 by the server, the location of the log file, network port number and
1187 NSS certificate database location respectively. These options are also
1188 supported within individual server configuration files. See stap-server
1189 and initscript/README.stap-server for details. The stap-server is no
1190 longer activated by default.
1192 - process("PATH").[library("PATH")].function("NAME").exported probes are now
1193 supported to filter function() to only exported instances.
1195 - The translator supports a new --suppress-handler-errors option, which
1196 causes most runtime errors to be turned into quiet skipped probes. This
1197 also disables the MAXERRORS and MAXSKIPPED limits.
1199 - Translator warnings have been standardized and controlled by the -w / -W
1202 - The translator supports a new --modinfo NAME=VALUE option to emit additional
1203 MODULE_INFO(n,v) macros into the generated code.
1205 - There is no more fixed maximum number of VMA pages that will be tracked
1206 at runtime. This reduces memory use for those scripts that don't need any,
1207 or only limited target process VMA tracking and allows easier system
1208 wide probes inspecting shared library variables and/or user backtraces.
1209 stap will now silently ignore -DTASK_FINDER_VMA_ENTRY_ITEMS.
1211 - The tapset functions remote_id() and remote_uri() identify the member of a
1212 swarm of "stap --remote FOO --remote BAR baz.stp" concurrent executions.
1214 - Systemtap now supports a new privilege level and group, "stapsys", which
1215 is equivalent to the privilege afforded by membership in the group "stapdev",
1216 except that guru mode (-g) functionality may not be used. To support this, a
1217 new option, --privilege=[stapusr|stapsys|stapdev] has been added.
1218 --privilege=stapusr is equivalent to specifying the existing --unprivileged
1219 option. --privilege=stapdev is the default. See man stap(1) for details.
1221 - Scripts that use kernel.trace("...") probes compile much faster.
1223 - The systemtap module cache is cleaned less frequently, governed by the
1224 number of seconds in the $SYSTEMTAP_DIR/cache/cache_clean_interval_s file.
1226 - SDT can now define up to 12 arguments in a probe point.
1228 - Parse errors no longer generate a cascade of false errors. Instead, a
1229 parse error skips the rest of the current probe or function, and resumes
1230 at the next one. This should generate fewer and better messages.
1232 - Global array wrapping is now supported for both associative and statistics typed
1233 arrays using the '%' character to signify a wrapped array. For example,
1234 'global foo%[100]' would allow the array 'foo' to be wrapped if more than 100
1235 elements are inserted.
1237 - process("PATH").library("PATH").plt("NAME") probes are now supported.
1238 Wildcards are supported in the plt-name part, to refer to any function in the
1239 program linkage table which matches the glob pattern and the rest of the
1242 - A new option, --dump-probe-types, will dump a list of supported probe types.
1243 If --unprivileged is also specified, the list will be limited to probe types
1244 which are available to unprivileged users.
1246 - Systemtap can now automatically download the required debuginfo
1247 using abrt. The --download-debuginfo[=OPTION] can be used to
1248 control this feature. Possible values are: 'yes', 'no', 'ask',
1249 and a positive number representing the timeout desired. The
1250 default behavior is to not automatically download the debuginfo.
1252 - The translator has better support for probing C++ applications by
1253 better undertanding of compilation units, nested types, templates,
1254 as used in probe point and @cast constructs.
1256 - On 2.6.29+ kernels, systemtap can now probe kernel modules that
1257 arrive and/or depart during the run-time of a session. This allows
1258 probing of device driver initialization functions, which had formerly been
1261 - New tapset functions for cpu_clock and local_clock access were added.
1263 - There is some limited preliminary support for user-space probing
1264 against kernels such as linux-next, which have no utrace but do have
1265 the newer inode-uprobes work by Srikar Dronamraju and colleagues.
1267 - The following probe types are deprecated in release 1.7 and will be
1268 removed in release 1.8:
1269 kernel.function(number).inline
1270 module(string).function(number).inline
1271 process.function(number).inline
1272 process.library(string).function(number).inline
1273 process(string).function(number).inline
1274 process(string).library(string).function(number).inline
1276 - The systemtap-grapher is deprecated in release 1.7 and will be removed in
1279 - The task_backtrace() tapset function was deprecated in 1.6 and has been
1282 - MAXBACKTRACE did work in earlier releases, but has now been documented
1283 in the stap 1 manual page.
1285 - New tapset function probe_type(). Returns a short string describing
1286 the low level probe handler type for the current probe point.
1288 - Both unwind and symbol data is now only collected and emitted for
1289 scripts actually using backtracing or function/data symbols.
1290 Tapset functions are marked with /* pragma:symbols */ or
1291 /* pragma:unwind */ to indicate they need the specific data.
1293 - Kernel backtraces can now be generated for non-pt_regs probe context
1294 if the kernel support dump_trace(). This enables backtraces from
1295 certain timer probes and tracepoints.
1297 - ubacktrace() should now also work for some kernel probes on x86 which can
1298 use the dwarf unwinder to recover the user registers to provide
1299 more accurate user backtraces.
1301 - For s390x the systemtap runtime now properly splits kernel and user
1302 addresses (which are in separate address spaces on that architecture)
1303 which enable user space introspection.
1305 - ppc and s390x now supports user backtraces through the DWARF unwinder.
1307 - ppc now handles function descriptors as symbol names correctly.
1309 - arm support kernel backtraces through the DWARF unwinder.
1311 - arm now have a uprobes port which enables user probes. This still
1312 requires some kernel patches (user_regsets and tracehook support for
1315 - Starting in release 1.7, these old variables will be deprecated:
1316 - The 'pid' variable in the 'kprocess.release' probe has been
1317 deprecated in favor of the new 'released_pid' variable.
1318 - The 'args' variable in the
1319 '_sunrpc.clnt.create_client.rpc_new_client_inline' probe has been
1320 deprecated in favor of the new internal-only '__args' variable.
1322 - Experimental support for recent kernels without utrace has been
1323 added for the following probe types:
1326 process("PATH").begin
1328 process(PID).thread.begin
1329 process("PATH").thread.begin
1330 process.thread.begin
1334 process(PID).thread.end
1335 process("PATH").thread.end
1337 process(PID).syscall
1338 process("PATH").syscall
1340 process(PID).syscall.return
1341 process("PATH").syscall.return
1342 process.syscall.return
1344 - staprun disables kprobe-optimizations in recent kernels, as problems
1345 were found. (PR13193)
1347 * What's new in version 1.6, 2011-07-25
1349 - Security fixes for CVE-2011-2503: read instead of mmap to load modules,
1350 CVE-2011-2502: Don't allow path-based auth for uprobes
1352 - The systemtap compile-server no longer uses the -k option when calling the
1353 translator (stap). As a result, the server will now take advantage of the
1354 module cache when compiling the same script more than once. You may observe
1355 an improvement in the performance of the server in this situation.
1357 - The systemtap compile-server and client now each check the version of the
1358 other, allowing both to adapt when communicating with a down-level
1359 counterpart. As a result, all version of the client can communicate
1360 with all versions of the server and vice-versa. Client will prefer newer
1361 servers when selecting a server automatically.
1363 - SystemTap has improved support for the ARM architecture. The
1364 kread() and kwrite() operations for ARM were corrected allowing many
1365 of the tapsets probes and function to work properly on the ARM
1368 - Staprun can now rename the module to a unique name with the '-R' option before
1369 inserting it. Systemtap itself will also call staprun with '-R' by default.
1370 This allows the same module to be inserted more than once, without conflicting
1373 - Systemtap error messages now provide feedback when staprun or any other
1374 process fails to launch. This also specifically covers when the user
1375 doesn't have the proper permissions to launch staprun.
1377 - Systemtap will now map - to _ in module names. Previously,
1378 stap -L 'module("i2c-core").function("*")' would be empty. It now returns
1379 a list had stap -L 'module("i2c_core").function("*") been specified.
1381 - Systemtap now fills in missing process names to probe points, to
1382 avoid having to name them twice twice:
1383 % stap -e 'probe process("a.out").function("*") {}' -c 'a.out ...'
1384 Now the probed process name is inferred from the -c CMD argument.
1385 % stap -e 'probe process.function("*") {}' -c 'a.out ...'
1387 - stap -L 'process("PATH").syscall' will now list context variables
1389 - Depends on elfutils 0.142+.
1391 - Deprecated task_backtrace:string (task:long). This function will go
1392 away after 1.6. Please run your scripts with stap --check-version.
1394 * What's new in version 1.5, 2011-05-23
1396 - Security fixes for CVE-2011-1781, CVE-2011-1769: correct DW_OP_{mod,div}
1397 division-by-zero bug
1399 - The compile server and its related tools (stap-gen-ert, stap-authorize-cert,
1400 stap-sign-module) have been re-implemented in C++. Previously, these
1401 components were a mix of bash scripts and C code. These changes should be
1402 transparent to the end user with the exception of NSS certificate database
1403 password prompting (see below). The old implementation would prompt more
1404 than once for the same password in some situations.
1406 - eventcount.stp now allows for event counting in the format of
1407 'stap eventcount.stp process.end syscall.* ...', and also reports
1408 corresponding event tid's.
1410 - Systemtap checks that the build-id of the module being probed matches the
1411 build-id saved in the systemtap module. Invoking systemtap with
1412 -DSTP_NO_BUILDID_CHECK will bypass this build-id runtime verification. See
1413 man ld(1) for info on --build-id.
1415 - stapio will now report if a child process has an abnormal exit along with
1416 the associated status or signal.
1418 - Compiler optimization may sometimes result in systemtap not being able to
1419 access a user-space probe argument. Compiling the application with
1420 -DSTAP_SDT_ARG_CONSTRAINT=nr will force the argument to be an immediate or
1421 register value which should enable systemtap to access the argument.
1423 - GNU Gettext has now been intergrated with systemtap. Our translation
1424 page can be found at http://www.transifex.net/projects/p/systemtap/ .
1425 "make update-po" will generate the necessary files to use translated
1426 messages. Please refer to the po/README file for more info and
1427 please consider contributing to this I18N effort!
1429 - The new addr() function returns the probe's instruction pointer.
1431 - process("...").library("...") probes are now supported. Wildcards
1432 are supported in the library-name part, to refer to any shared
1433 library that is required by process-name, which matches the glob
1434 pattern and the rest of the probe point.
1436 - The "--remote USER@HOST" functionality can now be specified multiple times
1437 to fan out on multiple targets. If the targets have distinct kernel and
1438 architecture configurations, stap will automatically build the script
1439 appropriately for each one. This option is also no longer considered
1442 - The NSS certificate database generated for use by the compile server is now
1443 generated with no password. Previously, a random password was generated and
1444 used to access the database. This change should be transparent to most users.
1445 However, if you are prompted for a password when using systemtap, then
1446 running $libexecdir/stap-gen-cert should correct the problem.
1448 - The timestamp tapset includes jiffies() and HZ() for lightweight approximate
1451 - A powerful new command line option --version has been added.
1453 - process.mark now supports $$parms for reading probe parameters.
1455 - A new command line option, --use-server-on-error[=yes|no] is available
1456 for stap. It instructs stap to retry compilation of a script using a
1457 compile server if it fails on the local host. The default setting
1460 - The following deprecated tools have been removed:
1462 stap-authorize-server-cert
1463 stap-authorize-signing-cert
1464 stap-find-or-start-server
1466 Use the --use-server, --trust-server and --list-servers options of stap
1469 * What's new in version 1.4, 2011-01-17
1471 - Security fixes for CVE-2010-4170, CVE-2010-4171: staprun module
1474 - A new /* myproc-unprivileged */ marker is now available for embedded C
1475 code and and expressions. Like the /* unprivileged */ marker, it makes
1476 the code or expression available for use in unprivileged mode (see
1477 --unprivileged). However, it also automatically adds a call to
1478 assert_is_myproc() to the code or expression, thus, making it available
1479 to the unprivileged user only if the target of the current probe is within
1480 the user's own process.
1482 - The experimental "--remote USER@HOST" option will run pass 5 on a given
1483 ssh host, after building locally (or with --use-server) for that target.
1485 - Warning messages from the script may now be suppressed with the stap
1486 and/or staprun -w option. By default, duplicate warning messages are
1487 suppressed (up to a certain limit). With stap --vp 00002 and above,
1488 the duplicate elimination is defeated.
1490 - The print_ubacktrace and usym* functions attempt to print the full
1491 path of the user-space binaries' paths, instead of just the basename.
1492 The maximum saved path length is set by -DTASK_FINDER_VMA_ENTRY_PATHLEN,
1493 default 64. Warning messages are produced if unwinding fails due to
1494 a missing 'stap -d MODULE' option, providing preloaded unwind data.
1496 - The new tz_ctime() tapset function prints times in the local time zone.
1498 - More kernel tracepoints are accessible to the kernel.trace("...") mechanism,
1499 if kernel source trees or debuginfo are available. These formerly "hidden"
1500 tracepoints are those that are declared somewhere other than the usual
1501 include/linux/trace/ headers, such as xfs and kvm.
1503 - debuginfo-based process("...").function/.statement/.mark probes support
1504 wildcards in the process-name part, to refer to any executable files that
1505 match the glob pattern and the rest of the probe point.
1507 - The -t option now displays information per probe-point rather than a summary
1508 for each probe. It also now shows the derivation chain for each probe-point.
1510 - A rewrite of the sys/sdt.h header file provides zero-cost startup (few or
1511 no ELF relocations) for the debuginfo-less near-zero-cost runtime probes.
1512 Binaries compiled with earlier sdt.h versions remain supported. The
1513 stap -L (listing) option now lists parameters for sys/sdt.h markers.
1515 - The implementation of the integrated compile-server client has been
1517 o --use-server now accepts an argument representing a particular server and
1518 may be specified more than once.
1519 o --list-servers now accepts an expanded range of arguments.
1520 o a new --trust-servers option has been added to stap to replace several
1521 old certificate-management scripts.
1522 o The following tools are now deprecated and will be removed in release 1.5:
1524 stap-authorize-server-cert
1525 stap-authorize-signing-cert
1526 stap-find-or-start-server
1528 See man stap(1) for complete details.
1530 - The compile-server now returns the uprobes.ko to the client when it is
1531 required by the script being compiled. The integrated compile-server client
1532 now makes it available to be loaded by staprun. The old (deprecated)
1533 stap-client does not do this.
1535 - process probes with scripts as the target are recognized by stap and the
1536 interpreter would be selected for probing.
1538 - Starting in release 1.5, these old variables/functions will be deprecated
1539 and will only be available when the '--compatible=1.4' flag is used:
1541 - In the 'syscall.add_key' probe, the 'description_auddr' variable
1542 has been deprecated in favor of the new 'description_uaddr'
1544 - In the 'syscall.fgetxattr', 'syscall.fsetxattr',
1545 'syscall.getxattr', 'syscall.lgetxattr', and
1546 'syscall.lremovexattr' probes, the 'name2' variable has been
1547 deprecated in favor of the new 'name_str' variable.
1548 - In the 'nd_syscall.accept' probe the 'flag_str' variable
1549 has been deprecated in favor of the new 'flags_str' variable.
1550 - In the 'nd_syscall.dup' probe the 'old_fd' variable has been
1551 deprecated in favor of the new 'oldfd' variable.
1552 - In the 'nd_syscall.fgetxattr', 'nd_syscall.fremovexattr',
1553 'nd_syscall.fsetxattr', 'nd_syscall.getxattr', and
1554 'nd_syscall.lremovexattr' probes, the 'name2' variable has been
1555 deprecated in favor of the new 'name_str' variable.
1556 - The tapset alias 'nd_syscall.compat_pselect7a' was misnamed. It should
1557 have been 'nd_syscall.compat_pselect7' (without the trailing 'a').
1558 - The tapset function 'cpuid' is deprecated in favor of the better known
1560 - In the i386 'syscall.sigaltstack' probe, the 'ussp' variable has
1561 been deprecated in favor of the new 'uss_uaddr' variable.
1562 - In the ia64 'syscall.sigaltstack' probe, the 'ss_uaddr' and
1563 'oss_uaddr' variables have been deprecated in favor of the new
1564 'uss_uaddr' and 'uoss_uaddr' variables.
1565 - The powerpc tapset alias 'syscall.compat_sysctl' was deprecated
1566 and renamed 'syscall.sysctl32'.
1567 - In the x86_64 'syscall.sigaltstack' probe, the 'regs_uaddr'
1568 variable has been deprecated in favor of the new 'regs' variable.
1570 * What's new in version 1.3, 2010-07-21
1572 - The uprobes kernel module now has about half the overhead when probing
1573 NOPs, which is particularly relevant for sdt.h markers.
1575 - New stap option -G VAR=VALUE allows overriding global variables
1576 by passing the settings to staprun as module options.
1578 - The tapset alias 'syscall.compat_pselect7a' was misnamed. It should
1579 have been 'syscall.compat_pselect7' (without the trailing 'a').
1580 Starting in release 1.4, the old name will be deprecated and
1581 will only be available when the '--compatible=1.3' flag is used.
1583 - A new procfs parameter .umask(UMASK) which provides modification of
1584 file permissions using the proper umask value. Default file
1585 permissions for a read probe are 0400, 0200 for a write probe, and
1586 0600 for a file with a read and write probe.
1588 - It is now possible in some situations to use print_ubacktrace() to
1589 get a user space stack trace from a kernel probe point. e.g. for
1590 user backtraces when there is a pagefault:
1591 $ stap -d /bin/sort --ldd -e 'probe vm.pagefault {
1592 if (pid() == target()) {
1593 printf("pagefault @0x%x\n", address); print_ubacktrace();
1596 pagefault @0x7fea0595fa70
1597 0x000000384f07f958 : __GI_strcmp+0x12b8/0x1440 [libc-2.12.so]
1598 0x000000384f02824e : __gconv_lookup_cache+0xee/0x5a0 [libc-2.12.so]
1599 0x000000384f021092 : __gconv_find_transform+0x92/0x2cf [libc-2.12.so]
1600 0x000000384f094896 : __wcsmbs_load_conv+0x106/0x2b0 [libc-2.12.so]
1601 0x000000384f08bd90 : mbrtowc+0x1b0/0x1c0 [libc-2.12.so]
1602 0x0000000000404199 : ismbblank+0x39/0x90 [sort]
1603 0x0000000000404a4f : inittables_mb+0xef/0x290 [sort]
1604 0x0000000000406934 : main+0x174/0x2510 [sort]
1605 0x000000384f01ec5d : __libc_start_main+0xfd/0x1d0 [libc-2.12.so]
1606 0x0000000000402509 : _start+0x29/0x2c [sort]
1609 - New tapset functions to get a string representation of a stack trace:
1610 sprint_[u]backtrace() and sprint_[u]stack().
1612 - New tapset function to get the module (shared library) name for a
1613 user space address umodname:string(long). The module name will now
1614 also be in the output of usymdata() and in backtrace addresses even
1615 when they were not given with -d at the command line.
1617 - Kernel backtraces are now much faster (replaced a linear search
1618 with a binary search).
1620 - A new integrated compile-server client is now available as part of stap.
1622 o 'stap --use-server ...' is equivalent to 'stap-client ...'
1623 o 'stap --list-servers' is equivalent to 'stap-find-servers'
1624 o 'stap --list-servers=online' is equivalent to 'stap-find-servers --all'
1625 o stap-client and its related tools will soon be deprecated.
1626 o the nss-devel and avahi-devel packages are required for building stap with
1627 the integrated client (checked during configuration).
1628 o nss and avahi are required to run the integrated client.
1630 - A new operator @entry is available for automatically saving an expression
1631 at entry time for use in a .return probe.
1632 probe foo.return { println(get_cycles() - @entry(get_cycles())) }
1634 - Probe $target variables and @cast() can now use a suffix to print complex
1635 data types as strings. Use a single '$' for a shallow view, or '$$' for a
1636 deeper view that includes nested types. For example, with fs_struct:
1637 $fs$ : "{.users=%i, .lock={...}, .umask=%i,
1638 .in_exec=%i, .root={...}, .pwd={...}}"
1639 $fs$$ : "{.users=%i, .lock={.raw_lock={.lock=%u}}, .umask=%i, .in_exec=%i,
1640 .root={.mnt=%p, .dentry=%p}, .pwd={.mnt=%p, .dentry=%p}}"
1642 - The <sys/sdt.h> user-space markers no longer default to an implicit
1643 MARKER_NAME_ENABLED() semaphore check for each marker. To check for
1644 enabled markers use a .d declaration file, then:
1645 if (MARKER_NAME_ENABLED()) MARKER_NAME()
1647 - Hyphenated <sys/sdt.h> marker names such as process(...).mark("foo-bar")
1648 are now accepted in scripts. They are mapped to the double-underscore
1651 - More robust <sys/sdt.h> user-space markers support is included. For
1652 some platforms (x86*, ppc*), this can let systemtap probe the markers
1653 without debuginfo. This implementation also supports preserving
1654 the "provider" name associated with a marker:
1655 probe process("foo").provider("bar").mark("baz") to match
1656 STAP_PROBE<n>(bar, baz <...>)
1657 (Compile with -DSTAP_SDT_V1 to revert to the previous implementation.
1658 Systemtap supports pre-existing or new binaries using them.)
1660 - Embedded-C may be used within expressions as values, when in guru mode:
1661 num = %{ LINUX_VERSION_CODE %} // int64_t
1662 name = %{ /* string */ THIS_MODULE->name %} // const char*
1663 printf ("%s %x\n", name, num)
1664 The usual /* pure */, /* unprivileged */, and /* guru */ markers may be used
1665 as with embedded-C functions.
1667 - By default the systemtap-runtime RPM builds now include a shared
1668 library, staplog.so, that allows crash to extract systemtap data from
1671 - Iterating with "foreach" can now explicitly save the value for the loop.
1672 foreach(v = [i,j] in array)
1673 printf("array[%d,%s] = %d\n", i, j, v /* array[i,j] */)
1675 - The new "--ldd" option automatically adds any additional shared
1676 libraries needed by probed or -d-listed userspace binaries to the -d
1677 list, to enable symbolic backtracing through them. Similarly, the
1678 new "--all-modules" option automatically adds any currently loaded
1679 kernel modules (listed in /proc/modules) to the -d list.
1681 - A new family of set_kernel_* functions make it easier for gurus to write
1682 new values at arbitrary memory addresses.
1684 - Probe wildcards can now use '**' to cross the '.' separator.
1685 $ stap -l 'sys**open'
1689 - Backward compatibility flags (--compatible=VERSION, and matching
1690 script preprocessing predicate %( systemtap_v CMP "version" %)
1691 and a deprecation policy are being introduced, in case future
1692 tapset/language changes break valid scripts.
1694 * What's new in version 1.2, 2010-03-22
1696 - Prototype support for "perf events", where the kernel supports the
1697 2.6.33 in-kernel API. Probe points may refer to low-level
1698 perf_event_attr type/config numbers, or to a number of aliases
1699 defined in the new perf.stp tapset:
1700 probe perf.sw.cpu_clock, perf.type(0).config(4) { }
1702 - Type-casting can now use multiple headers to resolve codependencies.
1703 @cast(task, "task_struct",
1704 "kernel<linux/sched.h><linux/fs_struct.h>")->fs->umask
1706 - Tapset-related man pages have been renamed. 'man -k 3stap' should show
1707 the installed list, which due to prefixing should no longer collide over
1708 ordinary system functions.
1710 - User space marker arguments no longer use volatile if the version of gcc,
1711 which must be at least 4.5.0, supports richer DWARF debuginfo. Use cflags
1712 -DSTAP_SDT_VOLATILE=volatile or -DSTAP_SDT_VOLATILE= when building
1713 the sys/sdt.h application to override this one way or another.
1715 - A new construct for error handling is available. It is similar to c++
1716 exception catching, using try and catch as new keywords. Within a handler
1717 or function, the following is valid and may be nested:
1718 try { /* arbitrary statements */ }
1719 catch (er) { /* e.g. println("caught error ", er) */ }
1721 - A new command line flag '-W' forces systemtap to abort translation of
1722 a script if any warnings are produced. It is similar to gcc's -Werror.
1723 (If '-w' is also supplied to suppress warnings, it wins.)
1725 - A new predicate @defined is available for testing whether a
1726 particular $variable/expression is resolvable at translate time:
1727 probe foo { if (@defined($bar)) log ("$bar is available here") }
1729 - Adjacent string literals are glued together, making this
1731 probe process("/usr" @1 "/bin").function("*") { ... }
1733 - In order to limit potential impact from future security problems,
1734 the stap-server process does not permit its being launched as root.
1736 - On recent kernels, for some architectures/configurations, hardware
1737 breakpoint probes are supported. The probe point syntax is:
1739 probe kernel.data(ADDRESS).write
1740 probe kernel.data(ADDRESS).length(LEN).write
1741 probe kernel.data("SYMBOL_NAME").write
1743 * What's new in version 1.1, 2010-01-15
1745 - New tracepoint based tapset for memory subsystem.
1747 - The loading of signed modules by staprun is no longer allowed for
1748 ordinary, unprivileged users. This means that only root, members of
1749 the group 'stapdev' and members of the group 'stapusr' can load
1750 systemtap modules using staprun, stap or stap-client. The minimum
1751 privilege required to run arbitrary --unprivileged scripts is now
1752 'stapusr' membership.
1754 - The stap-server initscript is available. This initscript allows you
1755 to start systemtap compile servers as a system service and to manage
1756 these servers as a group or individually. The stap-server initscript
1757 is installed by the systemtap-server rpm. The build directory for
1758 the uprobes module (/usr/share/systemtap/runtime/uprobes) is made
1759 writable by the 'stap-server' group. All of the files generated when
1760 building the uprobes module, including the digital signature, are
1761 also writable by members of stap-server.
1763 See initscript/README.stap-server for details.
1765 - Some of the compile server client, server and certificate management
1766 tools have been moved from $bindir to $libexecdir/systemtap.
1767 You should use the new stap-server script or the stap-server initscript
1768 for server management where possible. The stap-server script provides the same
1769 functionality as the stap-server initscript except that the servers are
1770 run by the invoking user by default as opposed to servers started by the
1771 stap-server initscript which are run by the user stap-server
1772 by default. See stap-server(8) for more information.
1774 You may continue to use these tools by adding $libexecdir/systemtap to
1775 your path. You would need to do this, for example, if you are not root,
1776 you want to start a compile server and you are not running systemtap from a
1777 private installation. In this case you still need to use stap-start-server.
1779 - Any diagnostic output line that starts with "ERROR", as in
1780 error("foo"), will promote a "Pass 5: run failed", and the return
1783 - Systemtap now warns about global variables being referenced from other
1784 script files. This aims to protect against unintended local-vs-global
1785 namespace collisions such as:
1787 % cat some_tapset.stp
1788 probe baz.one = bar { foo = $foo; bar = $bar }
1789 % cat end_user_script.stp
1790 global foo # intended to be private variable
1791 probe timer.s(1) { foo ++ }
1792 probe baz.* { println(foo, pp()) }
1793 % stap end_user_script.stp
1794 WARNING: cross-file global variable reference to foo from some_tapset.stp
1796 - Preprocessor conditional for kernel configuration testing:
1797 %( CONFIG_foo == "y" %? ... %)
1799 - ftrace(msg:string) tapset function to send strings to the system-wide
1800 ftrace ring-buffer (if any).
1802 - Better support for richer DWARF debuginfo output from GCC 4.5
1803 (variable tracking assignments). Kernel modules are now always resolved
1804 against all their dependencies to find any info referring to missing
1805 symbols. DW_AT_const_value is now supported when no DW_AT_location
1808 * What's new in verson 1.0, 2009-09-22
1810 - process().mark() probes now use an enabling semaphore to reduce the
1811 computation overhead of dormant probes.
1813 - The function spec for dwarf probes now supports C++ scopes, so you can
1814 limit the probes to specific namespaces or classes. Multiple scopes
1815 can be specified, and they will be matched progressively outward.
1816 probe process("foo").function("std::vector<*>::*") { }
1817 probe process("foo").function("::global_function") { }
1819 - It is now possible to cross-compile systemtap scripts for foreign
1820 architectures, using the new '-a ARCH' and '-B OPT=VALUE' flags.
1821 For example, put arm-linux-gcc etc. into your $PATH, and point
1822 systemtap at the target kernel build tree with:
1823 stap -a arm -B CROSS_COMPILE=arm-linux- -r /build/tree [...]
1824 The -B option is passed to kbuild make. -r identifies the already
1825 configured/built kernel tree and -a its architecture (kbuild ARCH=...).
1826 Systemtap will infer -p4.
1828 - Cross compilation using the systemtap client and server
1829 - stap-start-server now accepts the -r, -R, -I, -B and -a options in
1830 order to start a cross compiling server. The server will correctly
1831 advertise itself with respect to the kernel release and architecture
1832 that it compiles for.
1833 - When specified on stap-client, the -r and -a options will be
1834 considered when searching for a suitable server.
1836 - When using the systemtap client and server udp port 5353 must be open
1837 in your firewall in order for the client to find servers using
1838 avahi-browse. Also the systemtap server will choose a random port in
1839 the range 1024-63999 for accepting ssl connections.
1841 - Support for unprivileged users:
1842 ***********************************************************************
1843 * WARNING!!!!!!!!!! *
1844 * This feature is EXPERIMENTAL at this time and should be used with *
1845 * care. This feature allows systemtap kernel modules to be loaded by *
1846 * unprivileged users. The user interface and restrictions will change *
1847 * as this feature evolves. *
1848 ***********************************************************************
1849 - Systemtap modules generated from scripts which use a restricted
1850 subset of the features available may be loaded by staprun for
1851 unprivileged users. Previously, staprun would load modules only for
1852 root or for members of the groups stapdev and stapusr.
1853 - Using the --unprivileged option on stap enables translation-time
1854 checking for use by unprivileged users (see restrictions below).
1855 - All modules deemed suitable for use by unprivileged users will be
1856 signed by the systemtap server when --unprivileged is specified on
1857 stap-client. See module signing in release 0.9.8 and stap-server in
1859 - Modules signed by trusted signers (servers) and verified by staprun
1860 will be loaded by staprun regardless of the user's privilege level.
1861 - The system administrator asserts the trustworthiness of a signer
1862 (server) by running stap-authorize-signing-cert <cert-file> as root,
1863 where the <cert-file> can be found in
1864 ~<user>/.systemtap/ssl/server/stap.cert for servers started by
1865 ordinary users and in $sysconfdir/systemtap/ssl/server/stap.cert for
1866 servers started by root.
1867 - Restrictions are intentionally strict at this time and may be
1868 relaxed in the future:
1869 - probe points are restricted to:
1870 begin, begin(n), end, end(n), error, error(n), never,
1871 timer.{jiffies,s,sec,ms,msec,us,usec,ns,nsec}(n)*, timer.hz(n),
1872 process.* (for processes owned by the user).
1873 - use of embedded C code is not allowed.
1874 - use of tapset functions is restricted.
1875 - some tapset functions may not be used at all. A message will be
1876 generated at module compilation time.
1877 - some actions by allowed tapset functions may only be performed
1878 in the context of the user's own process. A runtime fault will
1879 occur in these situations, for example, direct memory access.
1880 - The is_myproc() tapset function has been provided so that
1881 tapset writers for unprivileged users can check that the
1882 context is of the users own process before attempting these
1884 - accessing the kernel memory space is not allowed.
1885 - The following command line options may not be used by stap-client
1887 - The following environment variables are ignored by stap-client:
1888 SYSTEMTAP_RUNTIME, SYSTEMTAP_TAPSET, SYSTEMTAP_DEBUGINFO_PATH
1889 - nss and nss-tools are required to use this feature.
1891 - Support output file switching by SIGUSR2. Users can command running
1892 stapio to switch output file by sending SIGUSR2.
1894 - Memory consumption for scripts involving many uprobes has been
1895 dramatically reduced.
1897 - The preprocessor now supports || and && in the conditions.
1898 e.g. %( arch == "x86_64" || arch == "ia64" %: ... %)
1900 - The systemtap notion of "architecture" now matches the kernel's, rather
1901 than that of "uname -m". This means that 32-bit i386 family are all
1902 known as "i386" rather than "i386" or "i686"; "ppc64" as "powerpc";
1903 "s390x" as "s390", and so on. This is consistent between the new
1904 "-a ARCH" flag and the script-level %( arch ... %) conditional.
1906 - It is now possible to define multiple probe aliases with the same name.
1907 A probe will expand to all matching aliases.
1910 probe foo { } # expands twice, once to bar and once to baz
1912 - A new experimental transport mechanism, using ftrace's ring_buffer,
1913 has been added. This may become the default transport mechanism in
1914 future versions of systemtap. To test this new transport mechanism,
1915 define 'STP_USE_RING_BUFFER'.
1917 - Support for recognizing DW_OP_{stack,implicit}_value DWARF expressions
1918 as emitted by GCC 4.5.
1920 * What's new in version 0.9.9, 2009-08-04
1922 - Systemwide kernel .function.return (kretprobe) maxactive defaults may
1923 be overridden with the -DKRETACTIVE=nnn parameter.
1925 - Translation pass 2 is significantly faster by avoiding unnecessary
1926 searching through a kernel build/module directory tree.
1928 - When compiled against elfutils 0.142 systemtap now handles the new
1929 DW_OP_call_frame_CFA generated by by GCC.
1931 - uprobes and ustack() are more robust when used on applications that
1932 depend on prelinked/separate debuginfo shared libraries.
1934 - User space PROBE marks are not always found with or without separate
1935 debuginfo. The .probes section itself is now always put in the main
1936 elf file and marked as allocated. When building pic code the section
1937 is marked writable. The selinux memory check problems seen with
1938 programs using STAP_PROBES is fixed.
1940 - statement() probes can now override "address not at start of statement"
1941 errors in guru mode. They also provide alternative addresses to use
1944 - The stapgraph application can generate graphs of data and events
1945 emitted by systemtap scripts in real time. Run "stapgraph
1946 testsuite/systemtap.examples/general/grapher.stp" for an example of
1947 graphing the system load average and keyboard events.
1949 - Dwarf probes now show parameters and local variables in the verbose
1952 - Symbol aliases are now resolved to their canonical dwarf names. For
1953 example, probing "malloc" in libc resolves to "__libc_malloc".
1955 - The syntax for dereferencing $target variables and @cast() gained new
1957 - Array indexes can now be arbitrary numeric expressions.
1958 - Array subscripts are now supported on pointer types.
1959 - An '&' operator before a @cast or $target returns the address of the
1960 final component, especially useful for nested structures.
1962 - For reading all probe variables, kernel.mark now supports $$vars and
1963 $$parms, and process.syscall now supports $$vars.
1965 - The SNMP tapset provides probes and functions for many network
1966 statistics. See stapprobes.snmp(3stap) for more details.
1968 - The dentry tapset provides functions to map kernel VFS directory entries
1969 to file or full path names: d_path(), d_name() and reverse_path_walk().
1971 - SystemTap now has userspace markers in its own binaries, and the stap
1972 tapset provides the available probepoints and local variables.
1974 - Miscellaneous new tapset functions:
1975 - pgrp() returns the process group ID of the current process
1976 - str_replace() performs string replacement
1978 * What's new in version 0.9.8, 2009-06-11
1980 - Miscellaneous new tapset functions:
1981 - sid() returns the session ID of the current process
1982 - stringat() indexes a single character from a string.
1984 - Using %M in print formats for hex dumps can now print entire buffers,
1985 instead of just small numbers.
1987 - Dwarfless syscalls: The nd_syscalls tapset is now available to probe
1988 system calls without requiring kernel debugging information. All of
1989 the same probepoints in the normal syscalls tapset are available with
1990 an "nd_" prefix, e.g. syscall.open becomes nd_syscall.open. Most
1991 syscall arguments are also available by name in nd_syscalls.
1993 - Module signing: If the appropriate nss libraries are available on your
1994 system, stap-server will sign each compiled module using a self-generated
1995 certificate. This is the first step toward extending authority to
1996 load certain modules to unprivileged users. For now, if the system
1997 administrator adds a certificate to a database of trusted signers
1998 (stap-authorize-signing-cert), modules signed using that certificate
1999 will be verified by staprun against tampering. Otherwise, you should
2000 notice no difference in the operation of stap or staprun.
2002 * What's new in version 0.9.7, 2009-04-23
2004 - @cast can now determine its type information using an explicit header
2005 specification. For example:
2006 @cast(tv, "timeval", "<sys/time.h>")->tv_sec
2007 @cast(task, "task_struct", "kernel<linux/sched.h>")->tgid
2009 - The overlapping process.* tapsets are now separated. Those probe points
2010 documented in stapprobes(3stap) remain the same. Those that were formerly
2011 in stapprobes.process(3stap) have been renamed to kprocess, to reflect
2012 their kernel perspective on processes.
2014 - The --skip-badvars option now also suppresses run-time error
2015 messages that would otherwise result from erroneous memory accesses.
2016 Such accesses can originate from $context expressions fueled by
2017 erroneous debug data, or by kernel_{long,string,...}() tapset calls.
2019 - New probes kprobe.function(FUNCTION) and kprobe.function(FUNCTION).return
2020 for dwarfless probing. These postpone function address resolution to
2021 run-time and use the kprobe symbol-resolution mechanism.
2022 Probing of absolute statements can be done using the
2023 kprobe.statement(ADDRESS).absolute construct.
2025 - EXPERIMENTAL support for user process unwinding. A new collection of
2026 tapset functions have been added to handle user space backtraces from
2027 probe points that support them (currently process and timer probes -
2028 for timer probes test whether or not in user space first with the
2029 already existing user_mode() function). The new tapset functions are:
2030 uaddr - User space address of current running task.
2031 usymname - Return the symbol of an address in the current task.
2032 usymdata - Return the symbol and module offset of an address.
2033 print_ustack - Print out stack for the current task from string.
2034 print_ubacktrace - Print stack back trace for current task.
2035 ubacktrace - Hex backtrace of current task stack.
2036 Please read http://sourceware.org/ml/systemtap/2009-q2/msg00364.html
2037 on the current restrictions and possible changes in the future and
2038 give feedback if you want to influence future developments.
2040 * What's new in version 0.9.5, 2009-03-27
2042 - New probes process().insn and process().insn.block that allows
2043 inspection of the process after each instruction or block of
2044 instructions executed. So to count the total number of instructions
2045 a process executes during a run do something like:
2046 $ stap -e 'global steps; probe process("/bin/ls").insn {steps++}
2047 probe end {printf("Total instructions: %d\n", steps);}' \
2049 This feature can slow down execution of a process somewhat.
2051 - Systemtap probes and function man pages extracted from the tapsets
2052 are now available under 3stap. To show the page for probe vm.pagefault
2053 or the stap function pexecname do:
2054 $ man 3stap vm.pagefault
2055 $ man 3stap pexecname
2057 - Kernel tracepoints are now supported for probing predefined kernel
2058 events without any debuginfo. Tracepoints incur less overhead than
2059 kprobes, and context parameters are available with full type
2060 information. Any kernel 2.6.28 and later should have defined
2061 tracepoints. Try the following to see what's available:
2062 $ stap -L 'kernel.trace("*")'
2064 - Typecasting with @cast now supports modules search paths, which is
2065 useful in case there are multiple places where the type definition
2066 may be found. For example:
2067 @cast(sdev, "scsi_device", "kernel:scsi_mod")->sdev_state
2069 - On-file flight recorder is supported. It allows stap to record huge
2070 trace log on the disk and to run in background.
2071 Passing -F option with -o option runs stap in background mode. In this
2072 mode, staprun is detached from console, and stap itself shows staprun's
2074 Specifying the max size and the max number of log files are also available
2075 by passing -S option. This option has one or two arguments seperated by
2076 a comma. The first argument is the max size of a log file in MB. If the
2077 size of a log file exceeds it, stap switches to the next log file
2078 automatically. The second is how many files are kept on the disk. If the
2079 number of log files exceeds it, the oldest log file is removed
2080 automatically. The second argument can be omitted.
2082 For example, this will record output on log files each of them is smaller
2083 than 1024MB and keep last 3 logs, in background.
2084 % stap -F -o /tmp/staplog -S 1024,3 script.stp
2086 - In guru mode (-g), the kernel probing blacklist is disabled, leaving
2087 only a subset - the kernel's own internal kprobe blacklist - to attempt
2088 to filter out areas unsafe to probe. The differences may be enough to
2089 probe more interrupt handlers.
2091 - Variables unavailable in current context may be skipped by setting a
2092 session level flag with command line option --skip-badvars now available.
2093 This replaces any dwarf $variable expressions that could not be resolved
2094 with literal numeric zeros, along with a warning message.
2096 - Both kernel markers and kernel tracepoint support argument listing
2097 through stap -L 'kernel.mark("*")' or stap -L 'kernel.trace("*")'
2099 - Users can use -DINTERRUPTIBLE=0 to prevent interrupt reentrancy in
2100 their script, at the cost of a bit more overhead to toggle the
2103 - Added reentrancy debugging. If stap is run with the arguments
2104 "-t -DDEBUG_REENTRANCY", additional warnings will be printed for
2105 every reentrancy event, including the probe points of the
2106 resident and interloper probes.
2108 - Default to --disable-pie for configure.
2109 Use --enable-pie to turn it back on.
2111 - Improved sdt.h compatibility and test suite for static dtrace
2112 compatible user space markers.
2114 - Some architectures now use syscall wrappers (HAVE_SYSCALL_WRAPPERS).
2115 The syscall tapset has been enhanced to take care of the syscall
2116 wrappers in this release.
2118 - Security fix for CVE-2009-0784: stapusr module-path checking race.
2120 * What's new in version 0.9, 2009-02-19
2122 - Typecasting is now supported using the @cast operator. A script can
2123 define a pointer type for a "long" value, and then access type members
2124 using the same syntax as with $target variables. For example, this will
2125 retrieve the parent pid from a kernel task_struct:
2126 @cast(pointer, "task_struct", "kernel")->parent->pid
2128 - process().mark() probes are now possible to trace static user space
2129 markers put in programs with the STAP_PROBE macro using the new
2130 sys/sdt.h include file. This also provides dtrace compatible markers
2131 through DTRACE_PROBE and an associated python 'dtrace' script that
2132 can be used in builds based on dtrace that need dtrace -h or -G
2135 - For those that really want to run stap from the build tree there is
2136 now the 'run-stap' script in the top-level build directory that sets
2137 up the SYSTEMTAP_TAPSET, SYSTEMTAP_RUNTIME, SYSTEMTAP_STAPRUN, and
2138 SYSTEMTAP_STAPIO environment variables (installing systemtap, in a
2139 local prefix, is still recommended for common use).
2141 - Systemtap now comes with a new Beginners Guide that walks the user
2142 through their first steps setting up stap, understanding how it all
2143 works, introduces some useful scripts and describes some common
2144 pitfalls. It isn't created by default since it needs a Publican
2145 setup, but full build instructions can be found in the wiki:
2146 http://sourceware.org/systemtap/wiki/PublicanQuikHowto
2147 An online version can be found at:
2148 http://sourceware.org/systemtap/SystemTap_Beginners_Guide.pdf
2150 - Standard tapsets included with Systemtap were modified to include
2151 extractable documentation information based on the kernel-doc
2152 infrastructure. When configured --enabled-docs a HTML and PDF
2153 version of the Tapset Reference Manual is produced explaining probes
2154 defined in each tapset.
2156 - The systemtap client and compile server are now available.
2157 These allow you to compile a systemtap module on a host other than
2158 the one which it will be run, providing the client and server
2159 are compatible. Other than using a server for passes 1 through
2160 4, the client behaves like the 'stap' front end itself. This
2161 means, among other things, that the client will automatically
2162 load the resulting module on the local host unless -p[1234]
2163 was specified. See stap-server(8) for more details.
2164 The client/server now use SSL for network connection security and
2167 The systemtap client and server are prototypes only. Interfaces, options
2168 and usage may change at any time.
2170 - function("func").label("label") probes are now supported to allow matching
2171 the label of a function.
2173 - Systemtap initscript is available. This initscript allows you to run
2174 systemtap scripts as system services (in flight recorder mode) and
2175 control those scripts individually.
2176 See README.systemtap for details.
2178 - The stap "-r DIR" option may be used to identify a hand-made kernel
2179 build directory. The tool determines the appropriate release string
2180 automatically from the directory.
2182 - Serious problems associated with user-space probing in shared libraries
2183 were corrected, making it now possible to experiment with probe shared
2184 libraries. Assuming dwarf debugging information is installed, use this
2185 twist on the normal syntax:
2187 probe process("/lib64/libc-2.8.so").function("....") { ... }
2189 This would probe all threads that call into that library. Running
2190 "stap -c CMD" or "stap -x PID" naturally restricts this to the target
2191 command+descendants only. $$vars etc. may be used.
2193 - For scripts that sometimes terminate with excessive "skipped" probes,
2194 rerunning the script with "-t" (timing) will print more details about
2195 the skippage reasons.
2197 - Symbol tables and unwind (backtracing) data support were formerly
2198 compiled in for all probed modules as identified by the script
2199 (kernel; module("name"); process("file")) plus those listed by the
2200 stap "-d BINARY" option. Now, this data is included only if the systemtap
2201 script uses tapset functions like probefunc() or backtrace() that require
2202 such information. This shrinks the probe modules considerably for the rest.
2204 - Per-pass verbosity control is available with the new "--vp {N}+" option.
2205 "stap --vp 040" adds 4 units of -v verbosity only to pass 2. This is useful
2206 for diagnosing errors from one pass without excessive verbosity from others.
2208 - Most probe handlers now run with interrupts enabled, for improved
2209 system responsiveness and less probing overhead. This may result
2210 in more skipped probes, for example if a reentrant probe handler
2211 is attempted from within an interrupt handler. It may also make the
2212 systemtap overload detection facility more likely to be triggered, as
2213 interrupt handlers' run time would be included in the self-assessed
2214 overhead of running probe handlers.
2216 * What's new in version 0.8, 2008-11-13
2218 - Cache limiting is now available. If the compiled module cache size is
2219 over a limit specified in the $SYSTEMTAP_DIR/cache/cache_mb_limit file,
2220 some old cache entries will be unlinked. See man stap(1) for more.
2222 - Error and warning messages are now followed by source context displaying
2223 the erroneous line/s and a handy '^' in the following line pointing to the
2226 - A bug reporting tool "stap-report" is now available which will quickly
2227 retrieve much of the information requested here:
2228 http://sourceware.org/systemtap/wiki/HowToReportBugs
2230 - The translator can resolve members of anonymous structs / unions:
2231 given struct { int foo; struct { int bar; }; } *p;
2232 this now works: $p->bar
2234 - The stap "-F" flag activates "flight recorder" mode, which consists of
2235 translating the given script as usual, but implicitly launching it into
2236 the background with staprun's existing "-L" (launch) option. A user
2237 can later reattach to the module with "staprun -A MODULENAME".
2239 - Additional context variables are available on user-space syscall probes.
2240 - $argN ($arg1, $arg2, ... $arg6) in process(PATH_OR_PID).syscall
2241 gives you the argument of the system call.
2242 - $return in process(PATH_OR_PID).syscall.return gives you the return
2243 value of the system call.
2245 - Target process mode (stap -c CMD or -x PID) now implicitly restricts all
2246 "process.*" probes to the given child process. (It does not affect
2247 kernel.* or other probe types.) The CMD string is normally run directly,
2248 rather than via a /bin/sh -c subshell, since then utrace/uprobe probes
2249 receive a fairly "clean" event stream. If metacharacters like
2250 redirection operators were present in CMD, then "sh -c CMD" is still
2251 used, and utrace/uprobe probes will receive events from the shell.
2253 % stap -e 'probe process.syscall, process.end {
2254 printf("%s %d %s\n", execname(), pid(), pp())}'\
2256 ls 2323 process.syscall
2257 ls 2323 process.syscall
2260 - Probe listing mode is improved: "-L" lists available script-level variables
2262 % stap -L 'syscall.*open*'
2263 syscall.mq_open name:string name_uaddr:long filename:string mode:long u_attr_uaddr:long oflag:long argstr:string
2264 syscall.open name:string filename:string flags:long mode:long argstr:string
2265 syscall.openat name:string filename:string flags:long mode:long argstr:string
2267 - All user-space-related probes support $PATH-resolved executable
2270 probe process("ls").syscall {}
2271 probe process("./a.out").syscall {}
2273 work now, instead of just
2275 probe process("/bin/ls").syscall {}
2276 probe process("/my/directory/a.out").syscall {}
2278 - Prototype symbolic user-space probing support:
2280 # stap -e 'probe process("ls").function("*").call {
2281 log (probefunc()." ".$$parms)
2286 - debugging information for the named program
2287 - a version of utrace in the kernel that is compatible with the "uprobes"
2288 kernel module prototype. This includes RHEL5 and older Fedora, but not
2289 yet current lkml-track utrace; a "pass 4a"-time build failure means
2290 your system cannot use this yet.
2292 - Global variables which are written to but never read are now
2293 automatically displayed when the session does a shutdown. For example:
2295 global running_tasks
2296 probe timer.profile {running_tasks[pid(),tid()] = execname()}
2297 probe timer.ms(8000) {exit()}
2299 - A formatted string representation of the variables, parameters, or local
2300 variables at a probe point is now supported via the special $$vars,
2301 $$parms, and $$locals context variables, which expand to a string
2302 containing a list "var1=0xdead var2=0xbeef var3=?". (Here, var3 exists
2303 but is for some reason unavailable.) In return probes only, $$return
2304 expands to an empty string for a void function, or "return=0xf00".
2307 * What's new in version 0.7, 2008-07-15
2309 - .statement("func@file:*") and .statement("func@file:M-N") probes are now
2310 supported to allow matching a range of lines in a function. This allows
2311 tracing the execution of a function.
2313 - Scripts relying on probe point wildcards like "syscall.*" that expand
2314 to distinct kprobes are processed significantly faster than before.
2316 - The vector of script command line arguments is available in a
2317 tapset-provided global array argv[]. It is indexed 1 ... argc,
2318 another global. This can substitute for of preprocessor
2319 directives @NNN that fail at parse time if there are not
2322 printf("argv: %s %s %s", argv[1], argv[2], argv[3])
2324 - .statement("func@file+line") probes are now supported to allow a
2325 match relative to the entry of the function incremented by line
2326 number. This allows using the same systemtap script if the rest
2327 of the file.c source only changes slightly.
2329 - A probe listing mode is available.
2338 - More user-space probe types are added:
2340 probe process(PID).begin { }
2341 probe process("PATH").begin { }
2342 probe process(PID).thread.begin { }
2343 probe process("PATH").thread.begin { }
2344 probe process(PID).end { }
2345 probe process("PATH").end { }
2346 probe process(PID).thread.end { }
2347 probe process("PATH").thread.end { }
2348 probe process(PID).syscall { }
2349 probe process("PATH").syscall { }
2350 probe process(PID).syscall.return { }
2351 probe process("PATH").syscall.return { }
2353 - Globals now accept ; terminators
2356 global little[10], big[5];
2358 * What's new in version 0.6, 2007-12-15
2360 - A copy of the systemtap tutorial and language reference guide
2363 - There is a new format specifier, %m, for the printf family of
2364 functions. It functions like %s, except that it does not stop when
2365 a nul ('\0') byte is encountered. The number of bytes output is
2366 determined by the precision specifier. The default precision is 1.
2369 printf ("%m", "My String") // prints one character: M
2370 printf ("%.5", myString) // prints 5 bytes beginning at the start
2373 - The %b format specifier for the printf family of functions has been enhanced
2376 1) When the width and precision are both unspecified, the default is %8.8b.
2377 2) When only one of the width or precision is specified, the other defaults
2378 to the same value. For example, %4b == %.4b == %4.4b
2379 3) Nul ('\0') bytes are used for field width padding. For example,
2381 printf ("%b", 0x1111deadbeef2222) // prints all eight bytes
2382 printf ("%4.2b", 0xdeadbeef) // prints \0\0\xbe\xef
2384 - Dynamic width and precision are now supported for all printf family format
2385 specifiers. For example:
2389 printf ("%*.*b", four, two, 0xdeadbbeef) // prints \0\0\xbe\xef
2390 printf ("%*d", four, two) // prints <space><space><space>2
2392 - Preprocessor conditional expressions can now include wildcard style
2393 matches on kernel versions.
2394 %( kernel_vr != "*xen" %? foo %: bar %)
2396 - Prototype support for user-space probing is showing some progress.
2397 No symbolic notations are supported yet (so no probing by function names,
2398 file names, process names, and no access to $context variables), but at
2399 least it's something:
2401 probe process(PID).statement(ADDRESS).absolute { }
2403 This will set a uprobe on the given process-id and given virtual address.
2404 The proble handler runs in kernel-space as usual, and can generally use
2405 existing tapset functions.
2407 - Crash utility can retrieve systemtap's relay buffer from a kernel dump
2408 image by using staplog which is a crash extension module. To use this
2409 feature, type commands as below from crash(8)'s command line:
2411 crash> extend staplog.so
2412 crash> help systemtaplog
2414 Then, you can see more precise help message.
2416 - You can share a relay buffer amoung several scripts and merge outputs from
2417 several scripts by using "-DRELAY_HOST" and "-DRELAY_GUEST" options.
2421 % stap -ve 'probe begin{}' -o merged.out -DRELAY_HOST &
2422 # wait until starting the host.
2423 % stap -ve 'probe begin{print("hello ");exit()}' -DRELAY_GUEST
2424 % stap -ve 'probe begin{print("world\n");exit()}' -DRELAY_GUEST
2426 Then, you'll see "hello world" in merged.out.
2428 - You can add a conditional statement for each probe point or aliase, which
2429 is evaluated when the probe point is hit. If the condition is false, the
2430 whole probe body(including aliases) is skipped. For example:
2433 probe syscall.* if (switch) { ... }
2434 probe procfs.write {switch = strtol($value,10)} /* enable/disable ctrl */
2436 - Systemtap will warn you if your script contains unused variables or
2437 functions. This is helpful in case of misspelled variables. If it
2438 doth protest too much, turn it off with "stap -w ...".
2440 - You can add error-handling probes to a script, which are run if a
2441 script was stopped due to errors. In such a case, "end" probes are
2442 not run, but "error" ones are.
2444 probe error { println ("oops, errors encountered; here's a report anyway")
2445 foreach (coin in mint) { println (coin) } }
2447 - In a related twist, one may list probe points in order of preference,
2448 and mark any of them as "sufficient" beyond just "optional". Probe
2449 point sequence expansion stops if a sufficient-marked probe point has a hit.
2450 This is useful for probes on functions that may be in a module (CONFIG_FOO=m)
2451 or may have been compiled into the kernel (CONFIG_FOO=y), but we don't know
2454 probe module("sd").function("sd_init_command") ? ,
2455 kernel.function("sd_init_command") ? { ... }
2457 which might match neither, now one can write this:
2459 probe module("sd").function("sd_init_command") ! , /* <-- note excl. mark */
2460 kernel.function("sd_init_command") { ... }
2462 - New security model. To install a systemtap kernel module, a user
2463 must be one of the following: the root user; a member of the
2464 'stapdev' group; or a member of the 'stapusr' group. Members of the
2465 stapusr group can only use modules located in the
2466 /lib/modules/VERSION/systemtap directory (where VERSION is the
2467 output of "uname -r").
2469 - .statement("...@file:line") probes now apply heuristics to allow an
2470 approximate match for the line number. This works similarly to gdb,
2471 where a breakpoint placed on an empty source line is automatically
2472 moved to the next statement. A silly bug that made many $target
2473 variables inaccessible to .statement() probes was also fixed.
2475 - LKET has been retired. Please let us know on <systemtap@sourceware.org>
2476 if you have been a user of the tapset/tools, so we can help you find
2479 - New families of printing functions println() and printd() have been added.
2480 println() is like print() but adds a newline at the end;
2481 printd() is like a sequence of print()s, with a specified field delimiter.
2483 * What's new since version 0.5.14?, 2007-07-03
2485 - The way in which command line arguments for scripts are substituted has
2486 changed. Previously, $1 etc. would interpret the corresponding command
2487 line argument as an numeric literal, and @1 as a string literal. Now,
2488 the command line arguments are pasted uninterpreted wherever $1 etc.
2489 appears at the beginning of a token. @1 is similar, but is quoted as
2490 a string. This change does not modify old scripts, but has the effect
2491 of permitting substitution of arbitrary token sequences.
2493 # This worked before, and still does:
2494 % stap -e 'probe timer.s($1) {}' 5
2495 # Now this also works:
2496 % stap -e 'probe syscall.$1 {log(@1)}' open
2497 # This won't crash, just signal a recursion error:
2499 # As before, $1... is recognized only at the beginning of a token
2500 % stap -e 'probe begin {foo$1=5}'
2502 * What's new since version 0.5.13?, 2007-03-26
2504 - The way in which systemtap resolves function/inline probes has changed:
2505 .function(...) - now refers to all functions, inlined or not
2506 .inline(...) - is deprecated, use instead:
2507 .function(...).inline - filters function() to only inlined instances
2508 .function(...).call - filters function() to only non-inlined instances
2509 .function(...).return - as before, but now pairs best with .function().call
2510 .statement() is unchanged.
2512 * What's new since version 0.5.12?, 2007-01-01
2514 - When running in -p4 (compile-only) mode, the compiled .ko file name
2515 is printed on standard output.
2517 - An array element with a null value such as zero or an empty string
2518 is now preserved, and will show up in a "foreach" loop or "in" test.
2519 To delete such an element, the scripts needs to use an explicit
2520 "delete array[idx]" statement rather than something like "array[idx]=0".
2522 - The new "-P" option controls whether prologue searching heuristics
2523 will be activated for function probes. This was needed to get correct
2524 debugging information (dwarf location list) data for $target variables.
2525 Modern compilers (gcc 4.1+) tend not to need this heuristic, so it is
2526 no longer default. A new configure flag (--enable-prologues) restores
2527 it as a default setting, and is appropriate for older compilers (gcc 3.*).
2529 - Each systemtap module prints a one-line message to the kernel informational
2530 log when it starts. This line identifies the translator version, base
2531 address of the probe module, a broken-down memory consumption estimate, and
2532 the total number of probes. This is meant as a debugging / auditing aid.
2534 - Begin/end probes are run with interrupts enabled (but with
2535 preemption disabled). This will allow begin/end probes to be
2536 longer, to support generating longer reports.
2538 - The numeric forms of kernel.statement() and kernel.function() probe points
2539 are now interpreted as relocatable values - treated as relative to the
2540 _stext symbol in that kernel binary. Since some modern kernel images
2541 are relocated to a different virtual address at startup, such addresses
2542 may shift up or down when actually inserted into a running kernel.
2544 kernel.statement(0xdeadbeef): validated, interpreted relative to _stext,
2545 may map to 0xceadbeef at run time.
2547 In order to specify unrelocated addresses, use the new ".absolute"
2548 probe point suffix for such numeric addresses. These are only
2549 allowed in guru mode, and provide access to no $target variables.
2550 They don't use debugging information at all, actually.
2552 kernel.statement(0xfeedface).absolute: raw, unvalidated, guru mode only
2554 * What's new since version 0.5.10?, 2006-10-19
2556 - Offline processing of debugging information, enabling general
2557 cross-compilation of probe scripts to remote hosts, without
2558 requiring identical module/memory layout. This slows down
2559 compilation/translation somewhat.
2561 - Kernel symbol table data is loaded by staprun at startup time
2562 rather than compiled into the module.
2564 - Support the "limit" keyword for foreach iterations:
2565 foreach ([x,y] in ary limit 5) { ... }
2566 This implicitly exits after the fifth iteration. It also enables
2567 more efficient key/value sorting.
2569 - Support the "maxactive" keyword for return probes:
2570 probe kernel.function("sdfsdf").maxactive(848) { ... }
2571 This allows up to 848 concurrently outstanding entries to
2572 the sdfsdf function before one returns. The default maxactive
2573 number is smaller, and can result in missed return probes.
2575 - Support accessing of saved function arguments from within
2576 return probes. These values are saved by a synthesized
2577 function-entry probe.
2579 - Add substantial version/architecture checking in compiled probes to
2580 assert correct installation of debugging information and correct
2581 execution on a compatible kernel.
2583 - Add probe-time checking for sufficient free stack space when probe
2584 handlers are invoked, as a safety improvement.
2586 - Add an optional numeric parameter for begin/end probe specifications,
2587 to order their execution.
2588 probe begin(10) { } /* comes after */ probe begin(-10) {}
2590 - Add an optional array size declaration, which is handy for very small
2592 global little[5], big[20000]
2594 - Include some example scripts along with the documentation.
2596 - Change the start-time allocation of probe memory to avoid causing OOM
2597 situations, and to abort cleanly if free kernel memory is short.
2599 - Automatically use the kernel DWARF unwinder, if present, for stack
2602 - Many minor bug fixes, performance, tapset, and error message