NEWS

   1 * What's new in version 2.6
   2
   3 - process("PATH").library("PATH").plt("NAME").return probes are now supported.
   4
   5 - SystemTap now supports SDT probes with operands that refer to symbols.
   6
   7 - While in listing mode (-l/-L), probes printed are now more consistent
   8   and precise.
   9
  10 - Statement probes now support enumerated linenos to probe discontiguous
  11   linenos using the form:
  12
  13   process.statement("foo@file.c:3,5-7,9")
  14
  15 - Statement counting is now suppressed in the generated c code for probes that
  16   are non-recursive and loop-free. Statement counting can be turned back on in
  17   unoptimize mode (-u).
  18
  19 - SystemTap now asserts that the PID provided for a process probe corresponds
  20   to a runnning process.
  21
  22 - DWARF process probes can be bound to a specific process using the form:
  23
  24   process(PID).function("*")
  25
  26 - SystemTap now accepts additional scripts through the new -E SCRIPT option.
  27   There still needs to be a main script specified through -e or file in order
  28   to provide an additional script. This makes it feasible to have scripts in
  29   the $HOME/.systemtap/rc file.  For example:
  30
  31   -E 'probe begin, end, error { log("systemtap script " . pn()) }'
  32   -E 'probe timer.s(30) { error ("timeout") }
  33
  34   The -E SCRIPT option can also be used in listing mode (-l/-L), such that
  35   probe points for the additional scripts will not listed, but other parts of
  36   the script are still available, such as macros or aliases.
  37
  38 * What's new in version 2.5, 2014-04-30
  39
  40 - Systemtap now supports backtracing through its own, invoking module.
  41
  42 - Java probes now support backtracing using the print_java_backtrace()
  43   and sprint_java_backtrace() functions.
  44
  45 - Statement probes (e.g. process.statement) are now faster to resolve,
  46   more precise, and work better with inlined functions.
  47
  48 - New switches have been added to help inspect the contents of installed
  49   library files:
  50
  51   stap --dump-functions --> list all library functions and their args
  52   stap --dump-probe-aliases --> list all library probe aliases
  53
  54 - The heuristic algorithms used to search for function-prologue
  55   endings were improved, to cover more optimization (or
  56   lack-of-optimization, or incorrect-debuginfo) cases.  These
  57   heuristics are necessary to find $context parameters for some
  58   function-call/entry probes.  We recommend programs be built with
  59   CFLAGS+=-grecord-gcc-switches to feed information to the heuristics.
  60
  61 - The stap --use-server option now more correctly supports address:port
  62   type parametrization, for manual use in the absence of avahi.
  63
  64 - A new probe alias "oneshot" allows a single quick script fragment to run,
  65   then exit.
  66
  67 - The argv tapset now merges translate-time and run-time positional
  68   arguments, so all of these work:
  69
  70   stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' hello world
  71
  72   stap -e 'probe oneshot {println(argv[1]," ",argv[2])}' \
  73        -G argv_1=hello -G argv_2=world
  74
  75   staprun hello.ko argv_1=hello argv_2=world
  76
  77 - SystemTap now falls back on the symbol table for probing
  78   functions in processes if the debuginfo is not available.
  79
  80 - SystemTap now supports a %( guru_mode == 0 /* or 1 */ %)
  81   conditional for making dual-use scripts.
  82
  83 - SystemTap now supports UEFI/SecureBoot systems, via
  84   machine-owner-keys maintained by a trusted stap-server on the
  85   network.  (Key enrollment requires a one-time reboot and BIOS
  86   conversation.)
  87   https://sourceware.org/systemtap/wiki/SecureBoot
  88
  89 - SystemTap now reports more accurate and succinct errors on type
  90   mismatches.
  91
  92 - Embedded-C functions may use STAP_RETURN(value) instead of the
  93   more wordy STAP_RETVALUE assignment followed by a "goto out".
  94   The macro supports numeric or string values as appropriate.
  95
  96 - Some struct-sockaddr fields are now individually decoded for
  97   socket-related syscalls:
  98   probe syscall.connect { println (uaddr_af, ":", uaddr_ip) }
  99
 100 - The documentation for the SystemTap initscript service and the
 101   SystemTap compile-server service have been completely converted from
 102   README files to man pages (see systemtap(8) and stap-server(8)).
 103
 104 - SystemTap is now capable of inserting modules early during the boot
 105   process on dracut-based systems. See the 'onboot' command in
 106   systemtap(8) for more information.
 107
 108 - DWARF probes can now use the '.callee[s]' variants, which allow more
 109   precise function probing. For example, the probe point
 110
 111     process("myproc").function("foo").callee("bar")
 112
 113   will fire upon entering bar() from foo(). A '.callees' probe will
 114   instead place probes on all callees of foo().
 115     Note that this also means that probe point wildcards should be used
 116   with more care. For example, use signal.*.return rather than
 117   signal.*.*, which would also match '.callees'. See stapprobes(3stap)
 118   for more info. This feature requires at least GCC 4.7.
 119
 120 - A few new functions in the task_time tapsets, as well as a new tapset
 121   function task_ancestry(), which prints out the parentage of a process.
 122
 123 - The kprocess.exec probe has been updated to use syscall.execve, which
 124   allows access to the new process' arguments (through the new 'argstr'
 125   or 'args' variables) as well as giving better support across kernel
 126   versions. Note also that the 'filename' variable now holds the
 127   filename (quoted), or the address (unquoted) if it couldn't be
 128   retrieved.
 129
 130 - The [s]println() function can now be called without any arguments to
 131   simply print a newline.
 132
 133 - Suggestions are now provided when markers could not be resolved. For
 134   example, process("stap").mark("benchmart") will suggest 'benchmark'.
 135
 136 - SystemTap colors can now be turned off by simply setting
 137   SYSTEMTAP_COLORS to be empty, rather than having to make it invalid.
 138
 139 - There is a new context tapset function, pnlabel(), which returns the
 140   name of the label which fired.
 141
 142 - The following tapset variables and functions are deprecated in
 143   release 2.5:
 144   - The 'clone_flags', 'stack_start', 'stack_size',
 145     'parent_tid_uaddr', and 'child_tid_uaddr' variables in the
 146     'syscall.fork' and 'nd_syscall.fork' probe aliases.
 147   - The '_sendflags_str()' and '_recvflags_str()' functions have been
 148     deprecated in favor of the new '_msg_flags_str()' function.
 149   - The 'flags' and 'flags_str' variables in the 'syscall.accept' and
 150     'nd_syscall.accept' probe alias.
 151   - The 'first', 'second', and 'uptr_uaddr' variables in the
 152     'syscall.compat_sys_shmctl', and 'nd_syscall.compat_sys_shmctl'
 153     probe aliases have been deprecated in favor of the new 'shmid',
 154     'cmd', and 'buf_uaddr' variables.
 155
 156 * What's new in version 2.4, 2013-11-06
 157
 158 - Better suggestions are given in many of the semantic errors in which
 159   alternatives are provided. Additionally, suggestions are now provided
 160   when plt and trace probes could not be resolved. For example,
 161   kernel.trace("sched_siwtch") will suggest 'sched_switch'.
 162
 163 - SystemTap is now smarter about error reporting. Errors from the same
 164   source are considered duplicates and suppressed. A message is
 165   displayed on exit if any errors/warnings were suppressed.
 166
 167 - Statistics aggregate typed objects are now implemented locklessly,
 168   if the translator finds that they are only ever read (using the
 169   foreach / @count / etc. constructs) in a probe-begin/end/error.
 170
 171 - SystemTap now supports probing inside virtual machines using the
 172   libvirt and unix schemes, e.g.
 173
 174       stap -ve 'probe timer.s(1) { printf("hello!\n") }' \
 175          --remote=libvirt://MyVirtualMachine
 176
 177   Virtual machines managed by libvirt can be prepared using stapvirt.
 178   See stapvirt(1) and the --remote option in stap(1) for more details.
 179
 180 - Systemtap now checks for and uses (when available) the .gnu_debugdata
 181   section which contains a subset of debuginfo, useful for backtraces
 182   and function probing
 183
 184 - SystemTap map variables are now allocated with vmalloc() instead of
 185   with kmalloc(), which should cause memory to be less fragmented.
 186
 187 - Although SystemTap itself requires elfutils 0.148+, staprun only
 188   requires elfutils 0.142+, which could be useful with the
 189   '--disable-translator' configure switch.
 190
 191 - Under FIPS mode (/proc/sys/crypto/fips_enabled=1), staprun will
 192   refuse to load systemtap modules (since these are not normally
 193   signed with the kernel's build-time keys).  This protection may
 194   be suppressed with the $STAP_FIPS_OVERRIDE environment variable.
 195
 196 - The stap-server client & server code now enable all SSL/TLS
 197   ciphers rather than just the "export" subset.
 198
 199 - For systems with in-kernel utrace, 'process.end' and 'thread.end'
 200   probes will hit before the target's parent process is notified of
 201   the target's death. This matches the behavior of newer kernels
 202   without in-kernel utrace.
 203
 204 * What's new in version 2.3, 2013-07-25
 205
 206 - More context-accessing functions throw systemtap exceptions upon a
 207   failure, whereas in previous versions they might return non-error
 208   sentinel values like "" or "<unknown>".  Use try { } / catch { }
 209   around these, or new wrapper functions such as user_string_{n_}quoted()
 210   that internally absorb exceptions.
 211
 212 - java("org.my.MyApp") probes are now restricted to pre-existing jvm pid's with
 213   a listing in jps -l output to avoid recursive calls
 214
 215 - The tapset [nd_]syscall.semop parameter tsops_uaddr is renamed sops_uaddr for
 216   consistency with [nd_]syscall.semtimedop.
 217
 218 - The udp.stp tapset adds some ip-address/port variables.
 219
 220 - A new guru-mode-only tapset function raise() is available to send signals
 221   to the current task.
 222
 223 - Support for the standard Posix ERE named character classes has been
 224   added to the regexp engine, e.g. [:digit:], [:alpha:], ...
 225
 226 - A substantial internal overhaul of the regexp engine has resulted in
 227   correct behaviour on further obscure edge cases. The regexp engine
 228   now implements the ERE standard and correctly passes the testsuite
 229   for the glibc regexp engine (minus portions corresponding to
 230   unimplemented features -- i.e. subexpression capture and reuse).
 231
 232 - Alternative functions are now suggested when function probes could not be
 233   resolved. For example, kernel.function("vfs_reads") will suggest vfs_read.
 234   Other probes for which suggestions are made are module.function,
 235   process.function, and process.library.function.
 236
 237 - Has life been a bit bland lately? Want to spice things up? Why not write a
 238   few faulty probes and feast your eyes upon the myriad of colours adorning
 239   your terminal as SystemTap softly whispers in your ear... 'parse error'.
 240   Search for '--color' in 'man stap' for more info.
 241
 242 - The following tapset functions are deprecated in release 2.3:
 243   'stap_NFS_CLIENT', '__getfh_inode', '_success_check',
 244   '_sock_prot_num', '_sock_fam_num', '_sock_state_num',
 245   '_sock_type_num', and '_sock_flags_num'.
 246
 247 * What's new in version 2.2.1, 2013-05-16
 248 * What's new in version 2.2, 2013-05-14
 249
 250 - Experimental support has been added for probing Java methods using
 251   Byteman 2.0 as a backend. Java method probes can target method entries,
 252   returns, or specific statements in the method as specified by line number.
 253
 254       probe java("org.my.MyApp").class("^java.lang.Object").method("foo(int)")
 255          { println($$parms) }
 256
 257   See java/README for information on how to set up Java/Byteman
 258   functionality.  Set env STAPBM_VERBOSE=yes for more tracing.
 259
 260 - The stap -l output and pn() tapset function's return value may be slightly
 261   different for complicated web of wildcarded/aliased probes.
 262
 263 - The dyninst backend has improved in several aspects:
 264
 265   - Setting custom values for global variables is now supported, both
 266     with -G when compiling a script, and from the stapdyn command line
 267     when loading a precompiled module.
 268
 269   - A high-performance shared-memory-based transport is used for
 270     trace data.
 271
 272 - A systemd service file and tmpfile have been added to allow
 273   systemtap-server to be managed natively by systemd.
 274
 275 - Due to the removal of register_timer_hook in recent kernels, the
 276   behaviour of timer.profile has been changed slightly. This probe is
 277   now an alias which uses the old mechanism where possible, but falls
 278   back to perf.sw.cpu_clock or another mechanism when the kernel timer
 279   hook is not available.
 280
 281   To require the kernel timer hook mechanism in your script, use
 282   timer.profile.tick instead of timer.profile.
 283
 284 - The following tapset variables are deprecated in release 2.2:
 285   - The 'origin' variables in the 'generic.fop.llseek',
 286     'generic.fop.llseek.return', and 'nfs.fop.llseek' probes. The
 287     'origin' variable has been replaced by the 'whence' variable.
 288   - The 'page_index' variable in the 'vfs.block_sync_page' and
 289     'vfs.buffer_migrate_page' probe aliases.
 290   - The 'write_from' and 'write_upto' variables in the
 291     '_vfs.block_prepare_write' and '_vfs.block_prepare_write.return'
 292     probe aliases.
 293   - The 'regs' variable in the 'syscall.sigaltstack',
 294     'nd_syscall.sigaltstack', 'syscall.fork', and 'nd_syscall.fork'
 295     probe aliases.
 296   - The 'first', 'second', 'third', and 'uptr_uaddr' variables in the
 297     'syscall.compat_sys_shmat' and 'nd_syscall.compat_sys_shmat' probe
 298     aliases.
 299
 300 - The following tapset functions are deprecated in release 2.2:
 301   'ppos_pos', '_dev_minor', and '_dev_major'
 302
 303 - The folowing tapset functions used to return error strings instead
 304   of raising an error. The original behavior is deprecated in release
 305   2.2.
 306
 307   'ctime', 'probemod', 'modname'
 308
 309 * What's new in version 2.1, 2013-02-13
 310
 311 - EMACS and VIM editor modes for systemtap source files are included / updated.
 312
 313 - The translator now eliminates duplicate tapset files between its
 314   preferred directory (as configured during the build with --prefix=/
 315   or specified with the -I /path option), and files it may find under
 316   $XDG_DATA_DIRS.  This should eliminate a class of conflicts between
 317   parallel system- and hand-built systemtap installations.
 318
 319 - The translator accepts a --suppress-time-limits option, which defeats
 320   time-related constraints, to allows probe handlers to run for indefinite
 321   periods.  It requires the guru mode (-g) flag to work.  Add the earlier
 322   --suppress-handler-errors flag for a gung-ho "just-keep-going" attitude.
 323
 324 - Perf event probes may now be read on demand. The counter probe is
 325   defined using the counter-name part:
 326   probe perf.type(0).config(0).counter("NAME").  The counter is
 327   read in a user space probe using @perf("NAME"), e.g.
 328    process("PROCESS").statement("func@file") {stat <<< @perf("NAME")}
 329
 330 - Perf event probes may now be bound to a specific task using the
 331   process-name part:  probe perf.type(0).config(0).process("NAME") { }
 332   If the probed process name is not specified, then it is inferred
 333   from the -c CMD argument.
 334
 335 - Some error messages and warnings now refer to additional information
 336   that is found in man pages.  These are generally named
 337   error::FOO or warning::BAR (in the 7stap man page section)
 338   and may be read via
 339      % man error::FOO
 340      % man warning::BAR
 341
 342 - The dyninst backend has improved in several aspects:
 343   - The runtime now allows much more concurrency when probing multithreaded
 344     processes, and will also follow probes across forks.
 345   - Several new probe types are now supported, including timers, function
 346     return, and process.begin/end and process.thread.begin/end.
 347   - Semaphores for SDT probes are now set properly.
 348   - Attaching to existing processes with -x PID now works.
 349
 350 - The foreach looping construct can now sort aggregate arrays by the user's
 351   choice of aggregating function.  Previously, @count was implied.  e.g.:
 352      foreach ([x,y] in array @sum +) { println(@sum(array[x,y])) }
 353
 354 - Proof of concept support for regular expression matching has been added:
 355      if ("aqqqqqb" =~ "q*b") { ... }
 356      if ("abc" !~ "q*b") { ... }
 357
 358   The eventual aim is to support roughly the same functionality as
 359   the POSIX Extended Regular Expressions implemented by glibc.
 360   Currently missing features include extraction of the matched string
 361   and subexpressions, and named character classes ([:alpha:], [:digit:], &c).
 362
 363   Special thanks go to the re2c project, whose public domain code this
 364   functionality has been based on. For more info on re2c, see:
 365      http://sourceforge.net/projects/re2c/
 366
 367 - The folowing tapset variables are deprecated in release 2.1 and will
 368   be removed in release 2.2:
 369   - The 'send2queue' variable in the 'signal.send' probe.
 370   - The 'oldset_addr' and 'regs' variables in the 'signal.handle' probe.
 371
 372 - The following tapset probes are deprecated in release 2.1 and will
 373   be removed in release 2.2:
 374   - signal.send.return
 375   - signal.handle.return
 376
 377 * What's new in version 2.0, 2012-10-09
 378
 379 - Systemtap includes a new prototype backend, which uses Dyninst to instrument
 380   a user's own processes at runtime. This backend does not use kernel modules,
 381   and does not require root privileges, but is restricted with respect to the
 382   kinds of probes and other constructs that a script may use.
 383
 384   Users from source should configure --with-dyninst and install a
 385   fresh dyninst snapshot such as that in Fedora rawhide.  It may be
 386   necessary to disable conflicting selinux checks; systemtap will advise.
 387
 388   Select this new backend with the new stap option --runtime=dyninst
 389   and a -c target process, along with normal options. (-x target
 390   processes are not supported in this prototype version.) For example:
 391
 392     stap --runtime=dyninst -c 'stap -l begin' \
 393       -e 'probe process.function("main") { println("hi from dyninst!") }'
 394
 395 - To aid diagnosis, when a kernel panic occurs systemtap now uses
 396   the panic_notifier_list facility to dump a summary of its trace
 397   buffers to the serial console.
 398
 399 - The systemtap preprocessor now has a simple macro facility as follows:
 400
 401     @define add(a,b) %( ((@a)+(@b)) %)
 402     @define probegin(x) %(
 403        probe begin {
 404          @x
 405        }
 406     %)
 407
 408     @probegin( foo = @add(40, 2); print(foo) )
 409
 410   Macros defined in the user script and regular tapset .stp files are
 411   local to the file. To get around this, the tapset library can define
 412   globally visible 'library macros' inside .stpm files. (A .stpm file
 413   must contain a series of @define directives and nothing else.)
 414
 415   The status of the feature is experimental; semantics of macroexpansion
 416   may change (unlikely) or expand in the future.
 417
 418 - Systemtap probe aliases may be used with additional suffixes
 419   attached. The suffixes are passed on to the underlying probe
 420   point(s) as shown below:
 421
 422     probe foo = bar, baz { }
 423     probe foo.subfoo.option("gronk") { }
 424     // expands to: bar.subfoo.option("gronk"), baz.subfoo.option("gronk")
 425
 426   In practical terms, this allows us to specify additional options to
 427   certain tapset probe aliases, by writing e.g.
 428     probe syscall.open.return.maxactive(5) { ... }
 429
 430 - To support the possibility of separate kernel and dyninst backends,
 431   the tapsets have been reorganized into separate folders according to
 432   backend. Thus kernel-specific tapsets are located under linux/, the
 433   dyninst-specific ones under dyninst/
 434
 435 - The backtrace/unwind tapsets have been expanded to allow random
 436   access to individual elements of the backtrace. (A caching mechanism
 437   ensures that the backtrace computation run at most once for each
 438   time a probe fires, regardless of how many times or what order the
 439   query functions are called in.) New tapset functions are:
 440     stack/ustack - return n'th element of backtrace
 441     callers/ucallers - return first n elements of backtrace
 442     print_syms/print_usyms - print full information on a list of symbols
 443     sprint_syms/sprint_usyms - as above, but return info as a string
 444
 445   The following existing functions have been superseded by print_syms()
 446   et al.; new scripts are recommended to avoid using them:
 447       print_stack()
 448       print_ustack()
 449       sprint_stack()
 450       sprint_ustack()
 451
 452 - The probefunc() tapset function is now myproc-unprivileged, and can
 453   now be used in unprivileged scripts for such things as profiling in
 454   userspace programs. For instance, try running
 455   systemtap.examples/general/para-callgraph.stp in unprivileged mode
 456   with a stapusr-permitted probe.  The previous implementation of
 457   probefunc() is available with "stap --compatible=1.8".
 458
 459 - Preprocessor conditional to vary code based on script privilege level:
 460   unprivileged -- %( systemtap_privilege == "stapusr" %? ... %)
 461   privileged   -- %( systemtap_privilege != "stapusr" %? ... %)
 462   or, alternately %( systemtap_privilege == "stapsys"
 463                   || systemtap_privilege == "stapdev" %? ... %)
 464
 465 - To ease migration to the embedded-C locals syntax introduced in 1.8
 466   (namely, STAP_ARG_* and STAP_RETVALUE), the old syntax can now be
 467   re-enabled on a per-function basis using the /* unmangled */ pragma:
 468
 469     function add_foo:long(a:long, b:long) %{ /* unmangled */
 470       THIS->__retvalue = THIS->a + STAP_ARG_b;
 471     %}
 472
 473   Note that both the old and the new syntax may be used in an
 474   /* unmangled */ function. Functions not marked /* unmangled */
 475   can only use the new syntax.
 476
 477 - Adjacent string literals are now glued together irrespective of
 478   intervening whitespace or comments:
 479     "foo " "bar" --> "foo bar"
 480     "foo " /* comment */ "bar" --> "foo bar"
 481   Previously, the first pair of literals would be glued correctly,
 482   while the second would cause a syntax error.
 483
 484 * What's new in version 1.8, 2012-06-17
 485
 486 - staprun accepts a -T timeout option to allow less frequent wake-ups
 487   to poll for low-throughput output from scripts.
 488
 489 - When invoked by systemtap, the kbuild $PATH environment is sanitized
 490   (prefixed with /usr/bin:/bin:) in an attempt to exclude compilers
 491   other than the one the kernel was presumed built with.
 492
 493 - Printf formats can now use "%#c" to escape non-printing characters.
 494
 495 - Pretty-printed bitfields use integers and chars use escaped formatting
 496   for printing.
 497
 498 - The systemtap compile-server and client now support IPv6 networks.
 499   - IPv6 addresses may now be specified on the --use-server option and will
 500     be displayed by --list-servers, if the avahi-daemon service is running and
 501     has IPv6 enabled.
 502   - Automatic server selection will automatically choose IPv4 or IPv6 servers
 503     according to the normal server selection criteria when avahi-daemon is
 504     running. One is not preferred over the other.
 505   - The compile-server will automatically listen on IPv6 addresses, if
 506     available.
 507   - To enable IPv6 in avahi-daemon, ensure that /etc/avahi/avahi-daemon.conf
 508     contains an active "use-ipv6=yes" line. After adding this line run
 509     "service avahi-daemon restart" to activate IPv6 support.
 510   - See man stap(1) for details on how to use IPv6 addresses with the
 511     --use-server option.
 512
 513 - Support for DWARF4 .debug_types sections (for executables and shared
 514   libraries compiled with recent GCC's -gdwarf-4 / -fdebug-types-section).
 515   PR12997.  SystemTap now requires elfutils 0.148+, full .debug_types support
 516   depends on elfutils 0.154+.
 517
 518 - Systemtap modules are somewhat smaller & faster to compile.  Their
 519   debuginfo is now suppressed by default; use -B CONFIG_DEBUG_INFO=y to
 520   re-enable.
 521
 522 - @var now an alternative language syntax for accessing DWARF variables
 523   in uprobe and kprobe handlers (process, kernel, module). @var("somevar")
 524   can be used where $somevar can be used. The @var syntax also makes it
 525   possible to access non-local, global compile unit (CU) variables by
 526   specifying the CU source file as follows @var("somevar@some/src/file.c").
 527   This will provide the target variable value of global "somevar" as defined
 528   in the source file "some/src/file.c". The @var syntax combines with all
 529   normal features of DWARF target variables like @defined(), @entry(),
 530   [N] array indexing, field access through ->, taking the address with
 531   the & prefix and shallow or deep pretty printing with a $ or $$ suffix.
 532
 533 - Stap now has resource limit options:
 534     --rlimit-as=NUM
 535     --rlimit-cpu=NUM
 536     --rlimit-nproc=NUM
 537     --rlimit-stack=NUM
 538     --rlimit-fsize=NUM
 539   All resource limiting has been moved from the compile server to stap
 540   itself. When running the server as "stap-server", default resource
 541   limit values are specified in ~stap-server/.systemtap/rc.
 542
 543 - Bug CVE-2012-0875 (kernel panic when processing malformed DWARF unwind data)
 544   is fixed.
 545
 546 - The systemtap compile-server now supports multiple concurrent connections.
 547   Specify the desired maximum number of concurrent connections with
 548   the new stap-server/stap-serverd --max-threads option. Specify a
 549   value of '0' to tell the server not to spawn any new threads (handle
 550   all connections serially in the main thread). The default value is
 551   the number of processor cores on the host.
 552
 553 - The following tapset functions are deprecated in release 1.8 and will be
 554   removed in release 1.9:
 555       daddr_to_string()
 556
 557 - SystemTap now mangles local variables to avoid collisions with C
 558   headers included by tapsets. This required a change in how
 559   embedded-C functions access local parameters and the return value slot.
 560
 561   Instead of THIS->foo in an embedded-C function, please use the newly
 562   defined macro STAP_ARG_foo (substitute the actual name of the
 563   argument for 'foo'); instead of THIS->__retvalue, use the newly
 564   defined STAP_RETVALUE. All of the tapsets and test cases have been
 565   adapted to use this new notation.
 566
 567   If you need to run code which uses the old THIS-> notation, run stap
 568   with the --compatible=1.7 option.
 569
 570 - There is updated support for user-space probing against kernels >=
 571   3.5, which have no utrace but do have the newer inode-uprobes work
 572   by Srikar Dronamraju and colleagues.  For kernels < 3.5, the
 573   following 3 sets of kernel patches would need to be backported to
 574   your kernel to use this preliminary user-space probing support:
 575
 576   - inode-uprobes patches:
 577     - 2b144498350860b6ee9dc57ff27a93ad488de5dc
 578     - 7b2d81d48a2d8e37efb6ce7b4d5ef58822b30d89
 579     - a5f4374a9610fd7286c2164d4e680436727eff71
 580     - 04a3d984d32e47983770d314cdb4e4d8f38fccb7
 581     - 96379f60075c75b261328aa7830ef8aa158247ac
 582     - 3ff54efdfaace9e9b2b7c1959a865be6b91de96c
 583     - 35aa621b5ab9d08767f7bc8d209b696df281d715
 584     - 900771a483ef28915a48066d7895d8252315607a
 585     - e3343e6a2819ff5d0dfc4bb5c9fb7f9a4d04da73
 586   - exec tracepoint kernel patch:
 587     - 4ff16c25e2cc48cbe6956e356c38a25ac063a64d
 588   - task_work_add kernel patches:
 589     - e73f8959af0439d114847eab5a8a5ce48f1217c4
 590     - 4d1d61a6b203d957777d73fcebf19d90b038b5b2
 591     - 413cd3d9abeaef590e5ce00564f7a443165db238
 592     - dea649b8ac1861107c5d91e1a71121434fc64193
 593     - f23ca335462e3c84f13270b9e65f83936068ec2c
 594
 595 * What's new in version 1.7, 2012-02-01
 596
 597 - Map inserting and deleting is now significantly faster due to
 598   improved hashing and larger hash tables. The hashes are also
 599   now randomized to provide better protection against deliberate
 600   collision attacks.
 601
 602 - Formatted printing is faster by compiling the formatting directives
 603   to C code rather than interpreting at run time.
 604
 605 - Systemtap loads extra command line options from $SYSTEMTAP_DIR/rc
 606   ($HOME/.systemtap/rc by default) before the normal argc/argv.  This
 607   may be useful to activate site options such as --use-server or
 608   --download-debuginfo or --modinfo.
 609
 610 - The stap-server has seen many improvements, and is no longer considered
 611   experimental.
 612
 613 - The stap-server service (initscript) now supports four new options:
 614     -D MACRO[=VALUE]
 615     --log LOGFILE
 616     --port PORT-NUMBER
 617     --SSL CERT-DATABASE
 618   These allow the specification of macro definitions to be passed to stap
 619   by the server, the location of the log file, network port number and
 620   NSS certificate database location respectively. These options are also
 621   supported within individual server configuration files. See stap-server
 622   and initscript/README.stap-server for details.  The stap-server is no
 623   longer activated by default.
 624
 625 - process("PATH").[library("PATH")].function("NAME").exported probes are now
 626   supported to filter function() to only exported instances.
 627
 628 - The translator supports a new --suppress-handler-errors option, which
 629   causes most runtime errors to be turned into quiet skipped probes.  This
 630   also disables the MAXERRORS and MAXSKIPPED limits.
 631
 632 - Translator warnings have been standardized and controlled by the -w / -W
 633   flags.
 634
 635 - The translator supports a new --modinfo NAME=VALUE option to emit additional
 636   MODULE_INFO(n,v) macros into the generated code.
 637
 638 - There is no more fixed maximum number of VMA pages that will be tracked
 639   at runtime. This reduces memory use for those scripts that don't need any,
 640   or only limited target process VMA tracking and allows easier system
 641   wide probes inspecting shared library variables and/or user backtraces.
 642   stap will now silently ignore -DTASK_FINDER_VMA_ENTRY_ITEMS.
 643
 644 - The tapset functions remote_id() and remote_uri() identify the member of a
 645   swarm of "stap --remote FOO --remote BAR baz.stp" concurrent executions.
 646
 647 - Systemtap now supports a new privilege level and group, "stapsys", which
 648   is equivalent to the privilege afforded by membership in the group "stapdev",
 649   except that guru mode (-g) functionality may not be used. To support this, a
 650   new option, --privilege=[stapusr|stapsys|stapdev] has been added.
 651   --privilege=stapusr is equivalent to specifying the existing --unprivileged
 652   option. --privilege=stapdev is the default. See man stap(1) for details.
 653
 654 - Scripts that use kernel.trace("...") probes compile much faster.
 655
 656 - The systemtap module cache is cleaned less frequently, governed by the
 657   number of seconds in the $SYSTEMTAP_DIR/cache/cache_clean_interval_s file.
 658
 659 - SDT can now define up to 12 arguments in a probe point.
 660
 661 - Parse errors no longer generate a cascade of false errors.  Instead, a
 662   parse error skips the rest of the current probe or function, and resumes
 663   at the next one.  This should generate fewer and better messages.
 664
 665 - Global array wrapping is now supported for both associative and statistics typed
 666   arrays using the '%' character to signify a wrapped array. For example,
 667   'global foo%[100]' would allow the array 'foo' to be wrapped if more than 100
 668   elements are inserted.
 669
 670 - process("PATH").library("PATH").plt("NAME") probes are now supported.
 671   Wildcards are supported in the plt-name part, to refer to any function in the
 672   program linkage table which matches the glob pattern and the rest of the
 673   probe point.
 674
 675 - A new option, --dump-probe-types, will dump a list of supported probe types.
 676   If --unprivileged is also specified, the list will be limited to probe types
 677   which are available to unprivileged users.
 678
 679 - Systemtap can now automatically download the required debuginfo
 680   using abrt. The --download-debuginfo[=OPTION] can be used to
 681   control this feature. Possible values are: 'yes', 'no', 'ask',
 682   and a positive number representing the timeout desired. The
 683   default behavior is to not automatically download the debuginfo.
 684
 685 - The translator has better support for probing C++ applications by
 686   better undertanding of compilation units, nested types, templates,
 687   as used in probe point and @cast constructs.
 688
 689 - On 2.6.29+ kernels, systemtap can now probe kernel modules that
 690   arrive and/or depart during the run-time of a session.  This allows
 691   probing of device driver initialization functions, which had formerly been
 692   blacklisted.
 693
 694 - New tapset functions for cpu_clock and local_clock access were added.
 695
 696 - There is some limited preliminary support for user-space probing
 697   against kernels such as linux-next, which have no utrace but do have
 698   the newer inode-uprobes work by Srikar Dronamraju and colleagues.
 699
 700 - The following probe types are deprecated in release 1.7 and will be
 701   removed in release 1.8:
 702       kernel.function(number).inline
 703       module(string).function(number).inline
 704       process.function(number).inline
 705       process.library(string).function(number).inline
 706       process(string).function(number).inline
 707       process(string).library(string).function(number).inline
 708
 709 - The systemtap-grapher is deprecated in release 1.7 and will be removed in
 710   release 1.8.
 711
 712 - The task_backtrace() tapset function was deprecated in 1.6 and has been
 713   removed in 1.7.
 714
 715 - MAXBACKTRACE did work in earlier releases, but has now been documented
 716   in the stap 1 manual page.
 717
 718 - New tapset function probe_type(). Returns a short string describing
 719   the low level probe handler type for the current probe point.
 720
 721 - Both unwind and symbol data is now only collected and emitted for
 722   scripts actually using backtracing or function/data symbols.
 723   Tapset functions are marked with /* pragma:symbols */ or
 724   /* pragma:unwind */ to indicate they need the specific data.
 725
 726 - Kernel backtraces can now be generated for non-pt_regs probe context
 727   if the kernel support dump_trace(). This enables backtraces from
 728   certain timer probes and tracepoints.
 729
 730 - ubacktrace() should now also work for some kernel probes on x86 which can
 731   use the dwarf unwinder to recover the user registers to provide
 732   more accurate user backtraces.
 733
 734 - For s390x the systemtap runtime now properly splits kernel and user
 735   addresses (which are in separate address spaces on that architecture)
 736   which enable user space introspection.
 737
 738 - ppc and s390x now supports user backtraces through the DWARF unwinder.
 739
 740 - ppc now handles function descriptors as symbol names correctly.
 741
 742 - arm support kernel backtraces through the DWARF unwinder.
 743
 744 - arm now have a uprobes port which enables user probes. This still
 745   requires some kernel patches (user_regsets and tracehook support for
 746   arm).
 747
 748 - Starting in release 1.7, these old variables will be deprecated:
 749   - The 'pid' variable in the 'kprocess.release' probe has been
 750     deprecated in favor of the new 'released_pid' variable.
 751   - The 'args' variable in the
 752     '_sunrpc.clnt.create_client.rpc_new_client_inline' probe has been
 753     deprecated in favor of the new internal-only '__args' variable.
 754
 755 - Experimental support for recent kernels without utrace has been
 756   added for the following probe types:
 757
 758     process(PID).begin
 759     process("PATH").begin
 760     process.begin
 761     process(PID).thread.begin
 762     process("PATH").thread.begin
 763     process.thread.begin
 764     process(PID).end
 765     process("PATH").end
 766     process.end
 767     process(PID).thread.end
 768     process("PATH").thread.end
 769     process.thread.end
 770     process(PID).syscall
 771     process("PATH").syscall
 772     process.syscall
 773     process(PID).syscall.return
 774     process("PATH").syscall.return
 775     process.syscall.return
 776
 777 - staprun disables kprobe-optimizations in recent kernels, as problems
 778   were found.  (PR13193)
 779
 780 * What's new in version 1.6, 2011-07-25
 781
 782 - Security fixes for CVE-2011-2503: read instead of mmap to load modules,
 783   CVE-2011-2502: Don't allow path-based auth for uprobes
 784
 785 - The systemtap compile-server no longer uses the -k option when calling the
 786   translator (stap). As a result, the server will now take advantage of the
 787   module cache when compiling the same script more than once. You may observe
 788   an improvement in the performance of the server in this situation.
 789
 790 - The systemtap compile-server and client now each check the version of the
 791   other, allowing both to adapt when communicating with a down-level
 792   counterpart. As a result, all version of the client can communicate
 793   with all versions of the server and vice-versa. Client will prefer newer
 794   servers when selecting a server automatically.
 795
 796 - SystemTap has improved support for the ARM architecture.  The
 797   kread() and kwrite() operations for ARM were corrected allowing many
 798   of the tapsets probes and function to work properly on the ARM
 799   architecture.
 800
 801 - Staprun can now rename the module to a unique name with the '-R' option before
 802   inserting it. Systemtap itself will also call staprun with '-R' by default.
 803   This allows the same module to be inserted more than once, without conflicting
 804   duplicate names.
 805
 806 - Systemtap error messages now provide feedback when staprun or any other
 807   process fails to launch.  This also specifically covers when the user
 808   doesn't have the proper permissions to launch staprun.
 809
 810 - Systemtap will now map - to _ in module names.  Previously,
 811   stap -L 'module("i2c-core").function("*")' would be empty.  It now returns
 812   a list had stap -L 'module("i2c_core").function("*") been specified.
 813
 814 - Systemtap now fills in missing process names to probe points, to
 815   avoid having to name them twice twice:
 816   % stap -e 'probe process("a.out").function("*") {}' -c 'a.out ...'
 817   Now the probed process name is inferred from the -c CMD argument.
 818   % stap -e 'probe process.function("*") {}' -c 'a.out ...'
 819
 820 - stap -L 'process("PATH").syscall' will now list context variables
 821
 822 - Depends on elfutils 0.142+.
 823
 824 - Deprecated task_backtrace:string (task:long). This function will go
 825   away after 1.6. Please run your scripts with stap --check-version.
 826
 827 * What's new in version 1.5, 2011-05-23
 828
 829 - Security fixes for CVE-2011-1781, CVE-2011-1769: correct DW_OP_{mod,div}
 830   division-by-zero bug
 831
 832 - The compile server and its related tools (stap-gen-ert, stap-authorize-cert,
 833    stap-sign-module) have been re-implemented in C++. Previously, these
 834    components were a mix of bash scripts and C code. These changes should be
 835    transparent to the end user with the exception of NSS certificate database
 836    password prompting (see below). The old implementation would prompt more
 837    than once for the same password in some situations.
 838
 839 - eventcount.stp now allows for event counting in the format of
 840   'stap eventcount.stp process.end syscall.* ...', and also reports
 841   corresponding event tid's.
 842
 843 - Systemtap checks that the build-id of the module being probed matches the
 844   build-id saved in the systemtap module.  Invoking systemtap with
 845   -DSTP_NO_BUILDID_CHECK will bypass this build-id runtime verification.  See
 846   man ld(1) for info on --build-id.
 847
 848 - stapio will now report if a child process has an abnormal exit along with
 849   the associated status or signal.
 850
 851 - Compiler optimization may sometimes result in systemtap not being able to
 852   access a user-space probe argument.  Compiling the application with
 853   -DSTAP_SDT_ARG_CONSTRAINT=nr will force the argument to be an immediate or
 854   register value which should enable systemtap to access the argument.
 855
 856 - GNU Gettext has now been intergrated with systemtap.  Our translation
 857   page can be found at http://www.transifex.net/projects/p/systemtap/ .
 858   "make update-po" will generate the necessary files to use translated
 859   messages.  Please refer to the po/README file for more info and
 860   please consider contributing to this I18N effort!
 861
 862 - The new addr() function returns the probe's instruction pointer.
 863
 864 - process("...").library("...") probes are now supported.  Wildcards
 865   are supported in the library-name part, to refer to any shared
 866   library that is required by process-name, which matches the glob
 867   pattern and the rest of the probe point.
 868
 869 - The "--remote USER@HOST" functionality can now be specified multiple times
 870   to fan out on multiple targets.  If the targets have distinct kernel and
 871   architecture configurations, stap will automatically build the script
 872   appropriately for each one.  This option is also no longer considered
 873   experimental.
 874
 875 - The NSS certificate database generated for use by the compile server is now
 876   generated with no password. Previously, a random password was generated and
 877   used to access the database. This change should be transparent to most users.
 878   However, if you are prompted for a password when using systemtap, then
 879   running $libexecdir/stap-gen-cert should correct the problem.
 880
 881 - The timestamp tapset includes jiffies() and HZ() for lightweight approximate
 882   timekeeping.
 883
 884 - A powerful new command line option --version has been added.
 885
 886 - process.mark now supports $$parms for reading probe parameters.
 887
 888 - A new command line option, --use-server-on-error[=yes|no] is available
 889   for stap.  It instructs stap to retry compilation of a script using a
 890   compile server if it fails on the local host.  The default setting
 891   is 'no'.
 892
 893 - The following deprecated tools have been removed:
 894       stap-client
 895       stap-authorize-server-cert
 896       stap-authorize-signing-cert
 897       stap-find-or-start-server
 898       stap-find-servers
 899   Use the --use-server, --trust-server and --list-servers options of stap
 900   instead.
 901
 902 * What's new in version 1.4, 2011-01-17
 903
 904 - Security fixes for CVE-2010-4170, CVE-2010-4171: staprun module
 905   loading/unloading
 906
 907 - A new /* myproc-unprivileged */ marker is now available for embedded C
 908   code and and expressions. Like the /* unprivileged */ marker, it makes
 909   the code or expression available for use in unprivileged mode (see
 910   --unprivileged). However, it also automatically adds a call to
 911   assert_is_myproc() to the code or expression, thus, making it available
 912   to the unprivileged user only if the target of the current probe is within
 913   the user's own process.
 914
 915 - The experimental "--remote USER@HOST" option will run pass 5 on a given
 916   ssh host, after building locally (or with --use-server) for that target.
 917
 918 - Warning messages from the script may now be suppressed with the stap
 919   and/or staprun -w option.  By default, duplicate warning messages are
 920   suppressed (up to a certain limit).  With stap --vp 00002 and above,
 921   the duplicate elimination is defeated.
 922
 923 - The print_ubacktrace and usym* functions attempt to print the full
 924   path of the user-space binaries' paths, instead of just the basename.
 925   The maximum saved path length is set by -DTASK_FINDER_VMA_ENTRY_PATHLEN,
 926   default 64.  Warning messages are produced if unwinding fails due to
 927   a missing 'stap -d MODULE' option, providing preloaded unwind data.
 928
 929 - The new tz_ctime() tapset function prints times in the local time zone.
 930
 931 - More kernel tracepoints are accessible to the kernel.trace("...") mechanism,
 932   if kernel source trees or debuginfo are available.  These formerly "hidden"
 933   tracepoints are those that are declared somewhere other than the usual
 934   include/linux/trace/ headers, such as xfs and kvm.
 935
 936 - debuginfo-based process("...").function/.statement/.mark probes support
 937   wildcards in the process-name part, to refer to any executable files that
 938   match the glob pattern and the rest of the probe point.
 939
 940 - The -t option now displays information per probe-point rather than a summary
 941   for each probe.  It also now shows the derivation chain for each probe-point.
 942
 943 - A rewrite of the sys/sdt.h header file provides zero-cost startup (few or
 944   no ELF relocations) for the debuginfo-less near-zero-cost runtime probes.
 945   Binaries compiled with earlier sdt.h versions remain supported.  The
 946   stap -L (listing) option now lists parameters for sys/sdt.h markers.
 947
 948 - The implementation of the integrated compile-server client has been
 949   extended.
 950   o --use-server now accepts an argument representing a particular server and
 951     may be specified more than once.
 952   o --list-servers now accepts an expanded range of arguments.
 953   o a new --trust-servers option has been added to stap to replace several
 954     old certificate-management scripts.
 955   o The following tools are now deprecated and will be removed in release 1.5:
 956       stap-client
 957       stap-authorize-server-cert
 958       stap-authorize-signing-cert
 959       stap-find-or-start-server
 960       stap-find-servers
 961   See man stap(1) for complete details.
 962
 963 - The compile-server now returns the uprobes.ko to the client when it is
 964   required by the script being compiled. The integrated compile-server client
 965   now makes it available to be loaded by staprun. The old (deprecated)
 966   stap-client does not do this.
 967
 968 - process probes with scripts as the target are recognized by stap and the
 969   interpreter would be selected for probing.
 970
 971 - Starting in release 1.5, these old variables/functions will be deprecated
 972   and will only be available when the '--compatible=1.4' flag is used:
 973
 974   - In the 'syscall.add_key' probe, the 'description_auddr' variable
 975     has been deprecated in favor of the new 'description_uaddr'
 976     variable.
 977   - In the 'syscall.fgetxattr', 'syscall.fsetxattr',
 978     'syscall.getxattr', 'syscall.lgetxattr', and
 979     'syscall.lremovexattr' probes, the 'name2' variable has been
 980     deprecated in favor of the new 'name_str' variable.
 981   - In the 'nd_syscall.accept' probe the 'flag_str' variable
 982     has been deprecated in favor of the new 'flags_str' variable.
 983   - In the 'nd_syscall.dup' probe the 'old_fd' variable has been
 984     deprecated in favor of the new 'oldfd' variable.
 985   - In the 'nd_syscall.fgetxattr', 'nd_syscall.fremovexattr',
 986     'nd_syscall.fsetxattr', 'nd_syscall.getxattr', and
 987     'nd_syscall.lremovexattr' probes, the 'name2' variable has been
 988     deprecated in favor of the new 'name_str' variable.
 989   - The tapset alias 'nd_syscall.compat_pselect7a' was misnamed.  It should
 990     have been 'nd_syscall.compat_pselect7' (without the trailing 'a').
 991   - The tapset function 'cpuid' is deprecated in favor of the better known
 992     'cpu'.
 993   - In the i386 'syscall.sigaltstack' probe, the 'ussp' variable has
 994     been deprecated in favor of the new 'uss_uaddr' variable.
 995   - In the ia64 'syscall.sigaltstack' probe, the 'ss_uaddr' and
 996     'oss_uaddr' variables have been deprecated in favor of the new
 997     'uss_uaddr' and 'uoss_uaddr' variables.
 998   - The powerpc tapset alias 'syscall.compat_sysctl' was deprecated
 999     and renamed 'syscall.sysctl32'.
1000   - In the x86_64 'syscall.sigaltstack' probe, the 'regs_uaddr'
1001     variable has been deprecated in favor of the new 'regs' variable.
1002
1003 * What's new in version 1.3, 2010-07-21
1004
1005 - The uprobes kernel module now has about half the overhead when probing
1006   NOPs, which is particularly relevant for sdt.h markers.
1007
1008 - New stap option -G VAR=VALUE allows overriding global variables
1009   by passing the settings to staprun as module options.
1010
1011 - The tapset alias 'syscall.compat_pselect7a' was misnamed.  It should
1012   have been 'syscall.compat_pselect7' (without the trailing 'a').
1013   Starting in release 1.4, the old name will be deprecated and
1014   will only be available when the '--compatible=1.3' flag is used.
1015
1016 - A new procfs parameter .umask(UMASK) which provides modification of
1017   file permissions using the proper umask value.  Default file
1018   permissions for a read probe are 0400, 0200 for a write probe, and
1019   0600 for a file with a read and write probe.
1020
1021 - It is now possible in some situations to use print_ubacktrace() to
1022   get a user space stack trace from a kernel probe point. e.g. for
1023   user backtraces when there is a pagefault:
1024   $ stap -d /bin/sort --ldd -e 'probe vm.pagefault {
1025       if (pid() == target()) {
1026          printf("pagefault @0x%x\n", address); print_ubacktrace();
1027       } }' -c /bin/sort
1028   [...]
1029   pagefault @0x7fea0595fa70
1030    0x000000384f07f958 : __GI_strcmp+0x12b8/0x1440 [libc-2.12.so]
1031    0x000000384f02824e : __gconv_lookup_cache+0xee/0x5a0 [libc-2.12.so]
1032    0x000000384f021092 : __gconv_find_transform+0x92/0x2cf [libc-2.12.so]
1033    0x000000384f094896 : __wcsmbs_load_conv+0x106/0x2b0 [libc-2.12.so]
1034    0x000000384f08bd90 : mbrtowc+0x1b0/0x1c0 [libc-2.12.so]
1035    0x0000000000404199 : ismbblank+0x39/0x90 [sort]
1036    0x0000000000404a4f : inittables_mb+0xef/0x290 [sort]
1037    0x0000000000406934 : main+0x174/0x2510 [sort]
1038    0x000000384f01ec5d : __libc_start_main+0xfd/0x1d0 [libc-2.12.so]
1039    0x0000000000402509 : _start+0x29/0x2c [sort]
1040   [...]
1041
1042 - New tapset functions to get a string representation of a stack trace:
1043   sprint_[u]backtrace() and sprint_[u]stack().
1044
1045 - New tapset function to get the module (shared library) name for a
1046   user space address umodname:string(long). The module name will now
1047   also be in the output of usymdata() and in backtrace addresses even
1048   when they were not given with -d at the command line.
1049
1050 - Kernel backtraces are now much faster (replaced a linear search
1051   with a binary search).
1052
1053 - A new integrated compile-server client is now available as part of stap.
1054
1055   o 'stap --use-server ...' is equivalent to 'stap-client ...'
1056   o 'stap --list-servers' is equivalent to 'stap-find-servers'
1057   o 'stap --list-servers=online' is equivalent to 'stap-find-servers --all'
1058   o stap-client and its related tools will soon be deprecated.
1059   o the nss-devel and avahi-devel packages are required for building stap with
1060     the integrated client (checked during configuration).
1061   o nss and avahi are required to run the integrated client.
1062
1063 - A new operator @entry is available for automatically saving an expression
1064   at entry time for use in a .return probe.
1065    probe foo.return { println(get_cycles() - @entry(get_cycles())) }
1066
1067 - Probe $target variables and @cast() can now use a suffix to print complex
1068   data types as strings.  Use a single '$' for a shallow view, or '$$' for a
1069   deeper view that includes nested types.  For example, with fs_struct:
1070    $fs$ : "{.users=%i, .lock={...}, .umask=%i,
1071             .in_exec=%i, .root={...}, .pwd={...}}"
1072    $fs$$ : "{.users=%i, .lock={.raw_lock={.lock=%u}}, .umask=%i, .in_exec=%i,
1073              .root={.mnt=%p, .dentry=%p}, .pwd={.mnt=%p, .dentry=%p}}"
1074
1075 - The <sys/sdt.h> user-space markers no longer default to an implicit
1076   MARKER_NAME_ENABLED() semaphore check for each marker.  To check for
1077   enabled markers use a .d declaration file, then:
1078      if (MARKER_NAME_ENABLED()) MARKER_NAME()
1079
1080 - Hyphenated <sys/sdt.h> marker names such as process(...).mark("foo-bar")
1081   are now accepted in scripts.  They are mapped to the double-underscore
1082   form ("foo__bar").
1083
1084 - More robust <sys/sdt.h> user-space markers support is included.  For
1085   some platforms (x86*, ppc*), this can let systemtap probe the markers
1086   without debuginfo.  This implementation also supports preserving
1087   the "provider" name associated with a marker:
1088     probe process("foo").provider("bar").mark("baz") to match
1089     STAP_PROBE<n>(bar, baz <...>)
1090   (Compile with -DSTAP_SDT_V1 to revert to the previous implementation.
1091   Systemtap supports pre-existing or new binaries using them.)
1092
1093 - Embedded-C may be used within expressions as values, when in guru mode:
1094      num = %{ LINUX_VERSION_CODE %}               // int64_t
1095      name = %{ /* string */ THIS_MODULE->name %}  // const char*
1096      printf ("%s %x\n", name, num)
1097   The usual /* pure */, /* unprivileged */, and /* guru */ markers may be used
1098   as with embedded-C functions.
1099
1100 - By default the systemtap-runtime RPM builds now include a shared
1101   library, staplog.so, that allows crash to extract systemtap data from
1102   a vmcore image.
1103
1104 - Iterating with "foreach" can now explicitly save the value for the loop.
1105      foreach(v = [i,j] in array)
1106        printf("array[%d,%s] = %d\n", i, j, v /* array[i,j] */)
1107
1108 - The new "--ldd" option automatically adds any additional shared
1109   libraries needed by probed or -d-listed userspace binaries to the -d
1110   list, to enable symbolic backtracing through them.  Similarly, the
1111   new "--all-modules" option automatically adds any currently loaded
1112   kernel modules (listed in /proc/modules) to the -d list.
1113
1114 - A new family of set_kernel_* functions make it easier for gurus to write
1115   new values at arbitrary memory addresses.
1116
1117 - Probe wildcards can now use '**' to cross the '.' separator.
1118      $ stap -l 'sys**open'
1119      syscall.mq_open
1120      syscall.open
1121
1122 - Backward compatibility flags (--compatible=VERSION, and matching
1123   script preprocessing predicate %( systemtap_v CMP "version" %)
1124   and a deprecation policy are being introduced, in case future
1125   tapset/language changes break valid scripts.
1126
1127 * What's new in version 1.2, 2010-03-22
1128
1129 - Prototype support for "perf events", where the kernel supports the
1130   2.6.33 in-kernel API.  Probe points may refer to low-level
1131   perf_event_attr type/config numbers, or to a number of aliases
1132   defined in the new perf.stp tapset:
1133      probe perf.sw.cpu_clock, perf.type(0).config(4) { }
1134
1135 - Type-casting can now use multiple headers to resolve codependencies.
1136      @cast(task, "task_struct",
1137            "kernel<linux/sched.h><linux/fs_struct.h>")->fs->umask
1138
1139 - Tapset-related man pages have been renamed.  'man -k 3stap' should show
1140   the installed list, which due to prefixing should no longer collide over
1141   ordinary system functions.
1142
1143 - User space marker arguments no longer use volatile if the version of gcc,
1144   which must be at least 4.5.0, supports richer DWARF debuginfo.  Use cflags
1145   -DSTAP_SDT_VOLATILE=volatile or -DSTAP_SDT_VOLATILE= when building
1146   the sys/sdt.h application to override this one way or another.
1147
1148 - A new construct for error handling is available.  It is similar to c++
1149   exception catching, using try and catch as new keywords.  Within a handler
1150   or function, the following is valid and may be nested:
1151      try { /* arbitrary statements */ }
1152      catch (er) { /* e.g. println("caught error ", er) */ }
1153
1154 - A new command line flag '-W' forces systemtap to abort translation of
1155   a script if any warnings are produced.  It is similar to gcc's -Werror.
1156   (If '-w' is also supplied to suppress warnings, it wins.)
1157
1158 - A new predicate @defined is available for testing whether a
1159   particular $variable/expression is resolvable at translate time:
1160   probe foo { if (@defined($bar)) log ("$bar is available here") }
1161
1162 - Adjacent string literals are glued together, making this
1163   construct valid:
1164      probe process("/usr" @1 "/bin").function("*") { ... }
1165
1166 - In order to limit potential impact from future security problems,
1167   the stap-server process does not permit its being launched as root.
1168
1169 - On recent kernels, for some architectures/configurations, hardware
1170   breakpoint probes are supported.  The probe point syntax is:
1171
1172      probe kernel.data(ADDRESS).write
1173      probe kernel.data(ADDRESS).length(LEN).write
1174      probe kernel.data("SYMBOL_NAME").write
1175
1176 * What's new in version 1.1, 2010-01-15
1177
1178 - New tracepoint based tapset for memory subsystem.
1179
1180 - The loading of signed modules by staprun is no longer allowed for
1181   ordinary, unprivileged users.  This means that only root, members of
1182   the group 'stapdev' and members of the group 'stapusr' can load
1183   systemtap modules using staprun, stap or stap-client.  The minimum
1184   privilege required to run arbitrary --unprivileged scripts is now
1185   'stapusr' membership.
1186
1187 - The stap-server initscript is available. This initscript allows you
1188   to start systemtap compile servers as a system service and to manage
1189   these servers as a group or individually. The stap-server initscript
1190   is installed by the systemtap-server rpm.  The build directory for
1191   the uprobes module (/usr/share/systemtap/runtime/uprobes) is made
1192   writable by the 'stap-server' group. All of the files generated when
1193   building the uprobes module, including the digital signature, are
1194   also writable by members of stap-server.
1195
1196   See initscript/README.stap-server for details.
1197
1198 - Some of the compile server client, server and certificate management
1199   tools have been moved from $bindir to $libexecdir/systemtap.
1200   You should use the new stap-server script or the stap-server initscript
1201   for server management where possible. The stap-server script provides the same
1202   functionality as the stap-server initscript except that the servers are
1203   run by the invoking user by default as opposed to servers started by the
1204   stap-server initscript which are run by the user stap-server
1205   by default. See stap-server(8) for more information.
1206
1207   You may continue to use these tools by adding $libexecdir/systemtap to
1208   your path. You would need to do this, for example, if you are not root,
1209   you want to start a compile server and you are not running systemtap from a
1210   private installation. In this case you still need to use stap-start-server.
1211
1212 - Any diagnostic output line that starts with "ERROR", as in
1213   error("foo"), will promote a "Pass 5: run failed", and the return
1214   code is 1.
1215
1216 - Systemtap now warns about global variables being referenced from other
1217   script files.  This aims to protect against unintended local-vs-global
1218   namespace collisions such as:
1219
1220      % cat some_tapset.stp
1221      probe baz.one = bar { foo = $foo; bar = $bar }
1222      % cat end_user_script.stp
1223      global foo # intended to be private variable
1224      probe timer.s(1) { foo ++ }
1225      probe baz.* { println(foo, pp()) }
1226      % stap end_user_script.stp
1227      WARNING: cross-file global variable reference to foo from some_tapset.stp
1228
1229 - Preprocessor conditional for kernel configuration testing:
1230   %( CONFIG_foo == "y" %? ... %)
1231
1232 - ftrace(msg:string) tapset function to send strings to the system-wide
1233   ftrace ring-buffer (if any).
1234
1235 - Better support for richer DWARF debuginfo output from GCC 4.5
1236   (variable tracking assignments). Kernel modules are now always resolved
1237   against all their dependencies to find any info referring to missing
1238   symbols. DW_AT_const_value is now supported when no DW_AT_location
1239   is available.
1240
1241 * What's new in verson 1.0, 2009-09-22
1242
1243 - process().mark() probes now use an enabling semaphore to reduce the
1244   computation overhead of dormant probes.
1245
1246 - The function spec for dwarf probes now supports C++ scopes, so you can
1247   limit the probes to specific namespaces or classes.  Multiple scopes
1248   can be specified, and they will be matched progressively outward.
1249       probe process("foo").function("std::vector<*>::*") { }
1250       probe process("foo").function("::global_function") { }
1251
1252 - It is now possible to cross-compile systemtap scripts for foreign
1253   architectures, using the new '-a ARCH' and '-B OPT=VALUE' flags.
1254   For example, put arm-linux-gcc etc. into your $PATH, and point
1255   systemtap at the target kernel build tree with:
1256      stap -a arm -B CROSS_COMPILE=arm-linux- -r /build/tree  [...]
1257   The -B option is passed to kbuild make.  -r identifies the already
1258   configured/built kernel tree and -a its architecture (kbuild ARCH=...).
1259   Systemtap will infer -p4.
1260
1261 - Cross compilation using the systemtap client and server
1262   - stap-start-server now accepts the -r, -R, -I, -B and -a options in
1263     order to start a cross compiling server. The server will correctly
1264     advertise itself with respect to the kernel release and architecture
1265     that it compiles for.
1266   - When specified on stap-client, the -r and -a options will be
1267     considered when searching for a suitable server.
1268
1269 - When using the systemtap client and server udp port 5353 must be open
1270   in your firewall in order for the client to find servers using
1271   avahi-browse.  Also the systemtap server will choose a random port in
1272   the range 1024-63999 for accepting ssl connections.
1273
1274 - Support for unprivileged users:
1275   ***********************************************************************
1276   * WARNING!!!!!!!!!!                                                   *
1277   * This feature is EXPERIMENTAL at this time and should be used with   *
1278   * care. This feature allows systemtap kernel modules to be loaded by  *
1279   * unprivileged users. The user interface and restrictions will change *
1280   * as this feature evolves.                                            *
1281   ***********************************************************************
1282   - Systemtap modules generated from scripts which use a restricted
1283     subset of the features available may be loaded by staprun for
1284     unprivileged users. Previously, staprun would load modules only for
1285     root or for members of the groups stapdev and stapusr.
1286   - Using the --unprivileged option on stap enables translation-time
1287     checking for use by unprivileged users (see restrictions below).
1288   - All modules deemed suitable for use by unprivileged users will be
1289     signed by the systemtap server when --unprivileged is specified on
1290     stap-client. See module signing in release 0.9.8 and stap-server in
1291     release 0.9 below.
1292   - Modules signed by trusted signers (servers) and verified by staprun
1293     will be loaded by staprun regardless of the user's privilege level.
1294   - The system administrator asserts the trustworthiness of a signer
1295     (server) by running stap-authorize-signing-cert <cert-file> as root,
1296     where the <cert-file> can be found in
1297     ~<user>/.systemtap/ssl/server/stap.cert for servers started by
1298     ordinary users and in $sysconfdir/systemtap/ssl/server/stap.cert for
1299     servers started by root.
1300   - Restrictions are intentionally strict at this time and may be
1301     relaxed in the future:
1302      - probe points are restricted to:
1303          begin, begin(n), end, end(n), error, error(n), never,
1304          timer.{jiffies,s,sec,ms,msec,us,usec,ns,nsec}(n)*, timer.hz(n),
1305          process.* (for processes owned by the user).
1306      - use of embedded C code is not allowed.
1307      - use of tapset functions is restricted.
1308        - some tapset functions may not be used at all. A message will be
1309          generated at module compilation time.
1310        - some actions by allowed tapset functions may only be performed
1311          in the context of the user's own process. A runtime fault will
1312          occur in these situations, for example, direct memory access.
1313        - The is_myproc() tapset function has been provided so that
1314          tapset writers for unprivileged users can check that the
1315          context is of the users own process before attempting these
1316          actions.
1317      - accessing the kernel memory space is not allowed.
1318      - The following command line options may not be used by stap-client
1319        -g, -I, -D, -R, -B
1320      - The following environment variables are ignored by stap-client:
1321        SYSTEMTAP_RUNTIME, SYSTEMTAP_TAPSET, SYSTEMTAP_DEBUGINFO_PATH
1322   - nss and nss-tools are required to use this feature.
1323
1324 - Support output file switching by SIGUSR2. Users can command running
1325   stapio to switch output file by sending SIGUSR2.
1326
1327 - Memory consumption for scripts involving many uprobes has been
1328   dramatically reduced.
1329
1330 - The preprocessor now supports || and && in the conditions.
1331   e.g. %( arch == "x86_64" || arch == "ia64" %: ... %)
1332
1333 - The systemtap notion of "architecture" now matches the kernel's, rather
1334   than that of "uname -m".  This means that 32-bit i386 family are all
1335   known as "i386" rather than "i386" or "i686"; "ppc64" as "powerpc";
1336   "s390x" as "s390", and so on.  This is consistent between the new
1337   "-a ARCH" flag and the script-level %( arch ... %) conditional.
1338
1339 - It is now possible to define multiple probe aliases with the same name.
1340   A probe will expand to all matching aliases.
1341     probe foo = bar { }
1342     probe foo = baz { }
1343     probe foo { } # expands twice, once to bar and once to baz
1344
1345 - A new experimental transport mechanism, using ftrace's ring_buffer,
1346   has been added.  This may become the default transport mechanism in
1347   future versions of systemtap.  To test this new transport mechanism,
1348   define 'STP_USE_RING_BUFFER'.
1349
1350 - Support for recognizing DW_OP_{stack,implicit}_value DWARF expressions
1351   as emitted by GCC 4.5.
1352
1353 * What's new in version 0.9.9, 2009-08-04
1354
1355 - Systemwide kernel .function.return (kretprobe) maxactive defaults may
1356   be overridden with the -DKRETACTIVE=nnn parameter.
1357
1358 - Translation pass 2 is significantly faster by avoiding unnecessary
1359   searching through a kernel build/module directory tree.
1360
1361 - When compiled against elfutils 0.142 systemtap now handles the new
1362   DW_OP_call_frame_CFA generated by by GCC.
1363
1364 - uprobes and ustack() are more robust when used on applications that
1365   depend on prelinked/separate debuginfo shared libraries.
1366
1367 - User space PROBE marks are not always found with or without separate
1368   debuginfo. The .probes section itself is now always put in the main
1369   elf file and marked as allocated. When building pic code the section
1370   is marked writable. The selinux memory check problems seen with
1371   programs using STAP_PROBES is fixed.
1372
1373 - statement() probes can now override "address not at start of statement"
1374   errors in guru mode. They also provide alternative addresses to use
1375   in non-guru mode.
1376
1377 - The stapgraph application can generate graphs of data and events
1378   emitted by systemtap scripts in real time.  Run "stapgraph
1379   testsuite/systemtap.examples/general/grapher.stp" for an example of
1380   graphing the system load average and keyboard events.
1381
1382 - Dwarf probes now show parameters and local variables in the verbose
1383   listing mode (-L).
1384
1385 - Symbol aliases are now resolved to their canonical dwarf names.  For
1386   example, probing "malloc" in libc resolves to "__libc_malloc".
1387
1388 - The syntax for dereferencing $target variables and @cast() gained new
1389   capabilities:
1390   - Array indexes can now be arbitrary numeric expressions.
1391   - Array subscripts are now supported on pointer types.
1392   - An '&' operator before a @cast or $target returns the address of the
1393     final component, especially useful for nested structures.
1394
1395 - For reading all probe variables, kernel.mark now supports $$vars and
1396   $$parms, and process.syscall now supports $$vars.
1397
1398 - The SNMP tapset provides probes and functions for many network
1399   statistics.  See stapprobes.snmp(3stap) for more details.
1400
1401 - The dentry tapset provides functions to map kernel VFS directory entries
1402   to file or full path names: d_path(), d_name() and reverse_path_walk().
1403
1404 - SystemTap now has userspace markers in its own binaries, and the stap
1405   tapset provides the available probepoints and local variables.
1406
1407 - Miscellaneous new tapset functions:
1408   - pgrp() returns the process group ID of the current process
1409   - str_replace() performs string replacement
1410
1411 * What's new in version 0.9.8, 2009-06-11
1412
1413 - Miscellaneous new tapset functions:
1414   - sid() returns the session ID of the current process
1415   - stringat() indexes a single character from a string.
1416
1417 - Using %M in print formats for hex dumps can now print entire buffers,
1418   instead of just small numbers.
1419
1420 - Dwarfless syscalls: The nd_syscalls tapset is now available to probe
1421   system calls without requiring kernel debugging information.  All of
1422   the same probepoints in the normal syscalls tapset are available with
1423   an "nd_" prefix, e.g. syscall.open becomes nd_syscall.open.  Most
1424   syscall arguments are also available by name in nd_syscalls.
1425
1426 - Module signing: If the appropriate nss libraries are available on your
1427   system, stap-server will sign each compiled module using a self-generated
1428   certificate.  This is the first step toward extending authority to
1429   load certain modules to unprivileged users. For now, if the system
1430   administrator adds a certificate to a database of trusted signers
1431   (stap-authorize-signing-cert), modules signed using that certificate
1432   will be verified by staprun against tampering.  Otherwise, you should
1433   notice no difference in the operation of stap or staprun.
1434
1435 * What's new in version 0.9.7, 2009-04-23
1436
1437 - @cast can now determine its type information using an explicit header
1438   specification.  For example:
1439     @cast(tv, "timeval", "<sys/time.h>")->tv_sec
1440     @cast(task, "task_struct", "kernel<linux/sched.h>")->tgid
1441
1442 - The overlapping process.* tapsets are now separated.  Those probe points
1443   documented in stapprobes(3stap) remain the same.  Those that were formerly
1444   in stapprobes.process(3stap) have been renamed to kprocess, to reflect
1445   their kernel perspective on processes.
1446
1447 - The --skip-badvars option now also suppresses run-time error
1448   messages that would otherwise result from erroneous memory accesses.
1449   Such accesses can originate from $context expressions fueled by
1450   erroneous debug data, or by kernel_{long,string,...}() tapset calls.
1451
1452 - New probes kprobe.function(FUNCTION) and kprobe.function(FUNCTION).return
1453   for dwarfless probing. These postpone function address resolution to
1454   run-time and use the kprobe symbol-resolution mechanism.
1455   Probing of absolute statements can be done using the
1456   kprobe.statement(ADDRESS).absolute construct.
1457
1458 - EXPERIMENTAL support for user process unwinding. A new collection of
1459   tapset functions have been added to handle user space backtraces from
1460   probe points that support them (currently process and timer probes -
1461   for timer probes test whether or not in user space first with the
1462   already existing user_mode() function). The new tapset functions are:
1463     uaddr - User space address of current running task.
1464     usymname - Return the symbol of an address in the current task.
1465     usymdata - Return the symbol and module offset of an address.
1466     print_ustack - Print out stack for the current task from string.
1467     print_ubacktrace - Print stack back trace for current task.
1468     ubacktrace - Hex backtrace of current task stack.
1469   Please read http://sourceware.org/ml/systemtap/2009-q2/msg00364.html
1470   on the current restrictions and possible changes in the future and
1471   give feedback if you want to influence future developments.
1472
1473 * What's new in version 0.9.5, 2009-03-27
1474
1475 - New probes process().insn and process().insn.block that allows
1476   inspection of the process after each instruction or block of
1477   instructions executed. So to count the total number of instructions
1478   a process executes during a run do something like:
1479     $ stap -e 'global steps; probe process("/bin/ls").insn {steps++}
1480                probe end {printf("Total instructions: %d\n", steps);}' \
1481            -c /bin/ls
1482   This feature can slow down execution of a process somewhat.
1483
1484 - Systemtap probes and function man pages extracted from the tapsets
1485   are now available under 3stap. To show the page for probe vm.pagefault
1486   or the stap function pexecname do:
1487     $ man 3stap vm.pagefault
1488     $ man 3stap pexecname
1489
1490 - Kernel tracepoints are now supported for probing predefined kernel
1491   events without any debuginfo.  Tracepoints incur less overhead than
1492   kprobes, and context parameters are available with full type
1493   information.  Any kernel 2.6.28 and later should have defined
1494   tracepoints.  Try the following to see what's available:
1495     $ stap -L 'kernel.trace("*")'
1496
1497 - Typecasting with @cast now supports modules search paths, which is
1498   useful in case there are multiple places where the type definition
1499   may be found.  For example:
1500     @cast(sdev, "scsi_device", "kernel:scsi_mod")->sdev_state
1501
1502 - On-file flight recorder is supported. It allows stap to record huge
1503   trace log on the disk and to run in background.
1504   Passing -F option with -o option runs stap in background mode. In this
1505   mode, staprun is detached from console, and stap itself shows staprun's
1506   pid and exits.
1507   Specifying the max size and the max number of log files are also available
1508   by passing -S option. This option has one or two arguments seperated by
1509   a comma. The first argument is the max size of a log file in MB. If the
1510   size of a log file exceeds it, stap switches to the next log file
1511   automatically. The second is how many files are kept on the disk. If the
1512   number of log files exceeds it, the oldest log file is removed
1513   automatically. The second argument can be omitted.
1514
1515   For example, this will record output on log files each of them is smaller
1516   than 1024MB and keep last 3 logs, in background.
1517     % stap -F -o /tmp/staplog -S 1024,3 script.stp
1518
1519 - In guru mode (-g), the kernel probing blacklist is disabled, leaving
1520   only a subset - the kernel's own internal kprobe blacklist - to attempt
1521   to filter out areas unsafe to probe.  The differences may be enough to
1522   probe more interrupt handlers.
1523
1524 - Variables unavailable in current context may be skipped by setting a
1525   session level flag with command line option --skip-badvars now available.
1526   This replaces any dwarf $variable expressions that could not be resolved
1527   with literal numeric zeros, along with a warning message.
1528
1529 - Both kernel markers and kernel tracepoint support argument listing
1530   through stap -L 'kernel.mark("*")' or stap -L 'kernel.trace("*")'
1531
1532 - Users can use -DINTERRUPTIBLE=0 to prevent interrupt reentrancy in
1533   their script, at the cost of a bit more overhead to toggle the
1534   interrupt mask.
1535
1536 - Added reentrancy debugging. If stap is run with the arguments
1537   "-t -DDEBUG_REENTRANCY", additional warnings will be printed for
1538   every reentrancy event, including the probe points of the
1539   resident and interloper probes.
1540
1541 - Default to --disable-pie for configure.
1542   Use --enable-pie to turn it back on.
1543
1544 - Improved sdt.h compatibility and test suite for static dtrace
1545   compatible user space markers.
1546
1547 - Some architectures now use syscall wrappers (HAVE_SYSCALL_WRAPPERS).
1548   The syscall tapset has been enhanced to take care of the syscall
1549   wrappers in this release.
1550
1551 - Security fix for CVE-2009-0784: stapusr module-path checking race.
1552
1553 * What's new in version 0.9, 2009-02-19
1554
1555 - Typecasting is now supported using the @cast operator.  A script can
1556   define a pointer type for a "long" value, and then access type members
1557   using the same syntax as with $target variables.  For example, this will
1558   retrieve the parent pid from a kernel task_struct:
1559     @cast(pointer, "task_struct", "kernel")->parent->pid
1560
1561 - process().mark() probes are now possible to trace static user space
1562   markers put in programs with the STAP_PROBE macro using the new
1563   sys/sdt.h include file. This also provides dtrace compatible markers
1564   through DTRACE_PROBE and an associated python 'dtrace' script that
1565   can be used in builds based on dtrace that need dtrace -h or -G
1566   functionality.
1567
1568 - For those that really want to run stap from the build tree there is
1569   now the 'run-stap' script in the top-level build directory that sets
1570   up the SYSTEMTAP_TAPSET, SYSTEMTAP_RUNTIME, SYSTEMTAP_STAPRUN, and
1571   SYSTEMTAP_STAPIO environment variables (installing systemtap, in a
1572   local prefix, is still recommended for common use).
1573
1574 - Systemtap now comes with a new Beginners Guide that walks the user
1575   through their first steps setting up stap, understanding how it all
1576   works, introduces some useful scripts and describes some common
1577   pitfalls.  It isn't created by default since it needs a Publican
1578   setup, but full build instructions can be found in the wiki:
1579   http://sourceware.org/systemtap/wiki/PublicanQuikHowto
1580   An online version can be found at:
1581   http://sourceware.org/systemtap/SystemTap_Beginners_Guide.pdf
1582
1583 - Standard tapsets included with Systemtap were modified to include
1584   extractable documentation information based on the kernel-doc
1585   infrastructure. When configured --enabled-docs a HTML and PDF
1586   version of the Tapset Reference Manual is produced explaining probes
1587   defined in each tapset.
1588
1589 - The systemtap client and compile server are now available.
1590   These allow you to compile a systemtap module on a host other than
1591   the one which it will be run, providing the client and server
1592   are compatible. Other than using a server for passes 1 through
1593   4, the client behaves like the 'stap' front end itself. This
1594   means, among other things, that the client will automatically
1595   load the resulting module on the local host unless -p[1234]
1596   was specified.  See stap-server(8) for more details.
1597   The client/server now use SSL for network connection security and
1598   for signing.
1599
1600   The systemtap client and server are prototypes only. Interfaces, options
1601   and usage may change at any time.
1602
1603 - function("func").label("label") probes are now supported to allow matching
1604   the label of a function.
1605
1606 - Systemtap initscript is available. This initscript allows you to run
1607   systemtap scripts as system services (in flight recorder mode) and
1608   control those scripts individually.
1609   See README.systemtap for details.
1610
1611 - The stap "-r DIR" option may be used to identify a hand-made kernel
1612   build directory.  The tool determines the appropriate release string
1613   automatically from the directory.
1614
1615 - Serious problems associated with user-space probing in shared libraries
1616   were corrected, making it now possible to experiment with probe shared
1617   libraries.  Assuming dwarf debugging information is installed, use this
1618   twist on the normal syntax:
1619
1620     probe process("/lib64/libc-2.8.so").function("....") { ... }
1621
1622   This would probe all threads that call into that library.  Running
1623   "stap -c CMD" or "stap -x PID" naturally restricts this to the target
1624   command+descendants only.  $$vars etc. may be used.
1625
1626 - For scripts that sometimes terminate with excessive "skipped" probes,
1627   rerunning the script with "-t" (timing) will print more details about
1628   the skippage reasons.
1629
1630 - Symbol tables and unwind (backtracing) data support were formerly
1631   compiled in for all probed modules as identified by the script
1632   (kernel; module("name"); process("file")) plus those listed by the
1633   stap "-d BINARY" option.  Now, this data is included only if the systemtap
1634   script uses tapset functions like probefunc() or backtrace() that require
1635   such information.  This shrinks the probe modules considerably for the rest.
1636
1637 - Per-pass verbosity control is available with the new "--vp {N}+" option.
1638   "stap --vp 040" adds 4 units of -v verbosity only to pass 2.  This is useful
1639   for diagnosing errors from one pass without excessive verbosity from others.
1640
1641 - Most probe handlers now run with interrupts enabled, for improved
1642   system responsiveness and less probing overhead.  This may result
1643   in more skipped probes, for example if a reentrant probe handler
1644   is attempted from within an interrupt handler.  It may also make the
1645   systemtap overload detection facility more likely to be triggered, as
1646   interrupt handlers' run time would be included in the self-assessed
1647   overhead of running probe handlers.
1648
1649 * What's new in version 0.8, 2008-11-13
1650
1651 - Cache limiting is now available.  If the compiled module cache size is
1652   over a limit specified in the $SYSTEMTAP_DIR/cache/cache_mb_limit file,
1653   some old cache entries will be unlinked.  See man stap(1) for more.
1654
1655 - Error and warning messages are now followed by source context displaying
1656   the erroneous line/s and a handy '^' in the following line pointing to the
1657   appropriate column.
1658
1659 - A bug reporting tool "stap-report" is now available which will quickly
1660   retrieve much of the information requested here:
1661   http://sourceware.org/systemtap/wiki/HowToReportBugs
1662
1663 - The translator can resolve members of anonymous structs / unions:
1664     given            struct { int foo; struct { int bar; }; } *p;
1665     this now works:  $p->bar
1666
1667 - The stap "-F" flag activates "flight recorder" mode, which consists of
1668   translating the given script as usual, but implicitly launching it into
1669   the background with staprun's existing "-L" (launch) option.  A user
1670   can later reattach to the module with "staprun -A MODULENAME".
1671
1672 - Additional context variables are available on user-space syscall probes.
1673   - $argN ($arg1, $arg2, ... $arg6) in process(PATH_OR_PID).syscall
1674     gives you the argument of the system call.
1675   - $return in process(PATH_OR_PID).syscall.return gives you the return
1676     value of the system call.
1677
1678 - Target process mode (stap -c CMD or -x PID) now implicitly restricts all
1679   "process.*" probes to the given child process.  (It does not affect
1680   kernel.* or other probe types.)  The CMD string is normally run directly,
1681   rather than via a /bin/sh -c subshell, since then utrace/uprobe probes
1682   receive a fairly "clean" event stream.  If metacharacters like
1683   redirection operators were present in CMD, then "sh -c CMD" is still
1684   used, and utrace/uprobe probes will receive events from the shell.
1685
1686      % stap -e 'probe process.syscall, process.end {
1687                    printf("%s %d %s\n", execname(), pid(), pp())}'\
1688             -c ls
1689      ls 2323 process.syscall
1690      ls 2323 process.syscall
1691      ls 2323 process.end
1692
1693 - Probe listing mode is improved: "-L" lists available script-level variables
1694
1695      % stap -L 'syscall.*open*'
1696      syscall.mq_open name:string name_uaddr:long filename:string mode:long u_attr_uaddr:long oflag:long argstr:string
1697      syscall.open name:string filename:string flags:long mode:long argstr:string
1698      syscall.openat name:string filename:string flags:long mode:long argstr:string
1699
1700 - All user-space-related probes support $PATH-resolved executable
1701   names, so
1702
1703      probe process("ls").syscall {}
1704      probe process("./a.out").syscall {}
1705
1706   work now, instead of just
1707
1708      probe process("/bin/ls").syscall {}
1709      probe process("/my/directory/a.out").syscall {}
1710
1711 - Prototype symbolic user-space probing support:
1712
1713      # stap -e 'probe process("ls").function("*").call {
1714                    log (probefunc()." ".$$parms)
1715                 }' \
1716             -c 'ls -l'
1717
1718   This requires:
1719   - debugging information for the named program
1720   - a version of utrace in the kernel that is compatible with the "uprobes"
1721     kernel module prototype.  This includes RHEL5 and older Fedora, but not
1722     yet current lkml-track utrace; a "pass 4a"-time build failure means
1723     your system cannot use this yet.
1724
1725 - Global variables which are written to but never read are now
1726   automatically displayed when the session does a shutdown.  For example:
1727
1728       global running_tasks
1729       probe timer.profile {running_tasks[pid(),tid()] = execname()}
1730       probe timer.ms(8000) {exit()}
1731
1732 - A formatted string representation of the variables, parameters, or local
1733   variables at a probe point is now supported via the special $$vars,
1734   $$parms, and $$locals context variables, which expand to a string
1735   containing a list "var1=0xdead var2=0xbeef var3=?".  (Here, var3 exists
1736   but is for some reason unavailable.)  In return probes only, $$return
1737   expands to an empty string for a void function, or "return=0xf00".
1738
1739
1740 * What's new in version 0.7, 2008-07-15
1741
1742 - .statement("func@file:*") and .statement("func@file:M-N") probes are now
1743   supported to allow matching a range of lines in a function.  This allows
1744   tracing the execution of a function.
1745
1746 - Scripts relying on probe point wildcards like "syscall.*" that expand
1747   to distinct kprobes are processed significantly faster than before.
1748
1749 - The vector of script command line arguments is available in a
1750   tapset-provided global array argv[].  It is indexed 1 ... argc,
1751   another global.  This can substitute for of preprocessor
1752   directives @NNN that fail at parse time if there are not
1753   enough arguments.
1754
1755       printf("argv: %s %s %s", argv[1], argv[2], argv[3])
1756
1757 - .statement("func@file+line") probes are now supported to allow a
1758   match relative to the entry of the function incremented by line
1759   number.  This allows using the same systemtap script if the rest
1760   of the file.c source only changes slightly.
1761
1762 - A probe listing mode is available.
1763   % stap -l vm.*
1764   vm.brk
1765   vm.mmap
1766   vm.munmap
1767   vm.oom_kill
1768   vm.pagefault
1769   vm.write_shared
1770
1771 - More user-space probe types are added:
1772
1773   probe process(PID).begin { }
1774   probe process("PATH").begin { }
1775   probe process(PID).thread.begin { }
1776   probe process("PATH").thread.begin { }
1777   probe process(PID).end { }
1778   probe process("PATH").end { }
1779   probe process(PID).thread.end { }
1780   probe process("PATH").thread.end { }
1781   probe process(PID).syscall { }
1782   probe process("PATH").syscall { }
1783   probe process(PID).syscall.return { }
1784   probe process("PATH").syscall.return { }
1785
1786 - Globals now accept ; terminators
1787
1788   global odds, evens;
1789   global little[10], big[5];
1790
1791 * What's new in version 0.6, 2007-12-15
1792
1793 - A copy of the systemtap tutorial and language reference guide
1794   are now included.
1795
1796 - There is a new format specifier, %m, for the printf family of
1797   functions.  It functions like %s, except that it does not stop when
1798   a nul ('\0') byte is encountered.  The number of bytes output is
1799   determined by the precision specifier.  The default precision is 1.
1800   For example:
1801
1802       printf ("%m", "My String") // prints one character: M
1803       printf ("%.5", myString)   // prints 5 bytes beginning at the start
1804                                  // of myString
1805
1806 - The %b format specifier for the printf family of functions has been enhanced
1807   as follows:
1808
1809   1) When the width and precision are both unspecified, the default is %8.8b.
1810   2) When only one of the width or precision is specified, the other defaults
1811      to the same value.  For example, %4b == %.4b == %4.4b
1812   3) Nul ('\0') bytes are used for field width padding.  For example,
1813
1814      printf ("%b", 0x1111deadbeef2222) // prints all eight bytes
1815      printf ("%4.2b", 0xdeadbeef)      // prints  \0\0\xbe\xef
1816
1817 - Dynamic width and precision are now supported for all printf family format
1818   specifiers.  For example:
1819
1820      four = 4
1821      two = 2
1822      printf ("%*.*b", four, two, 0xdeadbbeef) // prints  \0\0\xbe\xef
1823      printf ("%*d", four, two)                // prints  <space><space><space>2
1824
1825 - Preprocessor conditional expressions can now include wildcard style
1826   matches on kernel versions.
1827   %( kernel_vr != "*xen" %? foo %: bar %)
1828
1829 - Prototype support for user-space probing is showing some progress.
1830   No symbolic notations are supported yet (so no probing by function names,
1831   file names, process names, and no access to $context variables), but at
1832   least it's something:
1833
1834     probe process(PID).statement(ADDRESS).absolute { }
1835
1836   This will set a uprobe on the given process-id and given virtual address.
1837   The proble handler runs in kernel-space as usual, and can generally use
1838   existing tapset functions.
1839
1840 - Crash utility can retrieve systemtap's relay buffer from a kernel dump
1841   image by using staplog which is a crash extension module. To use this
1842   feature, type commands as below from crash(8)'s command line:
1843
1844     crash> extend staplog.so
1845     crash> help systemtaplog
1846
1847   Then, you can see more precise help message.
1848
1849 - You can share a relay buffer amoung several scripts and merge outputs from
1850   several scripts by using "-DRELAY_HOST" and "-DRELAY_GUEST" options.
1851   For example:
1852
1853     # run a host script
1854     % stap -ve 'probe begin{}' -o merged.out -DRELAY_HOST &
1855     # wait until starting the host.
1856     % stap -ve 'probe begin{print("hello ");exit()}' -DRELAY_GUEST
1857     % stap -ve 'probe begin{print("world\n");exit()}' -DRELAY_GUEST
1858
1859   Then, you'll see "hello world" in merged.out.
1860
1861 - You can add a conditional statement for each probe point or aliase, which
1862   is evaluated when the probe point is hit. If the condition is false, the
1863   whole probe body(including aliases) is skipped. For example:
1864
1865     global switch = 0;
1866     probe syscall.* if (switch) { ... }
1867     probe procfs.write {switch = strtol($value,10)} /* enable/disable ctrl */
1868
1869 - Systemtap will warn you if your script contains unused variables or
1870   functions.  This is helpful in case of misspelled variables.  If it
1871   doth protest too much, turn it off with "stap -w ...".
1872
1873 - You can add error-handling probes to a script, which are run if a
1874   script was stopped due to errors.  In such a case, "end" probes are
1875   not run, but "error" ones are.
1876
1877     probe error { println ("oops, errors encountered; here's a report anyway")
1878                   foreach (coin in mint) { println (coin) } }
1879
1880 - In a related twist, one may list probe points in order of preference,
1881   and mark any of them as "sufficient" beyond just "optional".  Probe
1882   point sequence expansion stops if a sufficient-marked probe point has a hit.
1883   This is useful for probes on functions that may be in a module (CONFIG_FOO=m)
1884   or may have been compiled into the kernel (CONFIG_FOO=y), but we don't know
1885   which.  Instead of
1886
1887     probe module("sd").function("sd_init_command") ? ,
1888           kernel.function("sd_init_command") ? { ... }
1889
1890   which might match neither, now one can write this:
1891
1892     probe module("sd").function("sd_init_command") ! , /* <-- note excl. mark */
1893           kernel.function("sd_init_command")  { ... }
1894
1895 - New security model.  To install a systemtap kernel module, a user
1896   must be one of the following: the root user; a member of the
1897   'stapdev' group; or a member of the 'stapusr' group.  Members of the
1898   stapusr group can only use modules located in the
1899   /lib/modules/VERSION/systemtap directory (where VERSION is the
1900   output of "uname -r").
1901
1902 - .statement("...@file:line") probes now apply heuristics to allow an
1903   approximate match for the line number.  This works similarly to gdb,
1904   where a breakpoint placed on an empty source line is automatically
1905   moved to the next statement.  A silly bug that made many $target
1906   variables inaccessible to .statement() probes was also fixed.
1907
1908 - LKET has been retired.  Please let us know on <systemtap@sourceware.org>
1909   if you have been a user of the tapset/tools, so we can help you find
1910   another way.
1911
1912 - New families of printing functions println() and printd() have been added.
1913   println() is like print() but adds a newline at the end;
1914   printd() is like a sequence of print()s, with a specified field delimiter.
1915
1916 * What's new since version 0.5.14?, 2007-07-03
1917
1918 - The way in which command line arguments for scripts are substituted has
1919   changed.  Previously, $1 etc. would interpret the corresponding command
1920   line argument as an numeric literal, and @1 as a string literal.  Now,
1921   the command line arguments are pasted uninterpreted wherever $1 etc.
1922   appears at the beginning of a token.  @1 is similar, but is quoted as
1923   a string.  This change does not modify old scripts, but has the effect
1924   of permitting substitution of arbitrary token sequences.
1925
1926   # This worked before, and still does:
1927   % stap -e 'probe timer.s($1) {}'        5
1928   # Now this also works:
1929   % stap -e 'probe syscall.$1 {log(@1)}'  open
1930   # This won't crash, just signal a recursion error:
1931   % stap -e '$1'                          '$1'
1932   # As before, $1... is recognized only at the beginning of a token
1933   % stap -e 'probe begin {foo$1=5}'
1934
1935 * What's new since version 0.5.13?, 2007-03-26
1936
1937 - The way in which systemtap resolves function/inline probes has changed:
1938    .function(...) - now refers to all functions, inlined or not
1939    .inline(...)   - is deprecated, use instead:
1940    .function(...).inline - filters function() to only inlined instances
1941    .function(...).call - filters function() to only non-inlined instances
1942    .function(...).return - as before, but now pairs best with .function().call
1943    .statement() is unchanged.
1944
1945 * What's new since version 0.5.12?, 2007-01-01
1946
1947 - When running in -p4 (compile-only) mode, the compiled .ko file name
1948   is printed on standard output.
1949
1950 - An array element with a null value such as zero or an empty string
1951   is now preserved, and will show up in a "foreach" loop or "in" test.
1952   To delete such an element, the scripts needs to use an explicit
1953   "delete array[idx]" statement rather than something like "array[idx]=0".
1954
1955 - The new "-P" option controls whether prologue searching heuristics
1956   will be activated for function probes.  This was needed to get correct
1957   debugging information (dwarf location list) data for $target variables.
1958   Modern compilers (gcc 4.1+) tend not to need this heuristic, so it is
1959   no longer default.  A new configure flag (--enable-prologues) restores
1960   it as a default setting, and is appropriate for older compilers (gcc 3.*).
1961
1962 - Each systemtap module prints a one-line message to the kernel informational
1963   log when it starts.  This line identifies the translator version, base
1964   address of the probe module, a broken-down memory consumption estimate, and
1965   the total number of probes.  This is meant as a debugging / auditing aid.
1966
1967 - Begin/end probes are run with interrupts enabled (but with
1968   preemption disabled).  This will allow begin/end probes to be
1969   longer, to support generating longer reports.
1970
1971 - The numeric forms of kernel.statement() and kernel.function() probe points
1972   are now interpreted as relocatable values - treated as relative to the
1973   _stext symbol in that kernel binary.  Since some modern kernel images
1974   are relocated to a different virtual address at startup, such addresses
1975   may shift up or down when actually inserted into a running kernel.
1976
1977      kernel.statement(0xdeadbeef): validated, interpreted relative to _stext,
1978                                    may map to 0xceadbeef at run time.
1979
1980   In order to specify unrelocated addresses, use the new ".absolute"
1981   probe point suffix for such numeric addresses.  These are only
1982   allowed in guru mode, and provide access to no $target variables.
1983   They don't use debugging information at all, actually.
1984
1985      kernel.statement(0xfeedface).absolute: raw, unvalidated, guru mode only
1986
1987 * What's new since version 0.5.10?, 2006-10-19
1988
1989 - Offline processing of debugging information, enabling general
1990   cross-compilation of probe scripts to remote hosts, without
1991   requiring identical module/memory layout.  This slows down
1992   compilation/translation somewhat.
1993
1994 - Kernel symbol table data is loaded by staprun at startup time
1995   rather than compiled into the module.
1996
1997 - Support the "limit" keyword for foreach iterations:
1998     foreach ([x,y] in ary limit 5) { ... }
1999   This implicitly exits after the fifth iteration.  It also enables
2000   more efficient key/value sorting.
2001
2002 - Support the "maxactive" keyword for return probes:
2003     probe kernel.function("sdfsdf").maxactive(848) { ... }
2004   This allows up to 848 concurrently outstanding entries to
2005   the sdfsdf function before one returns.  The default maxactive
2006   number is smaller, and can result in missed return probes.
2007
2008 - Support accessing of saved function arguments from within
2009   return probes.  These values are saved by a synthesized
2010   function-entry probe.
2011
2012 - Add substantial version/architecture checking in compiled probes to
2013   assert correct installation of debugging information and correct
2014   execution on a compatible kernel.
2015
2016 - Add probe-time checking for sufficient free stack space when probe
2017   handlers are invoked, as a safety improvement.
2018
2019 - Add an optional numeric parameter for begin/end probe specifications,
2020   to order their execution.
2021      probe begin(10) { } /* comes after */ probe begin(-10) {}
2022
2023 - Add an optional array size declaration, which is handy for very small
2024   or very large ones.
2025      global little[5], big[20000]
2026
2027 - Include some example scripts along with the documentation.
2028
2029 - Change the start-time allocation of probe memory to avoid causing OOM
2030   situations, and to abort cleanly if free kernel memory is short.
2031
2032 - Automatically use the kernel DWARF unwinder, if present, for stack
2033   tracebacks.
2034
2035 - Many minor bug fixes, performance, tapset, and error message
2036   improvements.