systemtap 3.0 release

["Frank Ch. Eigler" <fche at redhat dot com>]

The SystemTap team announces release 3.0!

  Experimental monitor and interactive modes, optimized associative
  arrays, two types of function overloading, probe point brace-pattern
  expansion, security band-aid samples, improved string quoting and
  escaping, pretty printing array aggregates, private scoping for
  variables / functions, oxford commas, and much more!


= Where to get it

  https://sourceware.org/systemtap/ - our project page
  https://sourceware.org/systemtap/ftp/releases/systemtap-3.0.tar.gz
  https://koji.fedoraproject.org/koji/packageinfo?packageID=615
  git tag release-3.0 (commit 616ec7a0b916df)

  There have been over 450.32 commits since the last release.
  There have been between 57 and 60 bugs fixed / features added since the 
        last release.

= How to build it

  See the README and NEWS files at
  https://sourceware.org/git/?p=systemtap.git;a=tree

  Further information at https://sourceware.org/systemtap/wiki/


= SystemTap frontend (stap) changes

- The new experimental "interactive" mode, specified by "stap -i",
  drops you into a command-line prompt where you can build up a script,
  run it, edit it, run it again, etc. Type "help" for a list of commands.

- New experimental --monitor[=INTERVAL] option similar to unix "top". This
  allows users to see statistics about the running module(uptime, module name,
  invoker uid, memory sizes, global variables, and the current probe list
  along with their statistics).

  An interface is also provided to allow control over the running
  module (resetting global variables, sorting the list of probes, deactivating
  and reactivating probes).

- Add macros @prints to print a scalar aggregate variable, @prints[1-9]
  to print an array aggregate (of given index-arity), formatted similarly
  to the automatic printing of written-only global variables.

  global a, b
  probe oneshot { a <<< 1; b[tid()] <<< 2 }
  probe end { @prints(a); @prints1(b) }

- Add Czech version of key manual pages.

- The stap compile server will log the stap client's options that are passed
  to the server. The options that get logged on the server will include the
  script name or the -e script, depending on which is used by the client.


= SystemTap script language changes

- The performance of associative arrays have been dramatically
  improved, especially for densely filled tables and for multiple
  indexes.  The hash tables behind these arrays is now sized as a
  function of the array maximum size with an optional MAPHASHBIAS
  space/time tradeoff knob.

- Functions may now be overloaded during module runtime using the "next"
  statement in script functions and STAP_NEXT macro for embedded-C functions.
  They may also be overloaded by number of parameters during compile time.
  For example,

  Runtime overloading:
    function f() { if (condition) next; print("first function") }
    function f() %{ STAP_NEXT; print("second function") %}
    function f() { print("third function") }

  For the given functions above, a functioncall f(), will execute the body of the
  third function if condition evaluates to true and print "third function".
  Note that the second function is unconditionally nexted.

  Parameter overloading:
    function g() { print("first function") }
    function g(x) { print("second function") }
    g() -> "first function"
    g(1) -> "second function"

  Note that runtime overloading does not occur in the above example as the number
  of parameters of the functions differ. The use of a next statement inside a function
  while no more overloads remain will trigger a runtime exception. The function
  candidates are selected at compile time and is determined by the number of arguments
  provided for the functioncall.

- Embedded-C functions and blocks may now access script level global variables
  using the STAP_GLOBAL_GET_* and STAP_GLOBAL_SET_* macros.

  To read or write the script global var, the /* pragma:read:var */ or
  /* pragma:write:var */ marker must be placed in the embedded-C function or block.
  The written type must match the type inferred at script level.

  Scalars:
    STAP_GLOBAL_SET_var(STAP_GLOBAL_GET_var()+1) -> increments script global var by 1
    STAP_GLOBAL_SET_var("hello")

  Associative arrays:
    STAP_GLOBAL_GET_var(index-1, ..., index-n)
    STAP_GLOBAL_SET_var(index-1, ..., index-n, new value)

- Probe point brace expansion is now supported to improve brevity in
  specifying probe points. For example,

  process.{function("a"), function("b").{call,return}}
  => process.function("a"), process.function("b").call, process.function("b").return

  process.{function("*").{callees,plt}?
  => process.function("*").callees?, process.plt?

  {kernel,module("nfs")}.function("nfs*")!
  => kernel.function("nfs*")!, module("nfs").function("nfs*")!

- Profiling timers at arbitrary frequencies are now provided and perf probes
  now support a frequency field as an alternative to sampling counts.

  probe timer.profile.freq.hz(N)
  probe perf.type(N).config(M).hz(X)

  The specified frequency is only accurate up to around 100hz. You may
  need to provide a higher value to achieve the desired rate.

- Added support for private global variables and private functions. The 'private'
  keyword limits these to the tapset file they are defined in.


= SystemTap tapset changes

  ansi.stp                      Functions ansi_set_color{2,3} are replaced by
                                overloaded ansi_set_color
  linux/[arm/]aux_syscalls.stp  Support for arm kernels less than 3.7.
  linux/arm/[nd_]syscalls.stp   Support for [nd_]syscall.execve for arm
                                kernels less than 3.7.
  linux/aux_syscalls.stp        New _stp_mlock2_str function to convert mlock2
                                syscall flags to a string.
  linux/context.stp             New module_size() function.
  linux/conversions.stp         - New kernel_string_quoted_utf[16|32] functions
                                combines @string_quoted and @kernel_string_utf*
                                - kernel_string* functions with alternative error
                                strings are replaced by overloaded variants
  linux/nd_syscalls.stp         Add nd_syscall.mlock2 kprobe based probe point
  linux/perf.stp                Update recent uapi/linux/perf_event.h bits
  linux/proc_mem.stp            proc_mem_*_pid functions are replaced by overloaded
                                proc_mem_*
  linux/syscalls.stp            Add syscall.mlock2 kernel function probe point
  linux/task.stp                New task_cwd_path and task_exe_file functions
  linux/task_time.stp           task_{s,u}time_tid functions are replaced by
                                overloaded task_{s,u}time
  linux/uconversions.stp        user_string functions with alternative warning strings
                                are replaced by overloaded variants
  logging.stp                   New overloaded variant of assert, assert(expression)
  print_stats.stpm              @prints* macros for printing stats
  timers.stp                    Probe point timer.profile.freq for profiling
  try_assign.stpm               New @try_assign macro
  uconversions.stp              Add user_string_quoted_utf[16|32] function that
                                quotes a given UTF-[16|32] string from a given
                                user address

- Internal tapset functions and global variables are marked as private,
  where possible.

- Some tapsets have been modified to make use of the new function overloading
  feature. Instead of having new function names with suffixes such as "2" or
  "pid" to indicate extra arguments, the functions now seem to have optional
  arguments.

- New tapset function string_quoted() to quote and \-escape general strings.
  String $context variables that are pretty-printed are now processed with
  such a quotation engine, falling back to a 0x%x (hex pointer) on errors

- Functions get_mmap_args() and get_32mmap_args() got deprecated.


= SystemTap sample scripts - now at 156 samples!

  who_sent_it.stp   Trace outgoing network packets using the netfilter probes,
                    printing the source thread name/id and destination host:port

- New to collection is a selection of security band aids for specific CVEs. They
  are historical emergency security band-aids, and are for reference/education
  only. The scripts can be found under the security-band-aids folder in the
  examples directory.

- A number of samples were tweaked for portability and demonstration of newer
  language/tapset facilities.


= Examples of tested kernel versions

  2.6.18 (RHEL 5 x86 and x86_64)
  2.6.32 (RHEL 6 x86 and x86_64)
  3.10.0 (RHEL 7 x86_64)
  4.1.6  (Fedora 22 x86_64)
  4.3.4  (Fedora 22 x86_64)
  4.6.0-rc0  (Fedora rawhide x86_64)


= Known issues with this release

- Some kernel crashes continue to be reported when a script probes
  broad kernel function wildcards.  (PR2725)

- The dyninst backend is still very much a prototype, with a number of
  issues, limitations, and general teething woes.  See dyninst/README
  and the systemtap/dyninst Bugzilla component (
  http://tinyurl. com/stapdyn-PR-list ) if you want all the gory
  details about the state of the feature.

- An upstream kernel commit #2062afb4f804a put "-fno-var-tracking-assignments"
  into KCFLAGS, reducing debuginfo quality which can cause debuginfo failures.
  A proposed workaround to this issue exists in:
  https://lkml.org/lkml/2014/11/21/505 . Fedora kernels are not affected by
  this issue.


= Contributors for this release

  Abegail Jakop, David Smith, Felix Lu, Frank Ch. Eigler, Jesper
  Brouer*, Jonathan Lebon, Josh Stone, Ken Raeburn*, Lukas Berk, Mark
  Wielaard, Martin Cermak, Nan Xiao*, Nikolay Borisov*, Orivej Desh*,
  Stan Cox, Torsten Polle, William Cohen, Zhou Wenjian*

  Special thanks to new contributors, marked with '*' above.
  Special thanks to Abegail for compiling these notes.


= Bugs fixed for this release <https://sourceware.org/PR#####>

6525    need utrace task-finder-based pid->execname, pid->cwd-path-name tables
6977    Incorrect resolution of variables in function scope
9968    Share global variable between embedded C and script
10607   need way to protect sensitive tapset globals
11347   intra-tapset visibility scoping for embedded-c functions
12414   Missing return value is not reported
13514   more robust -rt support
13608   Limit the size of the request from the compile server client
13643   log stap options
13721   local variable name collision
13820   timer.profile(NNN) variant using perf probes
14203   Embedded-C functions in various tapsets are not stress tested
14431   char * always being printed as a, possibly "<unknown>", without the address
15775   relay host/guest functionality is broken on RHEL6
16672   Allow brace expansion on probe points
17393   java.exp: ERROR: Cannot attach to module control channel
17587   too sensitive to duplicate globals across tapset files
17741   try-catch statement as a tapset function
18075   Add a return-probe variant of .callee
18431   runtime function overloading facility
18848   new --monitor option
18876   need to assert
18884   const-folded literals get unnecessary tmp storage
18889   lost ability to probe kernel module initializers
19120   kbuild warnings on RHEL5
19135   testsuite results hard to compare
19136   "private" keyword for tapsets
19216   kprobe.module("MOD").function("FOO") doesn't work on not loaded modules
19234   user_string_n2_quoted() vs. user_string_n_quoted.exp
19236   NULL global variable breakage
19241   systemtap fails to start when CPU0 is offline
19265   finding out-of-tree kernel modules works with full paths, not relative paths
19275   the stap_run testsuite function doesn't handle unexpected output
19345   RHEL 7.0 s390x crash in check.exp
19346   escape xml documentation
19358   netfilter probes are broken on rawhide
19362   translator puts two probes for each probe point in .../library.exp tests
19396   systemtap can't find certain kernel tracepoints
19470   process_by_pid.exp always fails on arches that don't support .plt probes
19497   linux 4.4 commit 7523e4dc50
19502   extend stap.1 with documentation overview
19510   the "private" keyword support has made -p1 output less useful
19521   the "private" keyword support has made error messages less useful
19525   script doesn't finish in bulk mode
19536   kretprobe data-pouch type issue
19537   the parseok/semko.stp test case always fails
19560   flight recorder mode intermittently fails
19639   parallel_exec.exp fails in beaker-project.org
19644   linux 4.5-rc4 commit 8244062ef1
19700   Support dash as /bin/sh in generated Makefile
19756   linux 4.5-rc1 commit 8463833590
19764   stap --tmpdir leaves useless tempdirs behind
19799   deleting from array of aggregate unreliable
19802   bad hash value distribution and horrible performance for large arrays
19812   _stp_target misdirected for pid_namespace containers
19833   monitor mode causes kernel crash when exited immediately 
19847   Entering ^C in interactive mode prompt prevents script from running
19873   staprun -o /NO/SUCH/FILE -c CMD imperfect cleanup