By popular demand, here is a summary comparing systemtap and similar systems. Corrections and improvements are welcome.
Individual characteristics are listed vertically, with the two tools' support for each listed alongside. While the tools are similar, key differentiators are highlighted in italics. In some cases, terms such as "soon" and "not yet" are used to indicate that the given feature has been planned or scheduled, however one should not infer any immediacy regarding the availability of that feature.
|
systemtap |
DTrace on solaris |
LTTng |
perf |
project |
||||
license |
GPL |
CDDL and proprietary |
Kernel tracer: GPLv2, userspace tracer: LGPLv2.1 (library) and GPLv2 (tools) |
GPL |
operating system support |
Linux |
Solaris, Mac OS X, BSD, partial: QNX, Linux |
Linux |
Linux |
processor support |
as per dyninst/kprobes: x86, x86_64, ppc64, ia64, s390, arm, sparc? |
x86, x86_64, SPARC, ppc, ppc64, ARM |
Kernel tracer: x86, x86_64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM, (arch-agnostic core), userspace tracer: x86, x86_64, ppc, ppc64 |
most kernel architectures |
kernel coupling - interlocked development/schedule |
none |
lot |
Kernel tracer: some, userspace tracer: none |
fully coupled |
core developers |
open community |
open community & proprietary |
open community |
lkml |
development began |
January 2005 |
October 2001 |
January 2005 |
2009 |
development status |
ongoing |
stable |
ongoing |
ongoing |
target audience |
developers, users, sysadmins |
developers, users, sysadmins |
developers, users, sysadmins |
developers, sysadmins |
target usage |
symbolic debugging, tracing, profiling |
debugging, tracing, profiling |
debugging, tracing, profiling, monitoring |
tracing, profiling |
concurrent users |
unlimited |
unlimited |
usually single |
usually single |
language |
||||
style |
scripting |
scripting |
hard-coded kernel C code |
command-line tool |
full control structures (conditionals, loops, functions) |
yes |
no |
yes? |
no |
variable typing |
implicit, inferred |
implicit |
explicit |
implicit |
complex reporting (join/projection/select) |
yes (first principles, iteration, conditionals) |
limited (with printa()) |
n/a |
perl/python post-processing |
scalable aggregates |
yes |
yes |
n/a |
n/a |
aggregate value readable by script |
yes |
no |
n/a |
n/a |
thread-local variables |
yes (from first principles via tid-indexed auxiliary arrays) |
yes (implemented efficiently) |
n/a |
n/a |
lock-protected global variables |
yes (automatic protection) |
no (unprotected) |
n/a |
n/a |
speculative tracing |
yes (from first principles via auxiliary data and control structures) |
yes |
work in progress |
no |
binary tracing record |
yes |
? |
yes |
? |
early boot tracing |
yes |
yes |
not "as early as it could" yet |
yes |
probe execution |
optimized native code or bytecodes |
interpreted bytecodes |
optimized native code |
hard-coded tracing |
execution site |
kernel or kernel-bpf or userspace |
kernel |
kernel |
kernel or userspace |
probing capability |
||||
number of available symbolic probe points in the kernel |
millions (statements, tracepoints, markers) |
thousands (functions, markers) |
thousands (functions, tracepoints, markers) |
millions (functions, tracepoints, statements w/ DWARF) |
number of available symbolic probe points in user-space |
zillions (statements, functions, markers) |
millions (functions, markers) |
in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html |
none |
high-speed breakpoint-less userspace data extraction |
yes (dyninst) |
no |
in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html |
no |
probe arbitrary statements in code symbolically (function entry, exit, interior, source code co-ordinates) |
yes (using debugging information) |
limited to ABI boundaries (function entry/exit, markers) |
no |
yes (kernel) |
symbolically extract arbitrary data at probe point |
yes (any context-visible variable as preserved by compiler) |
limited (all data visible; local variables may be accessed by using manual register offsets) |
no |
limited |
extraction of data by static instrumentation |
yes (markers/tracepoints) |
yes (with markers) |
yes (with tracepoints and markers) |
yes (tracepoints) |
non-symbolic (manually addressed) probe points / data |
yes |
yes |
no |
limited |
probe dynamically loaded kernel objects |
yes |
yes |
yes |
? |
concurrent probes on multiprocessors |
yes |
yes |
yes |
yes |
extracted value pointer type punning/casting |
yes |
yes |
no |
limited |
statically inserted probe points, kernel side |
yes (tracepoints, markers) |
yes (SDT) |
yes (tracepoints, markers) |
yes (tracepoints) |
end-user extendable probe library |
yes (script based tapsets) |
no |
yes |
no |
collect kernel-space stack backtraces |
yes |
yes (ABI helps) |
no |
yes (frame-pointers) |
collect user-space stack backtraces |
yes |
yes (ABI helps) |
no |
prototyped (frame-pointers) |
backtraces from fully-optimized code |
yes (DWARF unwinding) |
no (frame-pointers) |
no |
no (frame-pointers) |
statically inserted probe points, user side |
yes <sdt.h> |
yes (USDT) |
in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html |
no |
trace Java programs |
yes |
yes |
had previously http://ltt.polymtl.ca/svn/trunk/obsolete/ltt-usertrace/java/, not currently |
|
generate Java stack backtraces |
yes |
yes |
no |
no |
statically inserted probe points, Java |
from JVM |
yes |
no yet |
no |
per-method java probe points |
yes, via Byteman |
no |
no |
no |
generate Python stack backtraces |
yes |
yes |
no |
no |
trace script language programs |
yes, distributions to activate |
yes: Ruby, JavaScript, Perl, Python, PHP, APL, Bourne shell, ksh, zsh, Tcl |
yes, through debugfs file |
no |
timer-based probing |
yes (profiling interrupt, software timers) |
yes |
no |
yes |
hardware performance counter based probing |
yes |
yes |
soon |
yes |
safety |
||||
protected probe execution environment |
yes |
yes |
no (hand-written) |
n/a (post-processing only) |
time-limited probe handler execution |
yes (statement counting) |
yes (built into interpreter) |
no |
yes (not programmable) |
non-blocking, atomic probe handlers |
yes |
yes |
yes |
yes |
space-limited execution |
yes (static allocation of all data) |
yes |
yes (bounded to buffer size + probe size + memory allocated by probe) |
yes (not programmable) |
division-by-zero protection |
yes |
yes |
no |
n/a |
null pointer dereferencing protection |
yes |
yes |
no |
yes |
means available to bypass protection for advanced users |
yes (guru mode, embedded-C) |
limited (predefined destructive actions for well-defined modifications) |
Optimized probes shipped with mainline kernel, variable argument list (printf-like) type safe data serializer available for ad-hoc use |
no |
safe use on production systems |
yes |
yes |
yes |
yes |
translate-time error checks |
many |
many |
n/a |
n/a |
run-time error checks performed by |
automatically generated C code |
bytecode interpreter |
hand-written |
hand-written |
use by unprivileged users |
yes, three privilege levels, plus pure-userspace dyninst |
graduated access by privilege level |
work in progress ("tracing" group access) |
limited |
ability to change state in probed program |
yes (guru mode) |
limited |
no |
no |
can patch bugs (e.g. security band-aids for kernel or userspace) |
yes |
no |
no |
no |
can unfortunately trigger latent kernel bugs |
yes |
yes |
yes |
yes |
misc. |
||||
analysis performed |
online (or offline) |
online |
offline (post-mortem) |
offline |
built-into the system kernel |
not necessary |
yes |
yes (mainlining in progress) |
yes |
lockless tracing |
no |
yes |
yes |
yes |
zero-copy trace data extraction |
? |
no (designed to allow for concise traces) |
yes |
yes ? |
provides analysis tools to navigate in large multi-GB traces |
no |
no (designed to allow for concise traces) |
yes (lttv and Eclipse LTT/MF http://www.eclipse.org/linuxtools/projectPages/lttng/) |
no |