By popular demand, here is a summary comparing systemtap and similar systems. Corrections and improvements are welcome.

Individual characteristics are listed vertically, with the two tools' support for each listed alongside. While the tools are similar, key differentiators are highlighted in italics. In some cases, terms such as "soon" and "not yet" are used to indicate that the given feature has been planned or scheduled, however one should not infer any immediacy regarding the availability of that feature.


systemtap

DTrace on solaris

LTTng

perf

project

license

GPL

CDDL and proprietary

Kernel tracer: GPLv2, userspace tracer: LGPLv2.1 (library) and GPLv2 (tools)

GPL

operating system support

Linux

Solaris, Mac OS X, BSD, partial: QNX, Linux

Linux

Linux

processor support

as per dyninst/kprobes: x86, x86_64, ppc64, ia64, s390, arm, sparc?

x86, x86_64, SPARC, ppc, ppc64, ARM

Kernel tracer: x86, x86_64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM, (arch-agnostic core), userspace tracer: x86, x86_64, ppc, ppc64

most kernel architectures

kernel coupling - interlocked development/schedule

none

lot

Kernel tracer: some, userspace tracer: none

fully coupled

core developers

open community

open community & proprietary

open community

lkml

development began

January 2005

October 2001

January 2005

2009

development status

ongoing

stable

ongoing

ongoing

target audience

developers, users, sysadmins

developers, users, sysadmins

developers, users, sysadmins

developers, sysadmins

target usage

symbolic debugging, tracing, profiling

debugging, tracing, profiling

debugging, tracing, profiling, monitoring

tracing, profiling

concurrent users

unlimited

unlimited

usually single

usually single

language

style

scripting

scripting

hard-coded kernel C code

command-line tool

full control structures (conditionals, loops, functions)

yes

no

yes?

no

variable typing

implicit, inferred

implicit

explicit

implicit

complex reporting (join/projection/select)

yes (first principles, iteration, conditionals)

limited (with printa())

n/a

perl/python post-processing

scalable aggregates

yes

yes

n/a

n/a

aggregate value readable by script

yes

no

n/a

n/a

thread-local variables

yes (from first principles via tid-indexed auxiliary arrays)

yes (implemented efficiently)

n/a

n/a

lock-protected global variables

yes (automatic protection)

no (unprotected)

n/a

n/a

speculative tracing

yes (from first principles via auxiliary data and control structures)

yes

work in progress

no

binary tracing record

yes

?

yes

?

early boot tracing

yes

yes

not "as early as it could" yet

yes

probe execution

optimized native code or bytecodes

interpreted bytecodes

optimized native code

hard-coded tracing

execution site

kernel or kernel-bpf or userspace

kernel

kernel

kernel or userspace

probing capability

number of available symbolic probe points in the kernel

millions (statements, tracepoints, markers)

thousands (functions, markers)

thousands (functions, tracepoints, markers)

millions (functions, tracepoints, statements w/ DWARF)

number of available symbolic probe points in user-space

zillions (statements, functions, markers)

millions (functions, markers)

in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html

none

high-speed breakpoint-less userspace data extraction

yes (dyninst)

no

in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html

no

probe arbitrary statements in code symbolically (function entry, exit, interior, source code co-ordinates)

yes (using debugging information)

limited to ABI boundaries (function entry/exit, markers)

no

yes (kernel)

symbolically extract arbitrary data at probe point

yes (any context-visible variable as preserved by compiler)

limited (all data visible; local variables may be accessed by using manual register offsets)

no

limited

extraction of data by static instrumentation

yes (markers/tracepoints)

yes (with markers)

yes (with tracepoints and markers)

yes (tracepoints)

non-symbolic (manually addressed) probe points / data

yes

yes

no

limited

probe dynamically loaded kernel objects

yes

yes

yes

?

concurrent probes on multiprocessors

yes

yes

yes

yes

extracted value pointer type punning/casting

yes

yes

no

limited

statically inserted probe points, kernel side

yes (tracepoints, markers)

yes (SDT)

yes (tracepoints, markers)

yes (tracepoints)

end-user extendable probe library

yes (script based tapsets)

no

yes

no

collect kernel-space stack backtraces

yes

yes (ABI helps)

no

yes (frame-pointers)

collect user-space stack backtraces

yes

yes (ABI helps)

no

prototyped (frame-pointers)

backtraces from fully-optimized code

yes (DWARF unwinding)

no (frame-pointers)

no

no (frame-pointers)

statically inserted probe points, user side

yes <sdt.h>

yes (USDT)

in progress http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html

no

trace Java programs

yes

yes

had previously http://ltt.polymtl.ca/svn/trunk/obsolete/ltt-usertrace/java/, not currently

generate Java stack backtraces

yes

yes

no

no

statically inserted probe points, Java

from JVM

yes

no yet

no

per-method java probe points

yes, via Byteman

no

no

no

generate Python stack backtraces

yes

yes

no

no

trace script language programs

yes, distributions to activate

yes: Ruby, JavaScript, Perl, Python, PHP, APL, Bourne shell, ksh, zsh, Tcl

yes, through debugfs file

no

timer-based probing

yes (profiling interrupt, software timers)

yes

no

yes

hardware performance counter based probing

yes

yes

soon

yes

safety

protected probe execution environment

yes

yes

no (hand-written)

n/a (post-processing only)

time-limited probe handler execution

yes (statement counting)

yes (built into interpreter)

no

yes (not programmable)

non-blocking, atomic probe handlers

yes

yes

yes

yes

space-limited execution

yes (static allocation of all data)

yes

yes (bounded to buffer size + probe size + memory allocated by probe)

yes (not programmable)

division-by-zero protection

yes

yes

no

n/a

null pointer dereferencing protection

yes

yes

no

yes

means available to bypass protection for advanced users

yes (guru mode, embedded-C)

limited (predefined destructive actions for well-defined modifications)

Optimized probes shipped with mainline kernel, variable argument list (printf-like) type safe data serializer available for ad-hoc use

no

safe use on production systems

yes

yes

yes

yes

translate-time error checks

many

many

n/a

n/a

run-time error checks performed by

automatically generated C code

bytecode interpreter

hand-written

hand-written

use by unprivileged users

yes, three privilege levels, plus pure-userspace dyninst

graduated access by privilege level

work in progress ("tracing" group access)

limited

ability to change state in probed program

yes (guru mode)

limited

no

no

can patch bugs (e.g. security band-aids for kernel or userspace)

yes

no

no

no

can unfortunately trigger latent kernel bugs

yes

yes

yes

yes

misc.

analysis performed

online (or offline)

online

offline (post-mortem)

offline

built-into the system kernel

not necessary

yes

yes (mainlining in progress)

yes

lockless tracing

no

yes

yes

yes

zero-copy trace data extraction

?

no (designed to allow for concise traces)

yes

yes ?

provides analysis tools to navigate in large multi-GB traces

no

no (designed to allow for concise traces)

yes (lttv and Eclipse LTT/MF http://www.eclipse.org/linuxtools/projectPages/lttng/)

no

None: SystemtapDtraceComparison (last edited 2017-10-18 22:08:56 by FChE)