1 SYSTEMTAP EXAMPLES INDEX
2 (see also keyword-index.txt)
4 general/ansi_colors.stp - Color Table for ansi_set_color2() and ansi_set_color3()
7 The script prints a table showing the available color combinations
8 for the ansi_set_color2() and ans_set_color3() functions in the
11 # stap ansi_colors.stp
14 general/ansi_colors2.stp - Show Attribues in Table for ansi_set_color3()
17 The script prints a table showing the available attributes (bold,
18 underline, and inverse) with color combinations for the
19 ans_set_color3() function in the ansi.stp tapset.
21 # stap ansi_colors2.stp
24 general/badname.stp - Bad Filename Filter
27 The badname.stp script shows how one could prevent the creation of
28 files with undesirable names using guru mode.
30 # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \
34 general/eventcount.stp - Count Specified Events
35 keywords: statistics thread process
37 The script prints a count of specified events and their related tid's
38 over the course of execution.
40 # stap eventcount.stp syscall.* -c 'sleep 0.2'
43 general/func_time_stats.stp - Function Time Statistics
44 keywords: function statistics
46 The func_time_stats.stp script tracks the wall clock time for each
47 invocation of the function probe listed as the first command line
48 argument. When the script exits it prints out the minimum, average,
49 and maximum times in microseconds followed by a count of times that
50 the function was called and a histogram showing the distributions of
53 # stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 0.2"
56 general/graphs.stp - Graphing Disk and CPU Utilization
57 keywords: disk cpu utilization
59 The script tracks the disk and CPU utilization. The resulting output
60 of the script can be piped into gnuplot to generate a graph of disk
63 # stap graphs.stp -c "sleep 0.2"
66 general/helloworld.stp - SystemTap "Hello World" Program
69 A basic "Hello World" program implemented in SystemTap script. It
70 prints out "hello world" message and then immediately exits.
75 general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments
76 keywords: trace callgraph
78 Print a timed per-thread callgraph, complete with pretty-printed
79 function parameters and return values. The first parameter names the
80 function probe points to trace. The optional second parameter names
81 the probe points for trigger functions, which acts to enable tracing
82 for only those functions that occur while the current thread is
83 nested within the trigger.
85 # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \
86 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
89 general/para-callgraph.stp - Callgraph Tracing with Arguments
90 keywords: trace callgraph
92 Print a timed per-thread callgraph, complete with function parameters
93 and return values. The first parameter names the function probe
94 points to trace. The optional second parameter names the probe
95 points for trigger functions, which acts to enable tracing for only
96 those functions that occur while the current thread is nested within
99 # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \
100 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
103 general/sizeof.stp - Print the Size of a C Type
104 keywords: statistics memory
106 This script prints the size of a type, based on dwarf debuginfo for
107 any kernel or userspace module, or trial-compilation of a given
110 # stap sizeof.stp FILE '</usr/include/stdio.h>'
113 general/stopwatches.stp - See the amount of wall clock time a process spends in various states
116 The stopwatch.stp script illustrates how to use multiple stopwatches
117 record how much wallclock time a process spends in kernel- and
118 user-space. On exit the script prints out the time in seconds,
119 milliseconds, microseconds, and nanoseconds. Note that this output of
120 this script is not directly comparable to the time command because
121 time records the time that the process is actually active in kernel-
124 # stap stopwatches.stp -c "sleep 0.2"
127 general/varwatch.stp - Watch a Variable Changing Value in a Thread
130 This script places a set of probes (specified by $1), each of which
131 monitors the state of some context $variable expression (specified by
132 $2). Whenever the value changes, with respect to the active thread,
135 # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \
136 '$$vars' -c "sleep 0.2"
139 general/watchdog.stp - Watchdog Timer for Arbitrary Events
140 keywords: watchdog backtrace
142 The watchdog.stp script provides a watchdog timer mechanism for
143 arbitrary events. The script takes three arguments: the events to
144 start watchdog timer, the event to stop the watchdog timer, and the
145 time in millseconds for the watchdog. If the watchdog timer is
146 exceed, the script will trigger a stack backtrace of the user-process
147 that timed out using pstack. This script can be used to diagnose what
148 the userspace application is doing when a slower than expected
151 # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \
155 interrupt/interrupts-by-dev.stp - Record Interrupts on a Per-Device Basis
158 The interrupts-by-dev.stp script profiles interrupts received by each
161 # stap interrupts-by-dev.stp -c "sleep 0.2"
164 interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt
165 keywords: interrupt backtrace
167 The Linux kernel function smp_call_function causes expensive
168 inter-processor interrupts (IPIs). The scf.stp script tallies the
169 processes and backtraces causing the interprocessor interrupts to
170 identify the cause of the expensive IPI. On exit the script prints
171 the tallies in descending frequency.
173 # stap scf.stp -c "sleep 0.2"
176 io/deviceseeks.stp - Histograms of Seek Behavior for Each Device
179 The deviceseeks.stp script generates a histogram showing the
180 frequency of different sized seeks on each device.
182 # stap deviceseeks.stp -c "sleep 0.2"
185 io/disktop.stp - Summarize Disk Read/Write Traffic
188 Get the status of reading/writing disk every 5 seconds, output top
189 ten entries during that period.
191 # stap disktop.stp -c "sleep 0.2"
194 io/enospc.stp - Report an disk out-of-space condition.
195 keywords: disk filesystem
197 This script monitors a filesystem implementations for early internal
198 indications of ENOSPC, and reports these to the system logger and the
201 # stap enospc.stp -c "sleep 0.2"
204 io/inodewatch.stp - Monitoring Reads and Writes to a File
207 The inodewatch.stp outputs the executable name and process id each
208 time a read or write occurs to the specified inode on the specified
211 # stap inodewatch.stp 0x08 0x01 100 -c "sleep 0.2"
214 io/inodewatch2.stp - Monitoring Attribute Changes to a File
217 The inodewatch2.stp script outputs the executable name, process id,
218 and attributes each time the attributes are changed on the specified
219 inode on the specified major/minor device.
221 # stap inodewatch2.stp 0x08 0x01 100 -c "sleep 0.2"
224 io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call
225 keywords: io backtrace
227 When a reschedule occurs during an AIO io_submit call, accumulate the
228 traceback in a histogram. When the script exits prints out a sorted
229 list from most common to least common backtrace.
231 # stap io_submit.stp -c "sleep 0.2"
234 io/ioblktime.stp - Average Time Block IO Requests Spend in Queue
237 The ioblktime.stp script tracks the amount of time that each block IO
238 requests spend waiting for completion. The script computes the
239 average waiting time for block IO per device and prints list every 10
240 seconds. In some cases there can be too many outstanding block IO
241 operations and the script may exceed the default number of
242 MAXMAPENTRIES allowed. In this case the allowed number can be
243 increased with "-DMAXMAPENTRIES=10000" option on the stap command
246 # stap ioblktime.stp -c "sleep 0.2"
249 io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device
250 keywords: io profiling
252 The iodevstats.stp script measures the amount of data successfully
253 read and written by all the executables for each io device on the
254 system. The output is sorted from greatest sum of bytes read and
255 written to a device by an executable to the least. The output
256 contains device major/minor number, the count of operations (reads
257 and writes), the totals and averages for the number of bytes read and
260 # stap iodevstats.stp -c "sleep 0.2"
263 io/iostat-scsi.stp - IO Statistics for SCSI Devices
264 keywords: io profiling scsi
266 The iostat-scsi.stp script provides a breakdown of the number of blks
267 read and written on the machine's various SCSI devices. The script
268 takes one argument which is the number of seconds between reports.
270 # stap -g iostat-scsi.stp 1 -c "sleep 0.2"
273 io/iostats.stp - List Executables Reading and Writing the Most Data
274 keywords: io profiling
276 The iostat.stp script measures the amount of data successfully read
277 and written by all the executables on the system. The output is
278 sorted from most greatest sum of bytes read and written by an
279 executable to the least. The output contains the count of operations
280 (opens, reads, and writes), the totals and averages for the number of
281 bytes read and written.
283 # stap iostats.stp -c "sleep 0.2"
286 io/iotime.stp - Trace Time Spent in Read and Write for Files
287 keywords: profiling syscall io file
289 The script watches each open, close, read, and write syscalls on the
290 system. For each file the scripts observes opened it accumulates the
291 amount of wall clock time spent in read and write operations and the
292 number of bytes read and written. When a file is closed the script
293 prints out a pair of lines for the file. Both lines begin with a
294 timestamp in microseconds, the PID number, and the executable name in
295 parentheses. The first line with the "access" keyword lists the file
296 name, the attempted number of bytes for the read and write
297 operations. The second line with the "iotime" keyword list the file
298 name and the number of microseconds accumulated in the read and write
301 # stap iotime.stp -c "sleep 0.2"
304 io/iotop.stp - Periodically Print IO Activity by Process Name
307 Every five seconds print out the top ten executables generating I/O
308 traffic during that interval sorted in descending order.
310 # stap iotop.stp -c "sleep 0.2"
313 io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices
314 keywords: io monitoring disk
316 The mbrwatch.stp script reports any attempted reads/writes of the
317 first few sectors of a raw block device.
319 # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
320 /proc/partitions | grep . | awk '{print $4}' | head -1`"
323 io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process
324 keywords: io profiling
326 The nfs_func_users.stp script counts the uses of NFS functions in the
327 kernel on a per process bases. The output is sorted from the process
328 with the greatest number of NFS functions called to the least. The
329 output contains the executable name, the process number, and the
330 total number of NFS functions called by the process.
332 # stap nfs_func_users.stp -c "sleep 0.2"
335 io/traceio.stp - Track Cumulative IO Activity by Process Name
338 Every second print out the top ten executables sorted in descending
339 order based on cumulative I/O traffic observed.
341 # stap traceio.stp -c "sleep 0.2"
344 io/traceio2.stp - Watch IO Activity on a Particular Device
347 Print out the executable name and process number as reads and writes
348 to the specified device occur.
350 # stap traceio2.stp 0x0801 -c "sleep 0.2"
353 io/ttyspy.stp - Monitor TTY Typing
354 keywords: io tty monitoring
356 The ttyspy.stp script uses tty_audit hooks to monitor recent typing
357 activity on the system, printing a scrolling record of recent
358 keystrokes, on a per-tty basis.
360 # stap --skip-badvars -g ttyspy.stp -c "sleep 1"
363 locks/bkl.stp - Tracing Contention on Big Kernel Lock
366 The bkl.stp script can help determine whether the Big Kernel Lock
367 (BKL) is causing serialization on a multiprocessor system due to
368 excessive contention of the BKL. The bkl.stp script takes one
369 argument which is the number of processes waiting for the Big Kernel
370 Lock (BKL). When the number of processes waiting for the BKL is
371 reached or exceeded, the script will print a time stamp, the number
372 of processes waiting for the BKL, the holder of the BKL, and the
373 amount of time the BKL was held.
375 # stap bkl.stp -c "sleep 0.2" 1
378 locks/bkl_stats.stp - Per Process Statistics on Big Kernel Lock Use
381 The bkl_stats.stp script can indicate which processes have excessive
382 waits for the Big Kernel Lock (BKL) and which processes are taking
383 the BKL for long periods of time. The bkl_stats.stp script prints
384 lists of all the processes that require the BKL. Every five seconds
385 two tables are printed out. The first table lists the processes that
386 waited for the BKL followed by the number of times that the process
387 waited, the minimum time of the wait, the average and the maximum
388 time waited. The second table lists has similar information for the
389 time spent in holding the lock for each of the processes.
391 # stap bkl_stats.stp -c "sleep 0.2"
394 memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware
395 keywords: memory watchpoint
397 The script will watch accesses to a single kernel address and prints
398 a traceback each time the address is accessed. This script needs to
399 be run as root to allow access to the breakpoint hardware.
401 # stap --all-modules hw_watch_addr.stp 0x`grep "pid_max$" /proc/kallsyms` \
405 memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware
406 keywords: memory watchpoint
408 The script will watch accesses to the starting address of a single
409 kernel symbol and prints a traceback each time the symbol is
410 accessed. This script needs to be run as root to allow access to the
413 # stap --all-modules hw_watch_sym.stp pid_max -c "sleep 1"
416 memory/kmalloc-top - Show Paths to Kernel Malloc Invocations
419 The kmalloc-top perl program runs a small systemtap script to collect
420 stack traces for each call to the kmalloc function and counts the
421 time that each stack trace is observed. When kmalloc-top exits it
422 prints out sorted list. The output can be filtered to print only the
423 first N stack traces (-t), stack traces with a minimum counts (-m),
424 or exclude certain stack traces (-e).
426 # ./kmalloc-top -c "sleep 0.2"
429 memory/mmanonpage.stp - Track Virtual Memory System Actions on Anonymous Pages
432 The mmanonpage.stp script uses the virtual memory tracepoints
433 available in some kernels to track the number of faults, user space
434 frees, page ins, copy on writes and unmaps for anonymous pages. When
435 the script is terminated the counts are printed for each process that
436 allocated pages while the script was running. This script displays
437 the anonymous page statistics for each process that ran while the
438 script is active. It's useful in debugging leaks in the anonymous
439 regions of a process.
441 # stap mmanonpage.stp -c "sleep 0.2"
444 memory/mmfilepage.stp - Track Virtual Memory System Actions on File Backed Pages
447 The mmfilepage.stp script uses the virtual memory tracepoints
448 available in some kernels to track the number of faults, copy on
449 writes mapping, and unmapping operations for file backed pages. When
450 the script is terminated the counts are printed for each process that
451 allocated pages while the script was running. The mmfilepage.stp
452 script is useful in debugging leaks in the mapped file regions of a
455 # stap mmfilepage.stp -c "sleep 0.2"
458 memory/mmreclaim.stp - Track Virtual Memory System Page Reclamation
461 The mmreclaim.stp script uses the virtual memory tracepoints
462 available in some kernels to track page reclaim activity that
463 occurred while the script was running. It's useful in debugging
464 performance problems that occur due to page reclamation.
466 # stap mmreclaim.stp -c "sleep 0.2"
469 memory/mmwriteback.stp - Track Virtual Memory System Writing to Disk
472 The mmwriteback.stp script uses the virtual memory tracepoints
473 available in some kernels to report all of the file writebacks that
474 occur form kupdate, pdflush and kjournald while the script is
475 running. It's useful in determining where writes are coming from on
476 a supposedly idle system that is experiencing unexpected IO.
478 # stap mmwriteback.stp -c "sleep 0.2"
481 memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes
482 keywords: memory numa
484 The numa_faults.stp script tracks the read and write pages faults for
485 each process. When the script exits it prints out the total read and
486 write pages faults for each process. The script also provide a break
487 down of page faults per node for each process. This script is useful
488 for determining whether the program has good locality (page faults
489 limited to a single node) on a NUMA computer.
491 # stap numa_faults.stp -c "sleep 0.2"
494 memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits
495 keywords: memory limits
497 The overcommit.stp script prints a line each time the kernel refuses
498 a memory allocation request from a process because of
499 /proc/sys/vm/overcommit* limits.
501 # stap overcommit.stp -c "sleep 0.2"
504 memory/pfaults.stp - Generate Log of Major and Minor Page Faults
507 The pfaults.stp script generates a simple log for each major and
508 minor page fault that occurs on the system. Each line contains a
509 timestamp (in microseconds) when the page fault servicing was
510 completed, the pid of the process, the address of the page fault, the
511 type of access (read or write), the type of fault (major or minor),
512 and the elapsed time for page fault. This log can be examined to
513 determine where the page faults are occurring.
515 # stap pfaults.stp -c "sleep 0.2"
518 memory/vm.tracepoints.stp - Collect Slab Allocation Statistics
519 keywords: memory slab statistics
521 The script will probe all memory slab/slub allocations and collects
522 information about the size of the object (bytes requested) and
523 user-space process in execution. When run over a period of time, it
524 helps to correlate kernel-space memory consumption owing to
525 user-space processes.
527 # stap vm.tracepoints.stp -c "sleep 10"
530 network/autofs4.stp - Watch autofs4 Operations
531 keywords: network autofs nfs
533 Trace key autofs4 operations such as mounting or unmounting remote
536 # stap autofs4.stp -c "sleep 1"
539 network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel
540 keywords: network tracepoint socket
542 Every five seconds the dropwatch.stp script lists the number of
543 socket buffers freed at locations in the kernel.
545 # stap dropwatch.stp -c "sleep 0.2"
548 network/netdev.stp - Trace Activity on Network Devices
549 keywords: network device traffic
551 The netdev.stp script traces configuration and transmit/receive
552 activity on network devices.
554 # stap netdev.stp -c "sleep 0.2"
557 network/nettop.stp - Periodic Listing of Processes Using Network Interfaces
558 keywords: network traffic
560 Every five seconds the nettop.stp script prints out a list of
561 processed (PID and command) with the number of packets sent/received
562 and the amount of data sent/received by the process during that
565 # stap nettop.stp -c "sleep 0.2"
568 network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server
571 The nfsd_unlink.stp script lists the ip address and file name each
572 time time a file is being removed or unlinked by the nfsd. This
573 script is run on the nfs server.
575 # stap nfsd_unlink.stp -c "sleep 0.2"
578 network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space
579 keywords: network tcp process
581 The sk_stream-wait_memory.stp prints a time stamp, executable, and
582 pid each time a process blocks due to the send buffer being full. A
583 similar entry is printed each time a process continues because there
584 is room in the buffer.
586 # stap sk_stream_wait_memory.stp -c "sleep 0.2"
589 network/socket-trace.stp - Trace Functions Called in Network Socket Code
590 keywords: network socket
592 The script instruments each of the functions in the Linux kernel's
593 net/socket.c file. The script prints out trace data. The first
594 element of a line is time delta in microseconds from the previous
595 entry. This is followed by the command name and the PID. The "->" and
596 "<-" indicates function entry and function exit, respectively. The
597 last element of the line is the function name.
599 # stap socket-trace.stp -c "sleep 0.2"
602 network/socktop - Periodically Summarize Socket Activity on the System
603 keywords: network socket
605 The socktop script periodically prints out a list of the processes
606 with the highest socket activity. Command line options for the
607 script allow filtering to focus on particular types of sockets. The
608 "-h" option lists socktop script's filtering options.
613 network/tcp_connections.stp - Track Creation of Incoming TCP Connections
614 keywords: network tcp socket
616 The tcp_connections.stp script prints information for each new
617 incoming TCP connection accepted by the computer. The information
618 includes the UID, the command accepting the connection, the PID of
619 the command, the port the connection is on, and the IP address of the
620 originator of the request.
622 # stap tcp_connections.stp -c "sleep 0.2"
625 network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10
626 keywords: network tcp socket
628 Run the tcp_init_cwnd.stp script in the background to override a
629 kernel's default tcp cwnd value to 10, which has been found to
630 improve latency for web server type workloads. The script prints a
631 count of cwnd value changes when it is stopped.
633 # stap -g tcp_init_cwnd.stp -c "sleep 0.2"
636 network/tcp_trace.stp - TCP Connection Tracing Utility
637 keywords: network trace
639 This scripts traces a given TCP connection based on the filter
640 parameters given by the user. The indexing is done by the 4 tuples
641 local address, remote address, local port, remote port.
643 # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1
646 network/tcpdumplike.stp - Dump of Received TCP Packets
647 keywords: network traffic
649 The tcpdumplike.stp prints out a line for each TCP packet received.
650 Each line includes the source and destination IP addresses, the
651 source and destination ports, and flags.
653 # stap tcpdumplike.stp -c "sleep 0.2"
656 network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets
657 keywords: network statistics
659 The tcpipstat script collects and displays network statistics related
660 to individual TCP sockets or groups of sockets. The statistics that
661 are collected are simular to that of the command netstat -s, only
662 sorted and grouped by individual sockets.
664 # stap tcpipstat.stp timeout=1
667 process/chng_cpu.stp - Monitor Changes in Processor Executing a Task
670 The chng_cpu.stp script takes an argument which is the executable
671 name of the task it should monitor. Each time a task with that
672 executable name is found running on a different processor, the script
673 prints out the thread id (tid), the executable name, the processor
674 now running the task, the thread state, and a backtrace showing the
675 kernel functions that triggered the running of the task on the
678 # stap chng_cpu.stp -c "sleep 0.2" bash
681 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
682 keywords: process scheduler time tracepoint interrupt
684 The cycle_thief.stp script instruments the scheduler and IRQ handler
685 to determine which processes and interrupts are competing with the
686 specified task for the cpu cycles. This script uses the '-c' or '-x'
687 options to focus on a specific task. The script output the number of
688 times the task migrates between processors, histograms showing the
689 length of time on and off processor, lists of processes running while
690 the task is off the processor, and the interrupts that occurred while
691 the task was running.
693 # stap cycle_thief.stp -c "sleep 0.2"
696 process/errsnoop.stp - Tabulate System Call Errors
697 keywords: process syscall
699 Prints a periodic tabular report about failing system calls, by
700 process and by syscall failure. The first optional argument
701 specifies the reporting interval (in seconds, default 5); the second
702 optional argument gives a screen height (number of lines in the
705 # stap errsnoop.stp 1 10 -c "sleep 1"
708 process/forktracker.stp - Trace Creation of Processes
709 keywords: process scheduler
711 The forktracker.stp script prints out a time-stamped entry showing
712 each fork and exec operation on the machine. This can be useful to
713 determine what process is creating a flurry of short-lived processes.
715 # stap forktracker.stp -c "sleep 0.2"
718 process/futexes.stp - System-Wide Futex Contention
719 keywords: syscall locking futex
721 The script watches the futex syscall on the system. On exit the
722 futexes address, the number of contentions, and the average time for
723 each contention on the futex are printed from lowest pid number to
726 # stap futexes.stp -c "sleep 0.2"
729 process/migrate.stp - Track the Migration of Specific Executables
732 The migrate.stp script takes an argument which is the executable name
733 of the task it should monitor. Each time a task with that executable
734 name migrates between processors an entry is printed with the process
735 id (pid), the executable name, the processor off loading the task,
736 and the process taking the task. Note that the task may or may not be
737 executing at the time of the migration.
739 # stap migrate.stp -c "sleep 0.2" bash
742 process/noptrace.stp - Disable ptrace from Hierarchies of Processes
743 keywords: process security
745 Blocks ptrace(2) attempts from processes identified by stap -c/-x, as
746 also specifiable from /proc/systemtap/stap_XXX/ control files.
747 Processes may be added or removed from the blocked list.
749 # stap -g noptrace.stp -c 'strace ls || true'
752 process/pfiles.stp - Print Process File Descriptors
753 keywords: process file
755 Run pfiles.stp to produce a human-readable summary of all open file
756 descriptors of a given process. Specify the process-id as -x PID for
759 # stap -g pfiles.stp -x $$
762 process/plimit.stp - Print Resource Limits of Process
765 The script prints a variety of resource limits for a given pid, like
766 /proc/$$/limits on recent kernels.
768 # stap -g plimit.stp $$
771 process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands
772 keywords: process backtrace
774 The pstrace_exec.stp script watches each exec operation. If the exec
775 contains a substring that matches the script's command-line argument,
776 it prints out that process and all of its ancestors.
778 # stap pstrace_exec.stp -c "sleep 0.2" bash
781 process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints
782 keywords: process scheduler time tracepoint
784 The schedtimes.stp script instruments the scheduler to track the
785 amount of time that each process spends in running, sleeping,
786 queuing, and waiting for io. On exit the script prints out the
787 accumulated time for each state of processes observed. Optionally,
788 this script can be used with the '-c' or '-x' options to focus on a
791 # stap schedtimes.stp -c "sleep 0.2"
794 process/sig_by_pid.stp - Signal Counts by Process ID
797 Print signal counts by process ID in descending order.
799 # stap sig_by_pid.stp -c "sleep 0.2"
802 process/sig_by_proc.stp - Signal Counts by Process Name
805 Print signal counts by process name in descending order.
807 # stap sig_by_proc.stp -c "sleep 0.2"
810 process/sigkill.stp - Track SIGKILL Signals
813 The script traces any SIGKILL signals. When that SIGKILL signal is
814 sent to a process, the script prints out the signal name, the
815 destination executable and process ID, the executable name and user
816 ID that sents the signal.
818 # stap sigkill.stp -c "sleep 0.2"
821 process/sigmon.stp - Track a Particular Signal to a Specific Process
824 The script watches for a particular signal sent to a specific
825 process. When that signal is sent to the specified process, the
826 script prints out the PID and executable of the process sending the
827 signal, the PID and executable name of the process receiving the
828 signal, and the signal number and name.
830 # stap sigmon.stp -c "sleep 0.2" SIGKILL
833 process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations
834 keywords: io scheduler backtrace
836 The script monitors the time that threads spend in waiting for IO
837 operations (in "D" state) in the wait_for_completion function. If a
838 thread spends over 10ms, its name and backtrace is printed, and later
839 so is the total delay.
841 # stap sleepingBeauties.stp -c "sleep 0.2"
844 process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls
845 keywords: syscall nanosleep
847 The script watches each nanosleep syscall on the system. At the end
848 of each nanosleep syscall the script prints out a line with a
849 timestamp in microseconds, the pid, the executable name in
850 parentheses, the "nanosleep:" key, and the duration of the sleep in
853 # stap sleeptime.stp -c "sleep 0.2"
856 process/syscalls_by_pid.stp - System-Wide Count of Syscalls by PID
859 The script watches all syscall on the system. On exit the script
860 prints a list showing the number of systemcalls executed by each PID
861 ordered from greatest to least number of syscalls.
863 # stap syscalls_by_pid.stp -c "sleep 0.2"
866 process/syscalls_by_proc.stp - System-Wide Count of Syscalls by Executable
869 The script watches all syscall on the system. On exit the script
870 prints a list showing the number of systemcalls executed by each
871 executable ordered from greatest to least number of syscalls.
873 # stap syscalls_by_proc.stp -c "sleep 0.2"
876 process/syscalltimes - System-Wide Syscall Statistics with Filtering
879 Combination shell/systemtap script to measure system call counts and
880 times. Can be filtered by process IDs, process names and users.
882 # ./syscalltimes -c 'sleep 0.2'
885 process/wait4time.stp - Trace Time Spent in wait4 Syscalls
886 keywords: syscall process
888 The script watches each wait4 syscall on the system. At the end of
889 each wait4 syscall the script prints out a line with a timestamp in
890 microseconds, the pid, the executable name in parentheses, the
891 "wait4:" key, the duration of the wait and the PID that the wait4 was
892 waiting for. If the waited for PID is not specified , it is "-1".
894 # stap wait4time.stp -c "sleep 0.2"
897 profiling/errno.stp - Show Which Processes and System Calls Return Errors Most Frequently
900 On exit the errno.stp script provides a sorted list showing which
901 combination of PID, system call, and error occur most frequently.
903 # stap errno.stp -c "sleep 0.2"
906 profiling/fntimes.stp - Show Functions Taking Longer Than Usual
909 The fntimes.stp script monitors the execution time history of a given
910 function family (assumed non-recursive). Each time (beyond a warmup
911 interval) is then compared to the historical maximum. If it exceeds
912 a certain threshold (250%), a message is printed.
914 # stap fntimes.stp 'kernel.function("sys_*")' -c "sleep 7"
917 profiling/functioncallcount.stp - Count Times Functions Are Called
918 keywords: profiling function
920 The functioncallcount.stp script takes one argument, a list of
921 functions to probe. The script will run and count the number of times
922 that each of the functions on the list is called. On exit the script
923 will print a sorted list from most frequently to least frequently
926 # stap functioncallcount.stp "*@mm/*.c" -c "sleep 0.2"
929 profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping
932 The latencytap.stp script collects data on the intervals processes
933 are deactivated (sleeping). The script categorizes the reasons for
934 the sleeps by analyzing the backtraces and displays a sorted list of
935 the top 20 causes from largest total sum time sleeping to smallest.
936 The output is updated every 30 seconds. The script needs to be
937 compiled with the '--all-modules' option to produce reasons for
938 sleeps caused by modules. Optionally, this script can be used with
939 the '-c' or '-x' options to focus on a specific PID.
941 # stap latencytap.stp --all-modules -c "sleep 0.2"
944 profiling/linetimes.stp - Show Time Spent on Each Line of a Function
947 The linetimes.stp script takes two arguments: where to find the
948 function and the function name. linetimes.stp will instrument each
949 line in the function. It will print out the number of times that the
950 function is called, a table with the average and maximum time each
951 line takes, and control flow information when the script exits.
953 # stap linetimes.stp kernel sys_nanosleep -c "sleep 0.2"
956 profiling/periodic.stp - Show the Period of the Various Timers on the System
959 The periodic.stp script uses the kernel.trace("timer_expire_entry")
960 tracepoint to collect data on period and frequency of the various
961 timers on the system. The script displays a sorted list of the
962 timers observed on the system from most frequent to least frequent.
963 The script needs to be compiled with the '--all-modules' option to
964 produce list the function names. Optionally, this script can be used
965 with a numerical argument to indicate the interval in seconds between
968 # stap periodic.stp --all-modules -c "sleep 0.2"
971 profiling/pf2.stp - Profile Kernel Functions
974 The pf2.stp script sets up time-based sampling. Every five seconds it
975 prints out a sorted list with the top ten kernel functions with
978 # stap pf2.stp -c "sleep 0.2"
981 profiling/pf3.stp - Profile Kernel Functions
984 The pf3.stp script sets up time-based sampling. Every five seconds it
985 prints out a sorted list with the top twenty kernel and/or user
986 functions with samples. Use any of --ldd, --all-modules, -d MODULE,
987 -d /PATH/TO/EXEC to add more symbolic info.
989 # stap pf3.stp -c "sleep 6" --all-modules --ldd
992 profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler
993 keywords: profiling function
995 The sched_switch.stp script takes two arguments, first argument can
996 be "pid" or "name" to indicate what is being passed as second
997 argument. The script will trace the process based on pid/name and
998 print the scheduler switches happening with the process. If no
999 arguments are passed, it displays all the scheduler switches. This
1000 can be used to understand which tasks schedule out the current
1001 process being traced, and when it gets scheduled in again.
1003 # stap sched_switch.stp -c "sleep 1"
1006 profiling/thread-times.stp - Profile Kernel Functions
1009 The thread-times.stp script sets up time-based sampling. Every five
1010 seconds it prints out a sorted list with the top twenty threads
1011 occupying the CPUs, broken down as a percentage of user and kernel
1014 # stap thread-times.stp -c "sleep 0.2"
1017 profiling/timeout.stp - Show Processes Doing Polling Operations
1020 The timeout.stp script is based on a blog entry
1021 (http://udrepper.livejournal.com/19041.html) mentioning a need for a
1022 tool to help developers find applications that are polling. The
1023 timeout.stp script monitors systemcall used for polling and records
1024 the systemcalls that timed out rather than returned because some
1025 action occurred. The script updates the screen once a second with the
1026 top twenty processes.
1028 # stap timeout.stp -c "sleep 0.2"
1031 profiling/topsys.stp - Show Processes Doing Polling Operations
1034 The topsys.stp script lists out the top twenty systemcalls for the
1035 previous 5 seconds. The output is sorted from most frequent to least
1038 # stap topsys.stp -c "sleep 0.2"
1041 virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons
1042 keywords: virtualization kvm
1044 The kvm_service_time.stp script tracks the statistics about the
1045 amount of time that the processor left the guest virtual machine for
1046 each exit reason (for example fixing up a page table or handling an
1047 IO operation). When the script exits it prints out the number of
1048 times each exit reason was encountered, the total duration of time it
1049 left the guest VM, the minimum time, the average time, and the
1050 maximum time in microseconds for that exit reason. On Linux 2.6.38
1051 and newer kernel the script can automatically determine whether it is
1052 running on Intel or AMD processors. For older kernels with a
1053 kernel.trace("kvm_exit") tracepoint that does not have the $isa
1054 parameter you can explicitly state the kvm type with a "-G kvm=intel"
1055 or "-G kvm=amd" on the command line.
1057 # stap kvm_service_time.stp -c "sleep 0.2"
1060 virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events
1061 keywords: virtualization qemu kvm
1063 The qemu_count.stp script tallies the number of times each of the
1064 user-space qemu probepoints is encountered. When the script exits, it
1065 prints a list of the number of times each user-space qemu probepoint
1068 # stap qemu_count.stp -c "sleep 0.2"
1071 virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port
1072 keywords: virtualization qemu kvm io
1074 The qemu_io.stp script tallies the number of times each of the IO
1075 port on the guest virtual machines is touched by a input or output
1076 operation. When the script exits, it prints a count of the number of
1077 times each IO port read and written.
1079 # stap qemu_io.stp -c "sleep 0.2"