]>
Commit | Line | Data |
---|---|---|
210ff7bf | 1 | SYSTEMTAP EXAMPLES INDEX BY KEYWORD |
09a98ae0 | 2 | (see also index.txt) |
210ff7bf | 3 | |
3b33d8c4 | 4 | |
81a1a9e8 | 5 | For systemtap version 3.0. |
3b33d8c4 | 6 | |
b9ae0bc4 FCE |
7 | = _BEST = |
8 | ||
9 | general/eventcount.stp - Count Specified Events | |
10 | keywords: _best statistics thread process | |
11 | ||
12 | The script periodically prints a count of specified events and their | |
13 | related tid's over the course of execution. Numerous configuration | |
14 | options exist to control filtering / reporting, see the script | |
15 | source. | |
16 | ||
17 | # stap eventcount.stp syscall.* -c 'sleep 1' | |
18 | ||
19 | ||
20 | general/helloworld.stp - SystemTap "Hello World" Program | |
21 | keywords: _best simple | |
22 | ||
23 | A basic "Hello World" program implemented in SystemTap script. It | |
24 | prints out "hello world" message and then immediately exits. | |
25 | ||
26 | # stap helloworld.stp | |
27 | ||
28 | ||
29 | general/para-callgraph.stp - Callgraph Tracing with Arguments | |
30 | keywords: _best trace callgraph | |
31 | ||
32 | Print a timed per-thread microsecond-timed callgraph, complete with | |
33 | function parameters and return values. The first parameter names the | |
34 | function probe points to trace. The optional second parameter names | |
35 | the probe points for trigger functions, which acts to enable tracing | |
36 | for only those functions that occur while the current thread is | |
37 | nested within the trigger. | |
38 | ||
39 | # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \ | |
40 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" | |
41 | ||
42 | ||
43 | general/varwatch.stp - Watch a Variable Changing Value in a Thread | |
44 | keywords: _best monitoring | |
45 | ||
46 | This script places a set of probes (specified by $1), each of which | |
47 | monitors the state of some context $variable expression (specified by | |
48 | $2). Whenever the value changes, with respect to the active thread, | |
49 | the event is traced. | |
50 | ||
51 | # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \ | |
52 | '$$vars' -c "sleep 1" | |
53 | ||
54 | ||
75ed3b3c FCE |
55 | general/whythefail.stp - Why did the function fail? |
56 | keywords: speculation monitoring function trace _best | |
57 | ||
58 | The whythefail.stp script prints a statement-execution trace for a | |
59 | given function, but only for those runs of the function that ended up | |
60 | with a (configurable) post-return condition. | |
61 | ||
62 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
63 | /root/no-such-file || true' | |
64 | ||
65 | ||
b9ae0bc4 FCE |
66 | network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs |
67 | keywords: _best network traffic | |
68 | ||
69 | The script watches all IPv4 network traffic on the system. On exit | |
70 | the script prints a list showing the number of packets sent along | |
71 | source IP address / destination IP address pair encountered, and the | |
72 | total number of bytes sent among the pair. The list is ordered from | |
73 | greatest to least number of packets seen among the source/destination | |
74 | pairs. | |
75 | ||
76 | # stap netfilter_summary.stp -c "sleep 1" | |
77 | ||
78 | ||
a89c2e98 FL |
79 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs |
80 | keywords: _best network traffic json | |
81 | ||
82 | The script watches all IPv4 network traffic on the system. The data | |
83 | is output in JSON format and includes the number of packets sent | |
84 | along source IP address / destination IP address pair encountered, | |
85 | and the total number of bytes sent among the pair. | |
86 | ||
87 | # stap netfilter_summary_json.stp -c "sleep 1" | |
88 | ||
89 | ||
b9ae0bc4 FCE |
90 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
91 | keywords: _best process scheduler time tracepoint interrupt | |
92 | ||
93 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
94 | to determine which processes and interrupts are competing with the | |
95 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
96 | options to focus on a specific task. The script output the number of | |
97 | times the task migrates between processors, histograms showing the | |
98 | length of time on and off processor, lists of processes running while | |
99 | the task is off the processor, and the interrupts that occurred while | |
100 | the task was running. | |
101 | ||
102 | # stap cycle_thief.stp -c "sleep 1" | |
103 | ||
104 | ||
944d282f AJ |
105 | process/proctop.stp - Periodically Print Process Information With History |
106 | keywords: process scheduler _best | |
107 | ||
108 | Every 5 seconds, print out a list of 25 processes that took the most | |
109 | system time with information about the processes. Includes | |
110 | information on processes that may have exited while the script was | |
111 | running. The script contains configuration options listed in the | |
112 | script source. | |
113 | ||
114 | # stap proctop.stp -c "sleep 1" | |
115 | ||
116 | ||
b9ae0bc4 FCE |
117 | process/strace.stp - Trace system calls |
118 | keywords: _best process syscall | |
119 | ||
120 | The script loosely emulates strace, when applied to individual | |
121 | processes or hierarchies (via -c/-x), or the entire system (without | |
122 | -c/-x). A few output configuration parameters may be set with -G. | |
123 | ||
124 | # stap strace.stp -c "sleep 1" | |
125 | ||
126 | ||
127 | process/thread-business.stp - monitor syscall history | |
128 | keywords: _best process syscall | |
129 | ||
130 | Prints a periodic tabular report about the counts of syscall activity | |
131 | of all threads on the system, along with a textual | |
132 | recent-syscall-history for each | |
133 | ||
134 | # stap thread-business.stp -c "sleep 10" | |
135 | ||
136 | ||
137 | profiling/fntimes.stp - Show Functions Taking Longer Than Usual | |
138 | keywords: _best profiling | |
139 | ||
140 | The fntimes.stp script monitors the execution time history of a given | |
141 | function family (assumed non-recursive). Each time (beyond a warmup | |
142 | interval) is then compared to the historical maximum. If it exceeds | |
143 | a certain threshold (250%), a message is printed. | |
144 | ||
145 | # stap fntimes.stp 'kernel.function("sys_*")' -c "sleep 7" | |
146 | ||
147 | ||
148 | profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping | |
149 | keywords: _best profiling | |
150 | ||
151 | The latencytap.stp script collects data on the intervals processes | |
152 | are deactivated (sleeping). The script categorizes the reasons for | |
153 | the sleeps by analyzing the backtraces and displays a sorted list of | |
154 | the top 20 causes from largest total sum time sleeping to smallest. | |
155 | The output is updated every 30 seconds. The script needs to be | |
156 | compiled with the '--all-modules' option to produce reasons for | |
157 | sleeps caused by modules. Optionally, this script can be used with | |
158 | the '-c' or '-x' options to focus on a specific PID. | |
159 | ||
160 | # stap latencytap.stp --all-modules -c "sleep 1" | |
161 | ||
162 | ||
0a9d3bc8 FCE |
163 | profiling/linetimes.stp - Show Time Spent on Each Line of a Function |
164 | keywords: profiling _best | |
165 | ||
166 | The linetimes.stp script takes two arguments: where to find the | |
167 | function and the function name. linetimes.stp will instrument each | |
168 | line in the function. It will print out the number of times that the | |
169 | function is called, a table with the average and maximum time each | |
170 | line takes, and control flow information when the script exits. | |
171 | ||
172 | # stap linetimes.stp kernel sys_nanosleep -c "sleep 1" | |
173 | ||
174 | ||
b9ae0bc4 | 175 | profiling/pf4.stp - Profile Kernel/User Backtraces |
14a364c1 | 176 | keywords: _best profiling backtrace |
b9ae0bc4 FCE |
177 | |
178 | The pf4.stp script sets up time-based sampling. Every five seconds it | |
179 | prints out a sorted list with the top twenty kernel and/or user stack | |
180 | backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d | |
181 | MODULE, -d /PATH/TO/EXEC to add more symbolic info. | |
182 | ||
183 | # stap pf4.stp -c "sleep 6" --all-modules --ldd | |
184 | ||
185 | ||
96ef4c2a FCE |
186 | profiling/thread-times.stp - Profile Kernel Functions |
187 | keywords: _best profiling | |
188 | ||
189 | The thread-times.stp script sets up time-based sampling. Every five | |
190 | seconds it prints out a sorted list with the top twenty threads | |
191 | occupying the CPUs, broken down as a percentage of user and kernel | |
192 | time. | |
193 | ||
194 | # stap thread-times.stp -c "sleep 1" | |
195 | ||
196 | ||
b9ae0bc4 FCE |
197 | stapgames/2048.stp - 2048 |
198 | keywords: _best stapgames | |
199 | ||
200 | The modern classic 2048 sliding-tiles game, using local keyboard and | |
201 | ansi animation. | |
202 | ||
203 | # stap -p4 2048.stp | |
204 | ||
205 | ||
206 | virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons | |
207 | keywords: _best virtualization kvm | |
208 | ||
209 | The kvm_service_time.stp script tracks the statistics about the | |
210 | amount of time that the processor left the guest virtual machine for | |
211 | each exit reason (for example fixing up a page table or handling an | |
212 | IO operation). When the script exits it prints out the number of | |
213 | times each exit reason was encountered, the total duration of time it | |
214 | left the guest VM, the minimum time, the average time, and the | |
215 | maximum time in microseconds for that exit reason. On Linux 2.6.38 | |
216 | and newer kernel the script can automatically determine whether it is | |
217 | running on Intel or AMD processors. For older kernels with a | |
218 | kernel.trace("kvm_exit") tracepoint that does not have the $isa | |
219 | parameter you can explicitly state the kvm type with a "-G kvm=intel" | |
220 | or "-G kvm=amd" on the command line. | |
221 | ||
222 | # stap kvm_service_time.stp -c "sleep 1" | |
223 | ||
224 | ||
a8a7b9f4 FCE |
225 | = AUDIT = |
226 | ||
7080b0c3 | 227 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
228 | keywords: monitoring security audit backtrace |
229 | ||
7080b0c3 FCE |
230 | Attaches to the kernel audit-log paths (also used by libaudit), and |
231 | log every record being sent, along with a user-space backtrace of the | |
232 | process that caused it. | |
a8a7b9f4 | 233 | |
54a385da | 234 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
235 | |
236 | ||
8b88e771 FCE |
237 | = AUTOFS = |
238 | ||
cf29c85e | 239 | network/autofs4.stp - Watch autofs4 Operations |
8b88e771 FCE |
240 | keywords: network autofs nfs |
241 | ||
242 | Trace key autofs4 operations such as mounting or unmounting remote | |
243 | filesystems. | |
244 | ||
6a51196c FCE |
245 | # stap autofs4.stp -c "sleep 1" |
246 | ||
8b88e771 | 247 | |
210ff7bf FCE |
248 | = BACKTRACE = |
249 | ||
cf29c85e | 250 | general/watchdog.stp - Watchdog Timer for Arbitrary Events |
2dfeb3bf WC |
251 | keywords: watchdog backtrace |
252 | ||
253 | The watchdog.stp script provides a watchdog timer mechanism for | |
254 | arbitrary events. The script takes three arguments: the events to | |
255 | start watchdog timer, the event to stop the watchdog timer, and the | |
256 | time in millseconds for the watchdog. If the watchdog timer is | |
257 | exceed, the script will trigger a stack backtrace of the user-process | |
258 | that timed out using pstack. This script can be used to diagnose what | |
259 | the userspace application is doing when a slower than expected | |
260 | operation occurs. | |
261 | ||
262 | # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \ | |
1f115be7 | 263 | -c "sleep 1" |
2dfeb3bf WC |
264 | |
265 | ||
cf29c85e | 266 | interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt |
1cc8a4c7 WC |
267 | keywords: interrupt backtrace |
268 | ||
269 | The Linux kernel function smp_call_function causes expensive | |
270 | inter-processor interrupts (IPIs). The scf.stp script tallies the | |
271 | processes and backtraces causing the interprocessor interrupts to | |
272 | identify the cause of the expensive IPI. On exit the script prints | |
273 | the tallies in descending frequency. | |
274 | ||
1f115be7 | 275 | # stap scf.stp -c "sleep 1" |
6a51196c | 276 | |
1cc8a4c7 | 277 | |
210ff7bf | 278 | io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call |
09a98ae0 | 279 | keywords: io backtrace |
210ff7bf FCE |
280 | |
281 | When a reschedule occurs during an AIO io_submit call, accumulate the | |
282 | traceback in a histogram. When the script exits prints out a sorted | |
283 | list from most common to least common backtrace. | |
284 | ||
1f115be7 | 285 | # stap io_submit.stp -c "sleep 1" |
6a51196c | 286 | |
210ff7bf | 287 | |
f70a0bc5 FCE |
288 | memory/last_100_frees.stp - Log recent free(3) calls. |
289 | keywords: memory process backtrace | |
290 | ||
291 | This script reports on the last few free(3) libc calls done by | |
292 | processes (possibly restricted by stap -x/-c), along with a userspace | |
293 | backtrace at those moments. | |
294 | ||
295 | # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd | |
296 | ||
297 | ||
7080b0c3 | 298 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
299 | keywords: monitoring security audit backtrace |
300 | ||
7080b0c3 FCE |
301 | Attaches to the kernel audit-log paths (also used by libaudit), and |
302 | log every record being sent, along with a user-space backtrace of the | |
303 | process that caused it. | |
a8a7b9f4 | 304 | |
54a385da | 305 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
306 | |
307 | ||
04c36a84 MW |
308 | process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands |
309 | keywords: process backtrace | |
310 | ||
311 | The pstrace_exec.stp script watches each exec operation. If the exec | |
312 | contains a substring that matches the script's command-line argument, | |
313 | it prints out that process and all of its ancestors. | |
314 | ||
1f115be7 | 315 | # stap pstrace_exec.stp -c "sleep 1" bash |
04c36a84 MW |
316 | |
317 | ||
cf29c85e | 318 | process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations |
09a98ae0 | 319 | keywords: io scheduler backtrace |
cae71dd3 | 320 | |
f3c4da44 | 321 | The script monitors the time that threads spend in waiting for IO |
cae71dd3 FCE |
322 | operations (in "D" state) in the wait_for_completion function. If a |
323 | thread spends over 10ms, its name and backtrace is printed, and later | |
324 | so is the total delay. | |
325 | ||
1f115be7 | 326 | # stap sleepingBeauties.stp -c "sleep 1" |
6a51196c | 327 | |
cae71dd3 | 328 | |
14a364c1 FCE |
329 | profiling/pf4.stp - Profile Kernel/User Backtraces |
330 | keywords: _best profiling backtrace | |
331 | ||
332 | The pf4.stp script sets up time-based sampling. Every five seconds it | |
333 | prints out a sorted list with the top twenty kernel and/or user stack | |
334 | backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d | |
335 | MODULE, -d /PATH/TO/EXEC to add more symbolic info. | |
336 | ||
337 | # stap pf4.stp -c "sleep 6" --all-modules --ldd | |
338 | ||
339 | ||
210ff7bf FCE |
340 | = CALLGRAPH = |
341 | ||
81a1a9e8 FCE |
342 | general/callgraph.stp - Callgraph Tracing |
343 | keywords: simple trace callgraph | |
344 | ||
345 | Print a timed per-thread microsecond-timed nested callgraph. The | |
346 | first parameter names the function probe points to trace. | |
347 | ||
348 | # stap callgraph.stp 'kernel.function("*@fs/proc*.c")' -c "cat \ | |
349 | /proc/sys/vm/* || true" | |
350 | ||
351 | ||
cf29c85e | 352 | general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments |
2515d897 JS |
353 | keywords: trace callgraph |
354 | ||
03568589 FCE |
355 | Print a timed per-thread microsecond-timed callgraph, complete with |
356 | pretty-printed function parameters and return values. The first | |
357 | parameter names the function probe points to trace. The optional | |
358 | second parameter names the probe points for trigger functions, which | |
359 | acts to enable tracing for only those functions that occur while the | |
360 | current thread is nested within the trigger. | |
2515d897 | 361 | |
6a51196c | 362 | # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 363 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 364 | |
2515d897 | 365 | |
cf29c85e | 366 | general/para-callgraph.stp - Callgraph Tracing with Arguments |
b9ae0bc4 | 367 | keywords: _best trace callgraph |
210ff7bf | 368 | |
03568589 FCE |
369 | Print a timed per-thread microsecond-timed callgraph, complete with |
370 | function parameters and return values. The first parameter names the | |
371 | function probe points to trace. The optional second parameter names | |
372 | the probe points for trigger functions, which acts to enable tracing | |
373 | for only those functions that occur while the current thread is | |
374 | nested within the trigger. | |
210ff7bf | 375 | |
6a51196c | 376 | # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 377 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 378 | |
210ff7bf FCE |
379 | |
380 | = CPU = | |
381 | ||
382 | general/graphs.stp - Graphing Disk and CPU Utilization | |
cf29c85e | 383 | keywords: disk cpu utilization |
210ff7bf FCE |
384 | |
385 | The script tracks the disk and CPU utilization. The resulting output | |
386 | of the script can be piped into gnuplot to generate a graph of disk | |
387 | and CPU USE. | |
388 | ||
1f115be7 | 389 | # stap graphs.stp -c "sleep 1" |
6a51196c | 390 | |
210ff7bf | 391 | |
53e5699f JS |
392 | = DEVICE = |
393 | ||
394 | network/netdev.stp - Trace Activity on Network Devices | |
395 | keywords: network device traffic | |
396 | ||
397 | The netdev.stp script traces configuration and transmit/receive | |
398 | activity on network devices. | |
399 | ||
1f115be7 | 400 | # stap netdev.stp -c "sleep 1" |
6a51196c | 401 | |
53e5699f | 402 | |
9e749e50 JS |
403 | = DIAGRAM = |
404 | ||
405 | process/pstree.stp - Generates a process diagram in DOT form. | |
406 | keywords: process diagram | |
407 | ||
408 | The pstree.stp script generates a process diagram in DOT form. For | |
409 | instance, it may be useful on a 'make' command to see all the | |
410 | processes that are started. | |
411 | ||
1f115be7 | 412 | # stap pstree.stp -c "sleep 1" |
9e749e50 JS |
413 | |
414 | ||
210ff7bf FCE |
415 | = DISK = |
416 | ||
417 | general/graphs.stp - Graphing Disk and CPU Utilization | |
cf29c85e | 418 | keywords: disk cpu utilization |
210ff7bf FCE |
419 | |
420 | The script tracks the disk and CPU utilization. The resulting output | |
421 | of the script can be piped into gnuplot to generate a graph of disk | |
422 | and CPU USE. | |
423 | ||
1f115be7 | 424 | # stap graphs.stp -c "sleep 1" |
6a51196c | 425 | |
210ff7bf | 426 | |
b96bcce8 WC |
427 | io/deviceseeks.stp - Histograms of Seek Behavior for Each Device |
428 | keywords: disk | |
429 | ||
430 | The deviceseeks.stp script generates a histogram showing the | |
875e1b27 | 431 | frequency of different sized seeks (in sectors) on each device. |
b96bcce8 | 432 | |
1f115be7 | 433 | # stap deviceseeks.stp -c "sleep 1" |
6a51196c | 434 | |
b96bcce8 | 435 | |
210ff7bf | 436 | io/disktop.stp - Summarize Disk Read/Write Traffic |
09a98ae0 | 437 | keywords: disk |
210ff7bf FCE |
438 | |
439 | Get the status of reading/writing disk every 5 seconds, output top | |
440 | ten entries during that period. | |
441 | ||
1f115be7 | 442 | # stap disktop.stp -c "sleep 1" |
6a51196c | 443 | |
210ff7bf | 444 | |
7fcecf19 MW |
445 | io/enospc.stp - Report an disk out-of-space condition. |
446 | keywords: disk filesystem | |
447 | ||
448 | This script monitors a filesystem implementations for early internal | |
449 | indications of ENOSPC, and reports these to the system logger and the | |
450 | systemtap console. | |
451 | ||
1f115be7 | 452 | # stap enospc.stp -c "sleep 1" |
7fcecf19 MW |
453 | |
454 | ||
cf29c85e | 455 | io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices |
74e5c5cf FCE |
456 | keywords: io monitoring disk |
457 | ||
458 | The mbrwatch.stp script reports any attempted reads/writes of the | |
459 | first few sectors of a raw block device. | |
460 | ||
461 | # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \ | |
199197b8 FCE |
462 | /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \ |
463 | head -1`" | |
74e5c5cf FCE |
464 | |
465 | ||
cf29c85e | 466 | network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server |
b7edd338 WC |
467 | keywords: nfs disk |
468 | ||
469 | The nfsd_unlink.stp script lists the ip address and file name each | |
470 | time time a file is being removed or unlinked by the nfsd. This | |
471 | script is run on the nfs server. | |
472 | ||
1f115be7 | 473 | # stap nfsd_unlink.stp -c "sleep 1" |
b7edd338 WC |
474 | |
475 | ||
cf29c85e | 476 | = FILE = |
db5bfb14 | 477 | |
cf29c85e FCE |
478 | io/iotime.stp - Trace Time Spent in Read and Write for Files |
479 | keywords: profiling syscall io file | |
db5bfb14 | 480 | |
cf29c85e FCE |
481 | The script watches each open, close, read, and write syscalls on the |
482 | system. For each file the scripts observes opened it accumulates the | |
483 | amount of wall clock time spent in read and write operations and the | |
484 | number of bytes read and written. When a file is closed the script | |
485 | prints out a pair of lines for the file. Both lines begin with a | |
486 | timestamp in microseconds, the PID number, and the executable name in | |
487 | parentheses. The first line with the "access" keyword lists the file | |
488 | name, the attempted number of bytes for the read and write | |
489 | operations. The second line with the "iotime" keyword list the file | |
490 | name and the number of microseconds accumulated in the read and write | |
491 | syscalls. | |
db5bfb14 | 492 | |
1f115be7 | 493 | # stap iotime.stp -c "sleep 1" |
db5bfb14 | 494 | |
ecf33ff4 | 495 | |
cf29c85e FCE |
496 | process/pfiles.stp - Print Process File Descriptors |
497 | keywords: process file | |
ecf33ff4 ET |
498 | |
499 | Run pfiles.stp to produce a human-readable summary of all open file | |
500 | descriptors of a given process. Specify the process-id as -x PID for | |
501 | fastest performance. | |
502 | ||
6a51196c FCE |
503 | # stap -g pfiles.stp -x $$ |
504 | ||
ecf33ff4 | 505 | |
ac505f97 JS |
506 | = FILESYSTEM = |
507 | ||
508 | general/badname.stp - Bad Filename Filter | |
1ca36a99 | 509 | keywords: filesystem guru |
ac505f97 JS |
510 | |
511 | The badname.stp script shows how one could prevent the creation of | |
6287a9e6 | 512 | files with undesirable names using guru mode. |
ac505f97 | 513 | |
15ecb1fe FCE |
514 | # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \ |
515 | denied" | |
6a51196c | 516 | |
ac505f97 | 517 | |
7fcecf19 MW |
518 | io/enospc.stp - Report an disk out-of-space condition. |
519 | keywords: disk filesystem | |
520 | ||
521 | This script monitors a filesystem implementations for early internal | |
522 | indications of ENOSPC, and reports these to the system logger and the | |
523 | systemtap console. | |
524 | ||
1f115be7 | 525 | # stap enospc.stp -c "sleep 1" |
7fcecf19 MW |
526 | |
527 | ||
0e81b7f5 FCE |
528 | lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo) |
529 | keywords: io filesystem | |
530 | ||
531 | This dynamically traces two common file system functions: | |
532 | do_sync_read() and do_sync_write(), and reports a histogram | |
533 | distribution of latency. Many, but not all, file systems and | |
534 | workloads use these functions. Tracing their time provides one view | |
535 | of suffered file system latency. | |
536 | ||
537 | # stap fslatency-nd.stp 1 1 | |
538 | ||
539 | ||
540 | lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo) | |
541 | keywords: io filesystem | |
542 | ||
543 | This dynamically traces two common file system functions: | |
544 | do_sync_read() and do_sync_write(), and shows details of each call | |
545 | that is slower than a threshold. Many, but not all, file systems and | |
546 | workloads use these functions. Tracing their time provides one view | |
547 | of suffered file system latency. | |
548 | ||
549 | # stap fsslower-nd.stp -c "sleep 1" | |
550 | ||
551 | ||
0e0b566a WC |
552 | = FORMAT = |
553 | ||
554 | general/ansi_colors.stp - Color Table for ansi_set_color2() and ansi_set_color3() | |
555 | keywords: format | |
556 | ||
557 | The script prints a table showing the available color combinations | |
558 | for the ansi_set_color2() and ans_set_color3() functions in the | |
559 | ansi.stp tapset. | |
560 | ||
6a51196c FCE |
561 | # stap ansi_colors.stp |
562 | ||
0e0b566a WC |
563 | |
564 | general/ansi_colors2.stp - Show Attribues in Table for ansi_set_color3() | |
565 | keywords: format | |
566 | ||
567 | The script prints a table showing the available attributes (bold, | |
568 | underline, and inverse) with color combinations for the | |
569 | ans_set_color3() function in the ansi.stp tapset. | |
570 | ||
6a51196c FCE |
571 | # stap ansi_colors2.stp |
572 | ||
0e0b566a | 573 | |
fce7a922 WC |
574 | = FUNCTION = |
575 | ||
cf29c85e FCE |
576 | general/func_time_stats.stp - Function Time Statistics |
577 | keywords: function statistics | |
fce7a922 WC |
578 | |
579 | The func_time_stats.stp script tracks the wall clock time for each | |
580 | invocation of the function probe listed as the first command line | |
581 | argument. When the script exits it prints out the minimum, average, | |
582 | and maximum times in microseconds followed by a count of times that | |
583 | the function was called and a histogram showing the distributions of | |
584 | times. | |
585 | ||
1f115be7 | 586 | # stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1" |
fce7a922 WC |
587 | |
588 | ||
75ed3b3c FCE |
589 | general/whythefail.stp - Why did the function fail? |
590 | keywords: speculation monitoring function trace _best | |
591 | ||
592 | The whythefail.stp script prints a statement-execution trace for a | |
593 | given function, but only for those runs of the function that ended up | |
594 | with a (configurable) post-return condition. | |
595 | ||
596 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
597 | /root/no-such-file || true' | |
598 | ||
599 | ||
cf29c85e FCE |
600 | profiling/functioncallcount.stp - Count Times Functions Are Called |
601 | keywords: profiling function | |
210ff7bf FCE |
602 | |
603 | The functioncallcount.stp script takes one argument, a list of | |
604 | functions to probe. The script will run and count the number of times | |
605 | that each of the functions on the list is called. On exit the script | |
606 | will print a sorted list from most frequently to least frequently | |
607 | called function. | |
608 | ||
ca5d5a88 | 609 | # stap -w functioncallcount.stp "*@mm/*.c" -c "sleep 1" |
6a51196c | 610 | |
210ff7bf | 611 | |
cf29c85e FCE |
612 | profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler |
613 | keywords: profiling function | |
0449af03 JS |
614 | |
615 | The sched_switch.stp script takes two arguments, first argument can | |
616 | be "pid" or "name" to indicate what is being passed as second | |
617 | argument. The script will trace the process based on pid/name and | |
618 | print the scheduler switches happening with the process. If no | |
619 | arguments are passed, it displays all the scheduler switches. This | |
620 | can be used to understand which tasks schedule out the current | |
621 | process being traced, and when it gets scheduled in again. | |
622 | ||
6a51196c FCE |
623 | # stap sched_switch.stp -c "sleep 1" |
624 | ||
0449af03 | 625 | |
210ff7bf FCE |
626 | = FUTEX = |
627 | ||
628 | process/futexes.stp - System-Wide Futex Contention | |
09a98ae0 | 629 | keywords: syscall locking futex |
210ff7bf FCE |
630 | |
631 | The script watches the futex syscall on the system. On exit the | |
0fb74e68 | 632 | futex's address, the number of contentions, and the average time for |
210ff7bf FCE |
633 | each contention on the futex are printed from lowest pid number to |
634 | highest. | |
635 | ||
1f115be7 | 636 | # stap futexes.stp -c "sleep 1" |
6a51196c | 637 | |
210ff7bf | 638 | |
0fb74e68 JS |
639 | process/futexes2.stp - System-Wide Shared Futex Contention |
640 | keywords: syscall locking futex | |
641 | ||
642 | The script watches just shared futex syscalls on the system. On exit | |
643 | the futex's key, the number of contentions, and the average time for | |
644 | each contention on the futex are printed from lowest pid number to | |
645 | highest. | |
646 | ||
1f115be7 | 647 | # stap futexes2.stp -c "sleep 1" |
0fb74e68 JS |
648 | |
649 | ||
1ca36a99 FCE |
650 | = GURU = |
651 | ||
652 | general/badname.stp - Bad Filename Filter | |
653 | keywords: filesystem guru | |
654 | ||
655 | The badname.stp script shows how one could prevent the creation of | |
656 | files with undesirable names using guru mode. | |
657 | ||
658 | # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \ | |
659 | denied" | |
660 | ||
661 | ||
47300c64 | 662 | io/eatmydata.stp - disable fsync |
1ca36a99 FCE |
663 | keywords: io guru simple |
664 | ||
665 | Suppresses fsync() syscalls from processes identified by stap -c/-x | |
00b943ad FCE |
666 | by turning them into presumed-faster fsync() on some dummy or other |
667 | file descriptor | |
1ca36a99 FCE |
668 | |
669 | # stap -g eatmydata.stp -c 'strace ls || true' | |
670 | ||
671 | ||
672 | io/ttyspy.stp - Monitor TTY Typing | |
673 | keywords: io tty monitoring guru | |
674 | ||
675 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
676 | activity on the system, printing a scrolling record of recent | |
677 | keystrokes, on a per-tty basis. | |
678 | ||
679 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" | |
680 | ||
681 | ||
682 | network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool | |
683 | keywords: network packets guru | |
684 | ||
685 | The script drops the specified number of packets of the specified | |
686 | protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, | |
687 | all incoming packets are dropped. The number of packets to drop can | |
688 | be specified with a positive integer. A value of 0 indicates that | |
689 | packets should be dropped until the user manually exits. | |
690 | ||
691 | # stap -g netfilter_drop.stp TCP 1 -c "sleep 2" | |
692 | ||
693 | ||
694 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 | |
695 | keywords: network tcp socket guru | |
696 | ||
697 | Run the tcp_init_cwnd.stp script in the background to override a | |
698 | kernel's default tcp cwnd value to 10, which has been found to | |
699 | improve latency for web server type workloads. The script prints a | |
700 | count of cwnd value changes when it is stopped. | |
701 | ||
1f115be7 | 702 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
1ca36a99 FCE |
703 | |
704 | ||
705 | process/noptrace.stp - Disable ptrace from Hierarchies of Processes | |
706 | keywords: process security guru | |
707 | ||
708 | Blocks ptrace(2) attempts from processes identified by stap -c/-x, as | |
709 | also specifiable from /proc/systemtap/stap_XXX/ control files. | |
710 | Processes may be added or removed from the blocked list. | |
711 | ||
712 | # stap -g noptrace.stp -c 'strace ls || true' | |
713 | ||
714 | ||
715 | process/threadstacks.stp - Override default new-pthread stack sizes | |
716 | keywords: thread guru | |
717 | ||
718 | Overrides default NPTL pthread_create stack size for all new threads | |
719 | created by target processes. Reports one line per process when the | |
720 | related glibc variable __default_stacksize is updated. Moot for | |
721 | glibc versions that support $LIBC_PTHREAD_DEFAULT_STACKSIZE_NP. | |
722 | ||
f70a0bc5 | 723 | # stap -g threadstacks.stp -Gsize=65536 -c "sleep 1" -d `which stap` |
1ca36a99 FCE |
724 | |
725 | ||
1cc8a4c7 WC |
726 | = INTERRUPT = |
727 | ||
cf29c85e | 728 | interrupt/interrupts-by-dev.stp - Record Interrupts on a Per-Device Basis |
6b95efe9 JS |
729 | keywords: interrupt |
730 | ||
731 | The interrupts-by-dev.stp script profiles interrupts received by each | |
732 | device per 100 ms. | |
733 | ||
1f115be7 | 734 | # stap interrupts-by-dev.stp -c "sleep 1" |
6a51196c | 735 | |
6b95efe9 | 736 | |
cf29c85e | 737 | interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt |
1cc8a4c7 WC |
738 | keywords: interrupt backtrace |
739 | ||
740 | The Linux kernel function smp_call_function causes expensive | |
741 | inter-processor interrupts (IPIs). The scf.stp script tallies the | |
742 | processes and backtraces causing the interprocessor interrupts to | |
743 | identify the cause of the expensive IPI. On exit the script prints | |
744 | the tallies in descending frequency. | |
745 | ||
1f115be7 | 746 | # stap scf.stp -c "sleep 1" |
6a51196c | 747 | |
1cc8a4c7 | 748 | |
cf29c85e | 749 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 750 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
751 | |
752 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
753 | to determine which processes and interrupts are competing with the | |
754 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
755 | options to focus on a specific task. The script output the number of | |
756 | times the task migrates between processors, histograms showing the | |
757 | length of time on and off processor, lists of processes running while | |
758 | the task is off the processor, and the interrupts that occurred while | |
759 | the task was running. | |
760 | ||
1f115be7 | 761 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 762 | |
e01bc08e | 763 | |
210ff7bf FCE |
764 | = IO = |
765 | ||
beb59e38 FCE |
766 | general/alias_suffixes.stp - Count I/O Syscalls using Alias Suffixes |
767 | keywords: io statistics | |
768 | ||
769 | alias_suffixes.stp is a demonstration of how alias suffixes in the | |
770 | systemtap language might be used. The script tracks the wall clock | |
771 | time for each invocation of the system calls open, close, read, and | |
772 | write. When the script exists it prints out the minimum, average, and | |
773 | maximum times in microseconds for each system call, followed by a | |
774 | count of times that each syscall was invoked and a histogram showing | |
775 | the distributions of times. | |
776 | ||
1f115be7 | 777 | # stap alias_suffixes.stp -c "sleep 1" |
beb59e38 FCE |
778 | |
779 | ||
47300c64 | 780 | io/eatmydata.stp - disable fsync |
1ca36a99 FCE |
781 | keywords: io guru simple |
782 | ||
783 | Suppresses fsync() syscalls from processes identified by stap -c/-x | |
00b943ad FCE |
784 | by turning them into presumed-faster fsync() on some dummy or other |
785 | file descriptor | |
1ca36a99 FCE |
786 | |
787 | # stap -g eatmydata.stp -c 'strace ls || true' | |
788 | ||
789 | ||
d8878d94 WC |
790 | io/inodewatch.stp - Monitoring Reads and Writes to a File |
791 | keywords: io | |
792 | ||
793 | The inodewatch.stp outputs the executable name and process id each | |
794 | time a read or write occurs to the specified inode on the specified | |
795 | major/minor device. | |
796 | ||
1f115be7 | 797 | # stap inodewatch.stp 0x08 0x01 100 -c "sleep 1" |
6a51196c | 798 | |
d8878d94 | 799 | |
7ad79dbe WC |
800 | io/inodewatch2.stp - Monitoring Attribute Changes to a File |
801 | keywords: io | |
802 | ||
803 | The inodewatch2.stp script outputs the executable name, process id, | |
804 | and attributes each time the attributes are changed on the specified | |
805 | inode on the specified major/minor device. | |
806 | ||
1f115be7 | 807 | # stap inodewatch2.stp 0x08 0x01 100 -c "sleep 1" |
6a51196c | 808 | |
7ad79dbe | 809 | |
210ff7bf | 810 | io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call |
09a98ae0 | 811 | keywords: io backtrace |
210ff7bf FCE |
812 | |
813 | When a reschedule occurs during an AIO io_submit call, accumulate the | |
814 | traceback in a histogram. When the script exits prints out a sorted | |
815 | list from most common to least common backtrace. | |
816 | ||
1f115be7 | 817 | # stap io_submit.stp -c "sleep 1" |
6a51196c | 818 | |
210ff7bf | 819 | |
67f8611b WC |
820 | io/ioblktime.stp - Average Time Block IO Requests Spend in Queue |
821 | keywords: io | |
822 | ||
823 | The ioblktime.stp script tracks the amount of time that each block IO | |
cc20d853 | 824 | requests spend waiting for completion. The script computes the |
f3c4da44 MW |
825 | average waiting time for block IO per device and prints list every 10 |
826 | seconds. In some cases there can be too many outstanding block IO | |
827 | operations and the script may exceed the default number of | |
67f8611b WC |
828 | MAXMAPENTRIES allowed. In this case the allowed number can be |
829 | increased with "-DMAXMAPENTRIES=10000" option on the stap command | |
830 | line. | |
831 | ||
1f115be7 | 832 | # stap ioblktime.stp -c "sleep 1" |
6a51196c | 833 | |
67f8611b | 834 | |
a4f3198f WC |
835 | io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device |
836 | keywords: io profiling | |
837 | ||
838 | The iodevstats.stp script measures the amount of data successfully | |
839 | read and written by all the executables for each io device on the | |
840 | system. The output is sorted from greatest sum of bytes read and | |
841 | written to a device by an executable to the least. The output | |
842 | contains device major/minor number, the count of operations (reads | |
843 | and writes), the totals and averages for the number of bytes read and | |
844 | written. | |
845 | ||
1f115be7 | 846 | # stap iodevstats.stp -c "sleep 1" |
6a51196c | 847 | |
a4f3198f | 848 | |
cf29c85e | 849 | io/iostat-scsi.stp - IO Statistics for SCSI Devices |
54ff5e0c FCE |
850 | keywords: io profiling scsi |
851 | ||
852 | The iostat-scsi.stp script provides a breakdown of the number of blks | |
ad7e33d7 | 853 | read and written on the machine's various SCSI devices. The script |
54ff5e0c FCE |
854 | takes one argument which is the number of seconds between reports. |
855 | ||
1f115be7 | 856 | # stap -g iostat-scsi.stp 1 -c "sleep 1" |
6a51196c | 857 | |
54ff5e0c | 858 | |
3e4444ed WC |
859 | io/iostats.stp - List Executables Reading and Writing the Most Data |
860 | keywords: io profiling | |
861 | ||
862 | The iostat.stp script measures the amount of data successfully read | |
863 | and written by all the executables on the system. The output is | |
864 | sorted from most greatest sum of bytes read and written by an | |
865 | executable to the least. The output contains the count of operations | |
866 | (opens, reads, and writes), the totals and averages for the number of | |
867 | bytes read and written. | |
868 | ||
1f115be7 | 869 | # stap iostats.stp -c "sleep 1" |
6a51196c | 870 | |
3e4444ed | 871 | |
210ff7bf | 872 | io/iotime.stp - Trace Time Spent in Read and Write for Files |
cf29c85e | 873 | keywords: profiling syscall io file |
210ff7bf FCE |
874 | |
875 | The script watches each open, close, read, and write syscalls on the | |
876 | system. For each file the scripts observes opened it accumulates the | |
f3c4da44 | 877 | amount of wall clock time spent in read and write operations and the |
210ff7bf FCE |
878 | number of bytes read and written. When a file is closed the script |
879 | prints out a pair of lines for the file. Both lines begin with a | |
880 | timestamp in microseconds, the PID number, and the executable name in | |
ad7e33d7 | 881 | parentheses. The first line with the "access" keyword lists the file |
210ff7bf FCE |
882 | name, the attempted number of bytes for the read and write |
883 | operations. The second line with the "iotime" keyword list the file | |
884 | name and the number of microseconds accumulated in the read and write | |
885 | syscalls. | |
886 | ||
1f115be7 | 887 | # stap iotime.stp -c "sleep 1" |
6a51196c | 888 | |
210ff7bf | 889 | |
cf29c85e | 890 | io/iotop.stp - Periodically Print IO Activity by Process Name |
09a98ae0 | 891 | keywords: io |
210ff7bf FCE |
892 | |
893 | Every five seconds print out the top ten executables generating I/O | |
894 | traffic during that interval sorted in descending order. | |
895 | ||
1f115be7 | 896 | # stap iotop.stp -c "sleep 1" |
6a51196c | 897 | |
210ff7bf | 898 | |
cf29c85e | 899 | io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices |
74e5c5cf | 900 | keywords: io monitoring disk |
1beb5089 FCE |
901 | |
902 | The mbrwatch.stp script reports any attempted reads/writes of the | |
903 | first few sectors of a raw block device. | |
904 | ||
6a51196c | 905 | # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \ |
199197b8 FCE |
906 | /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \ |
907 | head -1`" | |
6a51196c | 908 | |
1beb5089 | 909 | |
111dd9ac WC |
910 | io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process |
911 | keywords: io profiling | |
912 | ||
913 | The nfs_func_users.stp script counts the uses of NFS functions in the | |
914 | kernel on a per process bases. The output is sorted from the process | |
915 | with the greatest number of NFS functions called to the least. The | |
916 | output contains the executable name, the process number, and the | |
917 | total number of NFS functions called by the process. | |
918 | ||
1f115be7 | 919 | # stap nfs_func_users.stp -c "sleep 1" |
6a51196c | 920 | |
111dd9ac | 921 | |
a11f34b0 FCE |
922 | io/slowvfs.stp - Trace slow vfs opens. |
923 | keywords: io | |
924 | ||
925 | This script prints a line for every kernel vfs_open operation that | |
926 | takes longer than a configurable number of microseconds. Highly | |
927 | contended or remote filesystems are likelier to hit this. | |
928 | ||
929 | # stap slowvfs.stp -G sloth=10 -c 'find /proc >/dev/null' | |
930 | ||
931 | ||
932 | io/switchfile.stp - Switch log files | |
933 | keywords: io | |
934 | ||
935 | Every second print a log message and switch log files every 5 | |
936 | seconds. | |
937 | ||
938 | # stap -o switchfile.stp.out switchfile.stp -c "sleep 8" | |
939 | ||
940 | ||
cf29c85e | 941 | io/traceio.stp - Track Cumulative IO Activity by Process Name |
09a98ae0 | 942 | keywords: io |
210ff7bf FCE |
943 | |
944 | Every second print out the top ten executables sorted in descending | |
945 | order based on cumulative I/O traffic observed. | |
946 | ||
1f115be7 | 947 | # stap traceio.stp -c "sleep 1" |
6a51196c | 948 | |
210ff7bf | 949 | |
cf29c85e | 950 | io/traceio2.stp - Watch IO Activity on a Particular Device |
09a98ae0 | 951 | keywords: io |
210ff7bf FCE |
952 | |
953 | Print out the executable name and process number as reads and writes | |
954 | to the specified device occur. | |
955 | ||
1f115be7 | 956 | # stap traceio2.stp 0x0801 -c "sleep 1" |
6a51196c | 957 | |
210ff7bf | 958 | |
cf29c85e | 959 | io/ttyspy.stp - Monitor TTY Typing |
1ca36a99 | 960 | keywords: io tty monitoring guru |
b7f6cfc5 FCE |
961 | |
962 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
963 | activity on the system, printing a scrolling record of recent | |
964 | keystrokes, on a per-tty basis. | |
965 | ||
6a51196c FCE |
966 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" |
967 | ||
b7f6cfc5 | 968 | |
0e81b7f5 FCE |
969 | lwtools/biolatency-nd.stp - Measure block I/O latency distribution (non-debuginfo) |
970 | keywords: io | |
971 | ||
972 | This measures block I/O latency (storage I/O, ie, disk I/O), and | |
973 | shows the distribution as a histogram. This can be useful to identify | |
974 | the characteristics of I/O latency, beyond the averages shown by | |
975 | iostat(1). For example, to study I/O latency outliers, or multi-modal | |
976 | distributions. | |
977 | ||
978 | # stap biolatency-nd.stp 1 1 | |
979 | ||
980 | ||
981 | lwtools/bitesize-nd.stp - Measure block I/O size distribution (non-debuginfo) | |
982 | keywords: io | |
983 | ||
984 | This uses the kernel tracepoint block_rq_insert to read the size of | |
985 | I/O. The output includes the name of the process or thread that was | |
986 | on-CPU when the I/O request was inserted on the issue queue. | |
987 | ||
988 | # stap bitesize-nd.stp -c "sleep 1" | |
989 | ||
990 | ||
991 | lwtools/execsnoop-nd.stp - Trace process exec() with command line argument details (non-debuginfo) | |
992 | keywords: io | |
993 | ||
994 | This can identify if CPU is consumed by short-lived processes, by | |
995 | tracing new process execution. It works by tracing exec() from the | |
996 | fork()->exec() sequence, which means it will not catch new processes | |
997 | that only fork(). It will also show every exec(), including those if | |
998 | a process re-execs. | |
999 | ||
1000 | # stap execsnoop-nd.stp -c "sleep 1" | |
1001 | ||
1002 | ||
1003 | lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo) | |
1004 | keywords: io filesystem | |
1005 | ||
1006 | This dynamically traces two common file system functions: | |
1007 | do_sync_read() and do_sync_write(), and reports a histogram | |
1008 | distribution of latency. Many, but not all, file systems and | |
1009 | workloads use these functions. Tracing their time provides one view | |
1010 | of suffered file system latency. | |
1011 | ||
1012 | # stap fslatency-nd.stp 1 1 | |
1013 | ||
1014 | ||
1015 | lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo) | |
1016 | keywords: io filesystem | |
1017 | ||
1018 | This dynamically traces two common file system functions: | |
1019 | do_sync_read() and do_sync_write(), and shows details of each call | |
1020 | that is slower than a threshold. Many, but not all, file systems and | |
1021 | workloads use these functions. Tracing their time provides one view | |
1022 | of suffered file system latency. | |
1023 | ||
1024 | # stap fsslower-nd.stp -c "sleep 1" | |
1025 | ||
1026 | ||
1027 | lwtools/killsnoop-nd.stp - Trace kill() signals showing process and signal details (non-debuginfo) | |
1028 | keywords: io | |
1029 | ||
1030 | This traces signals system-wide, including those sent by the kill(1) | |
1031 | command, and shows various details. | |
1032 | ||
1033 | # stap killsnoop-nd.stp -c "sleep 1" | |
1034 | ||
1035 | ||
1036 | lwtools/opensnoop-nd.stp - Trace open() syscalls showing filenames (non-debuginfo) | |
1037 | keywords: io | |
1038 | ||
1039 | This traces the open() syscall system-wide, to show which files are | |
1040 | being opened, and by who. | |
1041 | ||
1042 | # stap opensnoop-nd.stp -c "sleep 1" | |
1043 | ||
1044 | ||
1045 | lwtools/rwtime-nd.stp - Summarize read() and write() syscall latency (non-debuginfo) | |
1046 | keywords: io | |
1047 | ||
1048 | This traces read() and write() syscalls, producing a histogram | |
1049 | summary of their durations (aka latencies). | |
1050 | ||
1051 | # stap rwtime-nd.stp -c "sleep 1" | |
1052 | ||
1053 | ||
1054 | lwtools/syscallbypid-nd.stp - Count syscalls with process details (non-debuginfo) | |
1055 | keywords: io | |
1056 | ||
1057 | This traces syscalls system-wide, and produces a summary report | |
1058 | showing their counts by process ID, process name, and syscall types. | |
1059 | ||
1060 | # stap syscallbypid-nd.stp -c "sleep 1" | |
1061 | ||
1062 | ||
cf29c85e | 1063 | process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations |
09a98ae0 | 1064 | keywords: io scheduler backtrace |
210ff7bf | 1065 | |
f3c4da44 | 1066 | The script monitors the time that threads spend in waiting for IO |
cae71dd3 FCE |
1067 | operations (in "D" state) in the wait_for_completion function. If a |
1068 | thread spends over 10ms, its name and backtrace is printed, and later | |
1069 | so is the total delay. | |
210ff7bf | 1070 | |
1f115be7 | 1071 | # stap sleepingBeauties.stp -c "sleep 1" |
6a51196c | 1072 | |
210ff7bf | 1073 | |
cf29c85e | 1074 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
1075 | keywords: virtualization qemu kvm io |
1076 | ||
1077 | The qemu_io.stp script tallies the number of times each of the IO | |
1078 | port on the guest virtual machines is touched by a input or output | |
1079 | operation. When the script exits, it prints a count of the number of | |
1080 | times each IO port read and written. | |
1081 | ||
1f115be7 | 1082 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
1083 | |
1084 | ||
a89c2e98 FL |
1085 | = JSON = |
1086 | ||
1087 | network/net_xmit_json.stp - Tracks time between packet queue and transmit. | |
1088 | keywords: network statistics json | |
1089 | ||
1090 | This script tracks time between packet queue and transmit. The | |
1091 | information is provided to userspace via procfs in JSON format. | |
1092 | ||
1093 | # stap net_xmit_json.stp -c "sleep 1" | |
1094 | ||
1095 | ||
1096 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs | |
1097 | keywords: _best network traffic json | |
1098 | ||
1099 | The script watches all IPv4 network traffic on the system. The data | |
1100 | is output in JSON format and includes the number of packets sent | |
1101 | along source IP address / destination IP address pair encountered, | |
1102 | and the total number of bytes sent among the pair. | |
1103 | ||
1104 | # stap netfilter_summary_json.stp -c "sleep 1" | |
1105 | ||
1106 | ||
8c6bb289 WC |
1107 | = KVM = |
1108 | ||
cf29c85e | 1109 | virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons |
b9ae0bc4 | 1110 | keywords: _best virtualization kvm |
98ec0359 FCE |
1111 | |
1112 | The kvm_service_time.stp script tracks the statistics about the | |
1113 | amount of time that the processor left the guest virtual machine for | |
1114 | each exit reason (for example fixing up a page table or handling an | |
1115 | IO operation). When the script exits it prints out the number of | |
1116 | times each exit reason was encountered, the total duration of time it | |
1117 | left the guest VM, the minimum time, the average time, and the | |
1118 | maximum time in microseconds for that exit reason. On Linux 2.6.38 | |
1119 | and newer kernel the script can automatically determine whether it is | |
1120 | running on Intel or AMD processors. For older kernels with a | |
1121 | kernel.trace("kvm_exit") tracepoint that does not have the $isa | |
1122 | parameter you can explicitly state the kvm type with a "-G kvm=intel" | |
1123 | or "-G kvm=amd" on the command line. | |
1124 | ||
1f115be7 | 1125 | # stap kvm_service_time.stp -c "sleep 1" |
98ec0359 FCE |
1126 | |
1127 | ||
cf29c85e | 1128 | virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events |
8c6bb289 WC |
1129 | keywords: virtualization qemu kvm |
1130 | ||
1131 | The qemu_count.stp script tallies the number of times each of the | |
1132 | user-space qemu probepoints is encountered. When the script exits, it | |
1133 | prints a list of the number of times each user-space qemu probepoint | |
1134 | is encountered. | |
1135 | ||
1f115be7 | 1136 | # stap qemu_count.stp -c "sleep 1" |
8c6bb289 WC |
1137 | |
1138 | ||
cf29c85e | 1139 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
1140 | keywords: virtualization qemu kvm io |
1141 | ||
1142 | The qemu_io.stp script tallies the number of times each of the IO | |
1143 | port on the guest virtual machines is touched by a input or output | |
1144 | operation. When the script exits, it prints a count of the number of | |
1145 | times each IO port read and written. | |
1146 | ||
1f115be7 | 1147 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
1148 | |
1149 | ||
cf29c85e FCE |
1150 | = LIMITS = |
1151 | ||
1152 | memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits | |
1153 | keywords: memory limits | |
1154 | ||
1155 | The overcommit.stp script prints a line each time the kernel refuses | |
1156 | a memory allocation request from a process because of | |
1157 | /proc/sys/vm/overcommit* limits. | |
1158 | ||
1f115be7 FCE |
1159 | # stap overcommit.stp -c "sleep 1" |
1160 | ||
1161 | ||
1162 | process/rlimit_nofile.stp - Trace processes running out of file descriptors | |
1163 | keywords: limits | |
1164 | ||
1165 | This script watches processes being scheduled and which try to | |
1166 | allocate a file descriptor without luck. | |
1167 | ||
1168 | # stap rlimit_nofile.stp -c "sleep 1" | |
cf29c85e FCE |
1169 | |
1170 | ||
210ff7bf FCE |
1171 | = LOCKING = |
1172 | ||
cf29c85e | 1173 | locks/bkl.stp - Tracing Contention on Big Kernel Lock |
cf5023fb WC |
1174 | keywords: locking |
1175 | ||
1176 | The bkl.stp script can help determine whether the Big Kernel Lock | |
1177 | (BKL) is causing serialization on a multiprocessor system due to | |
a11f34b0 FCE |
1178 | excessive contention of the BKL. The bkl.stp script takes two |
1179 | arguments. The first one is optional, and used to enable backtraces, | |
1180 | and print them once a process has been holding the BKL for a user | |
1181 | specified number of nseconds is reached. The second option is | |
1182 | compulsory and is the number of processes waiting for the Big Kernel | |
cf5023fb WC |
1183 | Lock (BKL). When the number of processes waiting for the BKL is |
1184 | reached or exceeded, the script will print a time stamp, the number | |
1185 | of processes waiting for the BKL, the holder of the BKL, and the | |
a11f34b0 FCE |
1186 | amount of time the BKL was held. If backtraces are enabled, a |
1187 | backtrace will be printed as well. | |
cf5023fb | 1188 | |
1f115be7 | 1189 | # stap bkl.stp -c "sleep 1" 1 |
6a51196c | 1190 | |
cf5023fb | 1191 | |
cf29c85e | 1192 | locks/bkl_stats.stp - Per Process Statistics on Big Kernel Lock Use |
cf5023fb WC |
1193 | keywords: locking |
1194 | ||
1195 | The bkl_stats.stp script can indicate which processes have excessive | |
1196 | waits for the Big Kernel Lock (BKL) and which processes are taking | |
1197 | the BKL for long periods of time. The bkl_stats.stp script prints | |
1198 | lists of all the processes that require the BKL. Every five seconds | |
1199 | two tables are printed out. The first table lists the processes that | |
1200 | waited for the BKL followed by the number of times that the process | |
1201 | waited, the minimum time of the wait, the average and the maximum | |
1202 | time waited. The second table lists has similar information for the | |
f3c4da44 | 1203 | time spent in holding the lock for each of the processes. |
cf5023fb | 1204 | |
1f115be7 | 1205 | # stap bkl_stats.stp -c "sleep 1" |
6a51196c | 1206 | |
cf5023fb | 1207 | |
210ff7bf | 1208 | process/futexes.stp - System-Wide Futex Contention |
09a98ae0 | 1209 | keywords: syscall locking futex |
210ff7bf FCE |
1210 | |
1211 | The script watches the futex syscall on the system. On exit the | |
0fb74e68 | 1212 | futex's address, the number of contentions, and the average time for |
210ff7bf FCE |
1213 | each contention on the futex are printed from lowest pid number to |
1214 | highest. | |
1215 | ||
1f115be7 | 1216 | # stap futexes.stp -c "sleep 1" |
6a51196c | 1217 | |
210ff7bf | 1218 | |
0fb74e68 JS |
1219 | process/futexes2.stp - System-Wide Shared Futex Contention |
1220 | keywords: syscall locking futex | |
1221 | ||
1222 | The script watches just shared futex syscalls on the system. On exit | |
1223 | the futex's key, the number of contentions, and the average time for | |
1224 | each contention on the futex are printed from lowest pid number to | |
1225 | highest. | |
1226 | ||
1f115be7 | 1227 | # stap futexes2.stp -c "sleep 1" |
0fb74e68 JS |
1228 | |
1229 | ||
5a224324 FCE |
1230 | process/mutex-contention.stp - pthread mutex contention analysis |
1231 | keywords: locking | |
1232 | ||
1233 | Tracks pthread-mutex initialization/use and underlying futex | |
1234 | operations, to identify (with backtraces/symbol-names) the mutexes | |
f275b3f7 FCE |
1235 | suffering most contention. Invoke with "-d SHLIB --ldd", perhaps |
1236 | with -DMAXMAPENTRIES=NNNN for some large NNNN, if the arrays overflow | |
1237 | due to heavy activity. | |
5a224324 | 1238 | |
1f115be7 | 1239 | # stap mutex-contention.stp -c "sleep 1" |
5a224324 FCE |
1240 | |
1241 | ||
5cf50309 FCE |
1242 | process/semop-watch.stp - Watch semop(2)/semtimedop(2) operations |
1243 | keywords: process locking | |
1244 | ||
1245 | Prints a timed trace of semop(2)/semtimedop(2) syscalls | |
1246 | ||
1247 | # stap semop-watch.stp -c 'sleep 2' | |
1248 | ||
1249 | ||
22f971e8 WC |
1250 | = MEMORY = |
1251 | ||
cf29c85e FCE |
1252 | general/sizeof.stp - Print the Size of a C Type |
1253 | keywords: statistics memory | |
3cfd512b FCE |
1254 | |
1255 | This script prints the size of a type, based on dwarf debuginfo for | |
1256 | any kernel or userspace module, or trial-compilation of a given | |
1257 | header file name. | |
1258 | ||
6a51196c FCE |
1259 | # stap sizeof.stp FILE '</usr/include/stdio.h>' |
1260 | ||
3cfd512b | 1261 | |
c4abda95 SP |
1262 | memory/glibc-malloc.stp - Overview glibc malloc internal operations |
1263 | keywords: memory process | |
1264 | ||
1265 | This script reports on internal statistics of the glibc malloc | |
1266 | implementation, as used by a process restricted by stap -x/-c | |
1267 | ||
1268 | # stap glibc-malloc.stp -c 'stap --dump-functions' | |
1269 | ||
1270 | ||
cf29c85e | 1271 | memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware |
54d84647 WC |
1272 | keywords: memory watchpoint |
1273 | ||
1274 | The script will watch accesses to a single kernel address and prints | |
1275 | a traceback each time the address is accessed. This script needs to | |
1276 | be run as root to allow access to the breakpoint hardware. | |
1277 | ||
23063de1 | 1278 | # stap --all-modules hw_watch_addr.stp 0x`grep "vm_dirty_ratio" \ |
0a9d3bc8 | 1279 | /proc/kallsyms | awk '{print $1}'` -c "sleep 5" |
54d84647 WC |
1280 | |
1281 | ||
cf29c85e | 1282 | memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware |
54d84647 WC |
1283 | keywords: memory watchpoint |
1284 | ||
1285 | The script will watch accesses to the starting address of a single | |
1286 | kernel symbol and prints a traceback each time the symbol is | |
1287 | accessed. This script needs to be run as root to allow access to the | |
1288 | breakpoint hardware. | |
1289 | ||
23063de1 | 1290 | # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -c "sleep 5" |
54d84647 WC |
1291 | |
1292 | ||
cf29c85e | 1293 | memory/kmalloc-top - Show Paths to Kernel Malloc Invocations |
22f971e8 WC |
1294 | keywords: memory |
1295 | ||
1296 | The kmalloc-top perl program runs a small systemtap script to collect | |
1297 | stack traces for each call to the kmalloc function and counts the | |
1298 | time that each stack trace is observed. When kmalloc-top exits it | |
f3c4da44 MW |
1299 | prints out sorted list. The output can be filtered to print only the |
1300 | first N stack traces (-t), stack traces with a minimum counts (-m), | |
1301 | or exclude certain stack traces (-e). | |
22f971e8 | 1302 | |
1f115be7 | 1303 | # ./kmalloc-top -c "sleep 1" |
6a51196c | 1304 | |
22f971e8 | 1305 | |
f70a0bc5 FCE |
1306 | memory/last_100_frees.stp - Log recent free(3) calls. |
1307 | keywords: memory process backtrace | |
1308 | ||
1309 | This script reports on the last few free(3) libc calls done by | |
1310 | processes (possibly restricted by stap -x/-c), along with a userspace | |
1311 | backtrace at those moments. | |
1312 | ||
1313 | # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd | |
1314 | ||
1315 | ||
0dc23d1d WC |
1316 | memory/mmanonpage.stp - Track Virtual Memory System Actions on Anonymous Pages |
1317 | keywords: memory | |
1318 | ||
1319 | The mmanonpage.stp script uses the virtual memory tracepoints | |
1320 | available in some kernels to track the number of faults, user space | |
1321 | frees, page ins, copy on writes and unmaps for anonymous pages. When | |
1322 | the script is terminated the counts are printed for each process that | |
1323 | allocated pages while the script was running. This script displays | |
1324 | the anonymous page statistics for each process that ran while the | |
f3c4da44 | 1325 | script is active. It's useful in debugging leaks in the anonymous |
0dc23d1d WC |
1326 | regions of a process. |
1327 | ||
1f115be7 | 1328 | # stap mmanonpage.stp -c "sleep 1" |
6a51196c | 1329 | |
0dc23d1d WC |
1330 | |
1331 | memory/mmfilepage.stp - Track Virtual Memory System Actions on File Backed Pages | |
1332 | keywords: memory | |
1333 | ||
1334 | The mmfilepage.stp script uses the virtual memory tracepoints | |
1335 | available in some kernels to track the number of faults, copy on | |
1336 | writes mapping, and unmapping operations for file backed pages. When | |
1337 | the script is terminated the counts are printed for each process that | |
1338 | allocated pages while the script was running. The mmfilepage.stp | |
1339 | script is useful in debugging leaks in the mapped file regions of a | |
1340 | process. | |
1341 | ||
1f115be7 | 1342 | # stap mmfilepage.stp -c "sleep 1" |
6a51196c | 1343 | |
0dc23d1d WC |
1344 | |
1345 | memory/mmreclaim.stp - Track Virtual Memory System Page Reclamation | |
1346 | keywords: memory | |
1347 | ||
1348 | The mmreclaim.stp script uses the virtual memory tracepoints | |
ad7e33d7 | 1349 | available in some kernels to track page reclaim activity that |
f3c4da44 | 1350 | occurred while the script was running. It's useful in debugging |
ad7e33d7 | 1351 | performance problems that occur due to page reclamation. |
0dc23d1d | 1352 | |
1f115be7 | 1353 | # stap mmreclaim.stp -c "sleep 1" |
6a51196c | 1354 | |
0dc23d1d WC |
1355 | |
1356 | memory/mmwriteback.stp - Track Virtual Memory System Writing to Disk | |
1357 | keywords: memory | |
1358 | ||
1359 | The mmwriteback.stp script uses the virtual memory tracepoints | |
1360 | available in some kernels to report all of the file writebacks that | |
1361 | occur form kupdate, pdflush and kjournald while the script is | |
f3c4da44 MW |
1362 | running. It's useful in determining where writes are coming from on |
1363 | a supposedly idle system that is experiencing unexpected IO. | |
0dc23d1d | 1364 | |
1f115be7 | 1365 | # stap mmwriteback.stp -c "sleep 1" |
6a51196c | 1366 | |
0dc23d1d | 1367 | |
c728b7da WC |
1368 | memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes |
1369 | keywords: memory numa | |
1370 | ||
1371 | The numa_faults.stp script tracks the read and write pages faults for | |
1372 | each process. When the script exits it prints out the total read and | |
ad7e33d7 | 1373 | write pages faults for each process. The script also provide a break |
c728b7da WC |
1374 | down of page faults per node for each process. This script is useful |
1375 | for determining whether the program has good locality (page faults | |
1376 | limited to a single node) on a NUMA computer. | |
1377 | ||
1f115be7 | 1378 | # stap numa_faults.stp -c "sleep 1" |
6a51196c | 1379 | |
c728b7da | 1380 | |
cf29c85e FCE |
1381 | memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits |
1382 | keywords: memory limits | |
1ff1a65d FCE |
1383 | |
1384 | The overcommit.stp script prints a line each time the kernel refuses | |
1385 | a memory allocation request from a process because of | |
1386 | /proc/sys/vm/overcommit* limits. | |
1387 | ||
1f115be7 | 1388 | # stap overcommit.stp -c "sleep 1" |
6a51196c | 1389 | |
1ff1a65d | 1390 | |
413996e0 WC |
1391 | memory/pfaults.stp - Generate Log of Major and Minor Page Faults |
1392 | keywords: memory | |
1393 | ||
1394 | The pfaults.stp script generates a simple log for each major and | |
1395 | minor page fault that occurs on the system. Each line contains a | |
1396 | timestamp (in microseconds) when the page fault servicing was | |
1397 | completed, the pid of the process, the address of the page fault, the | |
1398 | type of access (read or write), the type of fault (major or minor), | |
1399 | and the elapsed time for page fault. This log can be examined to | |
ad7e33d7 | 1400 | determine where the page faults are occurring. |
413996e0 | 1401 | |
1f115be7 | 1402 | # stap pfaults.stp -c "sleep 1" |
6a51196c | 1403 | |
413996e0 | 1404 | |
cf29c85e FCE |
1405 | memory/vm.tracepoints.stp - Collect Slab Allocation Statistics |
1406 | keywords: memory slab statistics | |
42c55668 MW |
1407 | |
1408 | The script will probe all memory slab/slub allocations and collects | |
1409 | information about the size of the object (bytes requested) and | |
1410 | user-space process in execution. When run over a period of time, it | |
1411 | helps to correlate kernel-space memory consumption owing to | |
1412 | user-space processes. | |
1413 | ||
6a51196c FCE |
1414 | # stap vm.tracepoints.stp -c "sleep 10" |
1415 | ||
42c55668 | 1416 | |
14f0bb18 | 1417 | = MONITORING = |
b7f6cfc5 | 1418 | |
cf29c85e | 1419 | general/varwatch.stp - Watch a Variable Changing Value in a Thread |
b9ae0bc4 | 1420 | keywords: _best monitoring |
b7f6cfc5 | 1421 | |
14f0bb18 FCE |
1422 | This script places a set of probes (specified by $1), each of which |
1423 | monitors the state of some context $variable expression (specified by | |
1424 | $2). Whenever the value changes, with respect to the active thread, | |
1425 | the event is traced. | |
b7f6cfc5 | 1426 | |
6a51196c | 1427 | # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \ |
1f115be7 | 1428 | '$$vars' -c "sleep 1" |
6a51196c | 1429 | |
1beb5089 | 1430 | |
75ed3b3c FCE |
1431 | general/whythefail.stp - Why did the function fail? |
1432 | keywords: speculation monitoring function trace _best | |
1433 | ||
1434 | The whythefail.stp script prints a statement-execution trace for a | |
1435 | given function, but only for those runs of the function that ended up | |
1436 | with a (configurable) post-return condition. | |
1437 | ||
1438 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
1439 | /root/no-such-file || true' | |
1440 | ||
1441 | ||
cf29c85e | 1442 | io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices |
74e5c5cf | 1443 | keywords: io monitoring disk |
1beb5089 FCE |
1444 | |
1445 | The mbrwatch.stp script reports any attempted reads/writes of the | |
1446 | first few sectors of a raw block device. | |
1447 | ||
6a51196c | 1448 | # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \ |
199197b8 FCE |
1449 | /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \ |
1450 | head -1`" | |
6a51196c | 1451 | |
1beb5089 | 1452 | |
cf29c85e | 1453 | io/ttyspy.stp - Monitor TTY Typing |
1ca36a99 | 1454 | keywords: io tty monitoring guru |
14f0bb18 FCE |
1455 | |
1456 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
1457 | activity on the system, printing a scrolling record of recent | |
1458 | keystrokes, on a per-tty basis. | |
1459 | ||
6a51196c FCE |
1460 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" |
1461 | ||
14f0bb18 | 1462 | |
7080b0c3 | 1463 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
1464 | keywords: monitoring security audit backtrace |
1465 | ||
7080b0c3 FCE |
1466 | Attaches to the kernel audit-log paths (also used by libaudit), and |
1467 | log every record being sent, along with a user-space backtrace of the | |
1468 | process that caused it. | |
a8a7b9f4 | 1469 | |
54a385da | 1470 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
1471 | |
1472 | ||
677b471e JL |
1473 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
1474 | keywords: process monitoring syscall tracepoint | |
1475 | ||
1476 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
1477 | exit(), init_module(), and delete_module(). Event-specific details | |
1478 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
1479 | script does not require debuginfo. | |
1480 | ||
1f115be7 | 1481 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
1482 | |
1483 | ||
cf29c85e FCE |
1484 | = NANOSLEEP = |
1485 | ||
1486 | process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls | |
1487 | keywords: syscall nanosleep | |
1488 | ||
1489 | The script watches each nanosleep syscall on the system. At the end | |
1490 | of each nanosleep syscall the script prints out a line with a | |
1491 | timestamp in microseconds, the pid, the executable name in | |
1492 | parentheses, the "nanosleep:" key, and the duration of the sleep in | |
1493 | microseconds. | |
1494 | ||
1f115be7 | 1495 | # stap sleeptime.stp -c "sleep 1" |
cf29c85e FCE |
1496 | |
1497 | ||
0e81b7f5 FCE |
1498 | = NET = |
1499 | ||
1500 | lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo) | |
1501 | keywords: net socket | |
1502 | ||
1503 | This traces socket duration from the accept() syscall to close(), and | |
1504 | provides details on the lifespan of these passive connections, | |
1505 | showing the distribution as a histogram. | |
1506 | ||
1507 | # stap accept2close-nd.stp -c "sleep 1" | |
1508 | ||
1509 | ||
210ff7bf FCE |
1510 | = NETWORK = |
1511 | ||
cf29c85e | 1512 | network/autofs4.stp - Watch autofs4 Operations |
8b88e771 FCE |
1513 | keywords: network autofs nfs |
1514 | ||
1515 | Trace key autofs4 operations such as mounting or unmounting remote | |
1516 | filesystems. | |
1517 | ||
6a51196c FCE |
1518 | # stap autofs4.stp -c "sleep 1" |
1519 | ||
8b88e771 | 1520 | |
9e749e50 JS |
1521 | network/connect_stat.stp - Show Process Ancestry for IP Connections |
1522 | keywords: network socket process | |
1523 | ||
1524 | The connect_stat.stp script prints a task's entire ancestry (parent | |
1525 | process name/uid/gid) whenever it attempts an outgoing socket | |
1526 | connection to a given IP address. | |
1527 | ||
1f115be7 | 1528 | # stap connect_stat.stp 127.0.0.1 -c "sleep 1" |
9e749e50 JS |
1529 | |
1530 | ||
cf29c85e FCE |
1531 | network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel |
1532 | keywords: network tracepoint socket | |
0e4901b0 WC |
1533 | |
1534 | Every five seconds the dropwatch.stp script lists the number of | |
1535 | socket buffers freed at locations in the kernel. | |
1536 | ||
1f115be7 | 1537 | # stap dropwatch.stp -c "sleep 1" |
6a51196c | 1538 | |
0e4901b0 | 1539 | |
a89c2e98 FL |
1540 | network/net_xmit_json.stp - Tracks time between packet queue and transmit. |
1541 | keywords: network statistics json | |
1542 | ||
1543 | This script tracks time between packet queue and transmit. The | |
1544 | information is provided to userspace via procfs in JSON format. | |
1545 | ||
1546 | # stap net_xmit_json.stp -c "sleep 1" | |
1547 | ||
1548 | ||
53e5699f JS |
1549 | network/netdev.stp - Trace Activity on Network Devices |
1550 | keywords: network device traffic | |
1551 | ||
1552 | The netdev.stp script traces configuration and transmit/receive | |
1553 | activity on network devices. | |
1554 | ||
1f115be7 | 1555 | # stap netdev.stp -c "sleep 1" |
6a51196c | 1556 | |
53e5699f | 1557 | |
f1c6c6ed | 1558 | network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool |
1ca36a99 | 1559 | keywords: network packets guru |
f1c6c6ed MW |
1560 | |
1561 | The script drops the specified number of packets of the specified | |
1562 | protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, | |
1563 | all incoming packets are dropped. The number of packets to drop can | |
1564 | be specified with a positive integer. A value of 0 indicates that | |
1565 | packets should be dropped until the user manually exits. | |
1566 | ||
1567 | # stap -g netfilter_drop.stp TCP 1 -c "sleep 2" | |
1568 | ||
1569 | ||
57db0e6f | 1570 | network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs |
b9ae0bc4 | 1571 | keywords: _best network traffic |
57db0e6f MW |
1572 | |
1573 | The script watches all IPv4 network traffic on the system. On exit | |
1574 | the script prints a list showing the number of packets sent along | |
1575 | source IP address / destination IP address pair encountered, and the | |
1576 | total number of bytes sent among the pair. The list is ordered from | |
1577 | greatest to least number of packets seen among the source/destination | |
1578 | pairs. | |
1579 | ||
1f115be7 | 1580 | # stap netfilter_summary.stp -c "sleep 1" |
57db0e6f MW |
1581 | |
1582 | ||
a89c2e98 FL |
1583 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs |
1584 | keywords: _best network traffic json | |
1585 | ||
1586 | The script watches all IPv4 network traffic on the system. The data | |
1587 | is output in JSON format and includes the number of packets sent | |
1588 | along source IP address / destination IP address pair encountered, | |
1589 | and the total number of bytes sent among the pair. | |
1590 | ||
1591 | # stap netfilter_summary_json.stp -c "sleep 1" | |
1592 | ||
1593 | ||
210ff7bf | 1594 | network/nettop.stp - Periodic Listing of Processes Using Network Interfaces |
74e5c5cf | 1595 | keywords: network traffic |
210ff7bf FCE |
1596 | |
1597 | Every five seconds the nettop.stp script prints out a list of | |
1598 | processed (PID and command) with the number of packets sent/received | |
1599 | and the amount of data sent/received by the process during that | |
1600 | interval. | |
1601 | ||
1f115be7 | 1602 | # stap nettop.stp -c "sleep 1" |
6a51196c | 1603 | |
210ff7bf | 1604 | |
905728a0 | 1605 | network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space |
cf29c85e | 1606 | keywords: network tcp process |
905728a0 WC |
1607 | |
1608 | The sk_stream-wait_memory.stp prints a time stamp, executable, and | |
1609 | pid each time a process blocks due to the send buffer being full. A | |
1610 | similar entry is printed each time a process continues because there | |
1611 | is room in the buffer. | |
1612 | ||
1f115be7 | 1613 | # stap sk_stream_wait_memory.stp -c "sleep 1" |
6a51196c | 1614 | |
905728a0 | 1615 | |
cf29c85e | 1616 | network/socket-trace.stp - Trace Functions Called in Network Socket Code |
09a98ae0 | 1617 | keywords: network socket |
210ff7bf | 1618 | |
f3c4da44 MW |
1619 | The script instruments each of the functions in the Linux kernel's |
1620 | net/socket.c file. The script prints out trace data. The first | |
1621 | element of a line is time delta in microseconds from the previous | |
1622 | entry. This is followed by the command name and the PID. The "->" and | |
1623 | "<-" indicates function entry and function exit, respectively. The | |
1624 | last element of the line is the function name. | |
210ff7bf | 1625 | |
1f115be7 | 1626 | # stap socket-trace.stp -c "sleep 1" |
6a51196c | 1627 | |
210ff7bf | 1628 | |
9141644d WC |
1629 | network/socktop - Periodically Summarize Socket Activity on the System |
1630 | keywords: network socket | |
1631 | ||
1632 | The socktop script periodically prints out a list of the processes | |
1633 | with the highest socket activity. Command line options for the | |
1634 | script allow filtering to focus on particular types of sockets. The | |
1635 | "-h" option lists socktop script's filtering options. | |
1636 | ||
6a51196c FCE |
1637 | # ./socktop -c 1 |
1638 | ||
9141644d | 1639 | |
a89c2e98 FL |
1640 | network/stp_dump.stp - Dump of STP packets |
1641 | keywords: network traffic | |
1642 | ||
1643 | The stp_dump.stp prints out the packet contents. Each block contains | |
1644 | the STP protocol ID, version ID, flags, root and bridge MAC | |
1645 | addresses, and various times. | |
1646 | ||
9e6206fd | 1647 | # stap stp_dump.stp -c "sleep 5" |
a89c2e98 FL |
1648 | |
1649 | ||
492d227f WC |
1650 | network/tcp_connections.stp - Track Creation of Incoming TCP Connections |
1651 | keywords: network tcp socket | |
1652 | ||
1653 | The tcp_connections.stp script prints information for each new | |
1654 | incoming TCP connection accepted by the computer. The information | |
1655 | includes the UID, the command accepting the connection, the PID of | |
1656 | the command, the port the connection is on, and the IP address of the | |
1657 | originator of the request. | |
1658 | ||
1f115be7 | 1659 | # stap tcp_connections.stp -c "sleep 1" |
6a51196c | 1660 | |
492d227f | 1661 | |
cf29c85e | 1662 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 |
1ca36a99 | 1663 | keywords: network tcp socket guru |
65bdbf09 FCE |
1664 | |
1665 | Run the tcp_init_cwnd.stp script in the background to override a | |
1666 | kernel's default tcp cwnd value to 10, which has been found to | |
1667 | improve latency for web server type workloads. The script prints a | |
1668 | count of cwnd value changes when it is stopped. | |
1669 | ||
1f115be7 | 1670 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
65bdbf09 FCE |
1671 | |
1672 | ||
cf29c85e | 1673 | network/tcp_trace.stp - TCP Connection Tracing Utility |
2e251678 DW |
1674 | keywords: network trace |
1675 | ||
cf29c85e | 1676 | This scripts traces a given TCP connection based on the filter |
2e251678 DW |
1677 | parameters given by the user. The indexing is done by the 4 tuples |
1678 | local address, remote address, local port, remote port. | |
1679 | ||
6a51196c FCE |
1680 | # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1 |
1681 | ||
2e251678 | 1682 | |
5cf50309 | 1683 | network/tcpdumplike.stp - Dump of Received UDP/TCP Packets |
7a51212c WC |
1684 | keywords: network traffic |
1685 | ||
5cf50309 FCE |
1686 | The tcpdumplike.stp prints out a line for each TCP & UDP packet |
1687 | received. Each line includes the source and destination IP addresses, | |
1688 | the source and destination ports, and flags. | |
7a51212c | 1689 | |
1f115be7 | 1690 | # stap tcpdumplike.stp -c "sleep 1" |
6a51196c | 1691 | |
7a51212c | 1692 | |
cf29c85e | 1693 | network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets |
4bb6522c WC |
1694 | keywords: network statistics |
1695 | ||
cf29c85e FCE |
1696 | The tcpipstat script collects and displays network statistics related |
1697 | to individual TCP sockets or groups of sockets. The statistics that | |
1698 | are collected are simular to that of the command netstat -s, only | |
1699 | sorted and grouped by individual sockets. | |
4bb6522c | 1700 | |
6a51196c FCE |
1701 | # stap tcpipstat.stp timeout=1 |
1702 | ||
4bb6522c | 1703 | |
8b88e771 FCE |
1704 | = NFS = |
1705 | ||
cf29c85e | 1706 | network/autofs4.stp - Watch autofs4 Operations |
8b88e771 FCE |
1707 | keywords: network autofs nfs |
1708 | ||
1709 | Trace key autofs4 operations such as mounting or unmounting remote | |
1710 | filesystems. | |
1711 | ||
6a51196c FCE |
1712 | # stap autofs4.stp -c "sleep 1" |
1713 | ||
8b88e771 | 1714 | |
d12c1a86 FCE |
1715 | network/nfsd-recent.stp - Keep track of NFS server statistics |
1716 | keywords: nfs statistics | |
1717 | ||
1718 | This script tracks all nfsd server operations by client_ip address, | |
1719 | and periodically lists those clients that have made recent requests. | |
1720 | It's a way of finding out which nfs clients might be considered still | |
1721 | connected. | |
1722 | ||
1f115be7 | 1723 | # stap nfsd-recent.stp -c "sleep 1" |
d12c1a86 FCE |
1724 | |
1725 | ||
cf29c85e | 1726 | network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server |
b7edd338 WC |
1727 | keywords: nfs disk |
1728 | ||
1729 | The nfsd_unlink.stp script lists the ip address and file name each | |
1730 | time time a file is being removed or unlinked by the nfsd. This | |
1731 | script is run on the nfs server. | |
1732 | ||
1f115be7 | 1733 | # stap nfsd_unlink.stp -c "sleep 1" |
b7edd338 WC |
1734 | |
1735 | ||
5882f170 DS |
1736 | network/nfsdtop.stp - Keep track of NFS server statistics |
1737 | keywords: nfs statistics | |
1738 | ||
1739 | The nfsdtop.stp script gathers and displays NFS lookups, | |
1740 | ||
3b33d8c4 | 1741 | # stap nfsdtop.stp -c "sleep 1" |
5882f170 DS |
1742 | |
1743 | ||
c728b7da WC |
1744 | = NUMA = |
1745 | ||
1746 | memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes | |
1747 | keywords: memory numa | |
1748 | ||
1749 | The numa_faults.stp script tracks the read and write pages faults for | |
1750 | each process. When the script exits it prints out the total read and | |
ad7e33d7 | 1751 | write pages faults for each process. The script also provide a break |
c728b7da WC |
1752 | down of page faults per node for each process. This script is useful |
1753 | for determining whether the program has good locality (page faults | |
1754 | limited to a single node) on a NUMA computer. | |
1755 | ||
1f115be7 | 1756 | # stap numa_faults.stp -c "sleep 1" |
6a51196c | 1757 | |
c728b7da | 1758 | |
f1c6c6ed MW |
1759 | = PACKETS = |
1760 | ||
1761 | network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool | |
1ca36a99 | 1762 | keywords: network packets guru |
f1c6c6ed MW |
1763 | |
1764 | The script drops the specified number of packets of the specified | |
1765 | protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, | |
1766 | all incoming packets are dropped. The number of packets to drop can | |
1767 | be specified with a positive integer. A value of 0 indicates that | |
1768 | packets should be dropped until the user manually exits. | |
1769 | ||
1770 | # stap -g netfilter_drop.stp TCP 1 -c "sleep 2" | |
1771 | ||
1772 | ||
cf29c85e | 1773 | = PROCESS = |
db5bfb14 | 1774 | |
cf29c85e | 1775 | general/eventcount.stp - Count Specified Events |
b9ae0bc4 | 1776 | keywords: _best statistics thread process |
db5bfb14 | 1777 | |
875e1b27 FCE |
1778 | The script periodically prints a count of specified events and their |
1779 | related tid's over the course of execution. Numerous configuration | |
1780 | options exist to control filtering / reporting, see the script | |
1781 | source. | |
db5bfb14 | 1782 | |
1f115be7 | 1783 | # stap eventcount.stp syscall.* -c 'sleep 1' |
db5bfb14 FCE |
1784 | |
1785 | ||
c4abda95 SP |
1786 | memory/glibc-malloc.stp - Overview glibc malloc internal operations |
1787 | keywords: memory process | |
1788 | ||
1789 | This script reports on internal statistics of the glibc malloc | |
1790 | implementation, as used by a process restricted by stap -x/-c | |
1791 | ||
1792 | # stap glibc-malloc.stp -c 'stap --dump-functions' | |
1793 | ||
1794 | ||
f70a0bc5 FCE |
1795 | memory/last_100_frees.stp - Log recent free(3) calls. |
1796 | keywords: memory process backtrace | |
1797 | ||
1798 | This script reports on the last few free(3) libc calls done by | |
1799 | processes (possibly restricted by stap -x/-c), along with a userspace | |
1800 | backtrace at those moments. | |
1801 | ||
1802 | # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd | |
1803 | ||
1804 | ||
9e749e50 JS |
1805 | network/connect_stat.stp - Show Process Ancestry for IP Connections |
1806 | keywords: network socket process | |
1807 | ||
1808 | The connect_stat.stp script prints a task's entire ancestry (parent | |
1809 | process name/uid/gid) whenever it attempts an outgoing socket | |
1810 | connection to a given IP address. | |
1811 | ||
1f115be7 | 1812 | # stap connect_stat.stp 127.0.0.1 -c "sleep 1" |
9e749e50 JS |
1813 | |
1814 | ||
cf29c85e FCE |
1815 | network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space |
1816 | keywords: network tcp process | |
1ff1a65d | 1817 | |
cf29c85e FCE |
1818 | The sk_stream-wait_memory.stp prints a time stamp, executable, and |
1819 | pid each time a process blocks due to the send buffer being full. A | |
1820 | similar entry is printed each time a process continues because there | |
1821 | is room in the buffer. | |
1ff1a65d | 1822 | |
1f115be7 | 1823 | # stap sk_stream_wait_memory.stp -c "sleep 1" |
6a51196c | 1824 | |
1ff1a65d | 1825 | |
cf29c85e | 1826 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 1827 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
1828 | |
1829 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
1830 | to determine which processes and interrupts are competing with the | |
1831 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
1832 | options to focus on a specific task. The script output the number of | |
1833 | times the task migrates between processors, histograms showing the | |
1834 | length of time on and off processor, lists of processes running while | |
1835 | the task is off the processor, and the interrupts that occurred while | |
1836 | the task was running. | |
1837 | ||
1f115be7 | 1838 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 1839 | |
e01bc08e | 1840 | |
cf29c85e | 1841 | process/errsnoop.stp - Tabulate System Call Errors |
cc20d853 ET |
1842 | keywords: process syscall |
1843 | ||
cf29c85e FCE |
1844 | Prints a periodic tabular report about failing system calls, by |
1845 | process and by syscall failure. The first optional argument | |
1846 | specifies the reporting interval (in seconds, default 5); the second | |
1847 | optional argument gives a screen height (number of lines in the | |
1848 | report, default 20). | |
cc20d853 | 1849 | |
6a51196c FCE |
1850 | # stap errsnoop.stp 1 10 -c "sleep 1" |
1851 | ||
cc20d853 | 1852 | |
e6b653c8 WC |
1853 | process/forktracker.stp - Trace Creation of Processes |
1854 | keywords: process scheduler | |
1855 | ||
1856 | The forktracker.stp script prints out a time-stamped entry showing | |
f3c4da44 | 1857 | each fork and exec operation on the machine. This can be useful to |
e6b653c8 WC |
1858 | determine what process is creating a flurry of short-lived processes. |
1859 | ||
1f115be7 | 1860 | # stap forktracker.stp -c "sleep 1" |
6a51196c | 1861 | |
e6b653c8 | 1862 | |
abb4cfbb FCE |
1863 | process/ltrace.stp - uprobes-based ltrace |
1864 | keywords: process | |
1865 | ||
1866 | The ltrace.stp script lists calls that the designated process makes | |
1867 | through PLTs (procedure linkage tables), generally into shared | |
1868 | libraries. | |
1869 | ||
1870 | # stap ltrace.stp -c ls || echo PR14738 | |
1871 | ||
1872 | ||
cf29c85e | 1873 | process/noptrace.stp - Disable ptrace from Hierarchies of Processes |
1ca36a99 | 1874 | keywords: process security guru |
5481d677 | 1875 | |
cf29c85e FCE |
1876 | Blocks ptrace(2) attempts from processes identified by stap -c/-x, as |
1877 | also specifiable from /proc/systemtap/stap_XXX/ control files. | |
1878 | Processes may be added or removed from the blocked list. | |
5481d677 | 1879 | |
15ecb1fe | 1880 | # stap -g noptrace.stp -c 'strace ls || true' |
6a51196c | 1881 | |
5481d677 | 1882 | |
cf29c85e FCE |
1883 | process/pfiles.stp - Print Process File Descriptors |
1884 | keywords: process file | |
ecf33ff4 ET |
1885 | |
1886 | Run pfiles.stp to produce a human-readable summary of all open file | |
1887 | descriptors of a given process. Specify the process-id as -x PID for | |
1888 | fastest performance. | |
1889 | ||
6a51196c FCE |
1890 | # stap -g pfiles.stp -x $$ |
1891 | ||
ecf33ff4 | 1892 | |
cf29c85e | 1893 | process/plimit.stp - Print Resource Limits of Process |
5b8642a2 FCE |
1894 | keywords: process |
1895 | ||
1896 | The script prints a variety of resource limits for a given pid, like | |
1897 | /proc/$$/limits on recent kernels. | |
1898 | ||
6a51196c FCE |
1899 | # stap -g plimit.stp $$ |
1900 | ||
5b8642a2 | 1901 | |
677b471e JL |
1902 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
1903 | keywords: process monitoring syscall tracepoint | |
1904 | ||
1905 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
1906 | exit(), init_module(), and delete_module(). Event-specific details | |
1907 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
1908 | script does not require debuginfo. | |
1909 | ||
1f115be7 | 1910 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
1911 | |
1912 | ||
944d282f AJ |
1913 | process/proctop.stp - Periodically Print Process Information With History |
1914 | keywords: process scheduler _best | |
1915 | ||
1916 | Every 5 seconds, print out a list of 25 processes that took the most | |
1917 | system time with information about the processes. Includes | |
1918 | information on processes that may have exited while the script was | |
1919 | running. The script contains configuration options listed in the | |
1920 | script source. | |
1921 | ||
1922 | # stap proctop.stp -c "sleep 1" | |
1923 | ||
1924 | ||
f1c6c6ed | 1925 | process/psig.stp - Print Process File Descriptors |
1ca36a99 | 1926 | keywords: process signals |
f1c6c6ed MW |
1927 | |
1928 | Run psig.stp to produce a human-readable summary of the signal | |
1929 | handling configuration of a given process. Specify the process-id as | |
1930 | -x PID for fastest performance. | |
1931 | ||
1932 | # stap -DMAXACTION=10000 -g psig.stp -x $$ | |
1933 | ||
1934 | ||
04c36a84 MW |
1935 | process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands |
1936 | keywords: process backtrace | |
1937 | ||
1938 | The pstrace_exec.stp script watches each exec operation. If the exec | |
1939 | contains a substring that matches the script's command-line argument, | |
1940 | it prints out that process and all of its ancestors. | |
1941 | ||
1f115be7 | 1942 | # stap pstrace_exec.stp -c "sleep 1" bash |
04c36a84 MW |
1943 | |
1944 | ||
9e749e50 JS |
1945 | process/pstree.stp - Generates a process diagram in DOT form. |
1946 | keywords: process diagram | |
1947 | ||
1948 | The pstree.stp script generates a process diagram in DOT form. For | |
1949 | instance, it may be useful on a 'make' command to see all the | |
1950 | processes that are started. | |
1951 | ||
1f115be7 | 1952 | # stap pstree.stp -c "sleep 1" |
9e749e50 JS |
1953 | |
1954 | ||
cf29c85e | 1955 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
1956 | keywords: process scheduler time tracepoint |
1957 | ||
1958 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
1959 | amount of time that each process spends in running, sleeping, |
1960 | queuing, and waiting for io. On exit the script prints out the | |
1961 | accumulated time for each state of processes observed. Optionally, | |
1962 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 1963 | specific PID and its children. |
34029cd3 | 1964 | |
1f115be7 | 1965 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 1966 | |
34029cd3 | 1967 | |
5cf50309 FCE |
1968 | process/semop-watch.stp - Watch semop(2)/semtimedop(2) operations |
1969 | keywords: process locking | |
1970 | ||
1971 | Prints a timed trace of semop(2)/semtimedop(2) syscalls | |
1972 | ||
1973 | # stap semop-watch.stp -c 'sleep 2' | |
1974 | ||
1975 | ||
b9ae0bc4 FCE |
1976 | process/spawn_seeker.stp - Track Creation of Processes by process and execname |
1977 | keywords: process scheduler | |
1978 | ||
1979 | The spawn_seeker.stp script every minute (and on exit) prints out the | |
1980 | local time and sorted lists of which processes and executables | |
1981 | spawned tasks during the previous minute. This can be useful to | |
1982 | determine what process is creating a flurry of short-lived processes. | |
1983 | When a process exits its count of tasks created is added to its | |
1984 | parent's count to better account for the indirect task creation by | |
1985 | children processes. For more detailed examination of task creation | |
1986 | consider using forktracker.stp. | |
1987 | ||
1988 | # stap spawn_seeker.stp -c "sleep 1" | |
1989 | ||
1990 | ||
5cf50309 | 1991 | process/strace.stp - Trace system calls |
b9ae0bc4 | 1992 | keywords: _best process syscall |
5cf50309 FCE |
1993 | |
1994 | The script loosely emulates strace, when applied to individual | |
1995 | processes or hierarchies (via -c/-x), or the entire system (without | |
1996 | -c/-x). A few output configuration parameters may be set with -G. | |
1997 | ||
1f115be7 | 1998 | # stap strace.stp -c "sleep 1" |
5cf50309 FCE |
1999 | |
2000 | ||
c29c6f83 | 2001 | process/thread-business.stp - monitor syscall history |
b9ae0bc4 | 2002 | keywords: _best process syscall |
c29c6f83 FCE |
2003 | |
2004 | Prints a periodic tabular report about the counts of syscall activity | |
2005 | of all threads on the system, along with a textual | |
2006 | recent-syscall-history for each | |
2007 | ||
2008 | # stap thread-business.stp -c "sleep 10" | |
2009 | ||
2010 | ||
74e5c5cf FCE |
2011 | process/wait4time.stp - Trace Time Spent in wait4 Syscalls |
2012 | keywords: syscall process | |
2013 | ||
2014 | The script watches each wait4 syscall on the system. At the end of | |
2015 | each wait4 syscall the script prints out a line with a timestamp in | |
2016 | microseconds, the pid, the executable name in parentheses, the | |
2017 | "wait4:" key, the duration of the wait and the PID that the wait4 was | |
2018 | waiting for. If the waited for PID is not specified , it is "-1". | |
2019 | ||
1f115be7 | 2020 | # stap wait4time.stp -c "sleep 1" |
74e5c5cf FCE |
2021 | |
2022 | ||
210ff7bf FCE |
2023 | = PROFILING = |
2024 | ||
a4f3198f WC |
2025 | io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device |
2026 | keywords: io profiling | |
2027 | ||
2028 | The iodevstats.stp script measures the amount of data successfully | |
2029 | read and written by all the executables for each io device on the | |
2030 | system. The output is sorted from greatest sum of bytes read and | |
2031 | written to a device by an executable to the least. The output | |
2032 | contains device major/minor number, the count of operations (reads | |
2033 | and writes), the totals and averages for the number of bytes read and | |
2034 | written. | |
2035 | ||
1f115be7 | 2036 | # stap iodevstats.stp -c "sleep 1" |
6a51196c | 2037 | |
a4f3198f | 2038 | |
cf29c85e | 2039 | io/iostat-scsi.stp - IO Statistics for SCSI Devices |
54ff5e0c FCE |
2040 | keywords: io profiling scsi |
2041 | ||
2042 | The iostat-scsi.stp script provides a breakdown of the number of blks | |
ad7e33d7 | 2043 | read and written on the machine's various SCSI devices. The script |
54ff5e0c FCE |
2044 | takes one argument which is the number of seconds between reports. |
2045 | ||
1f115be7 | 2046 | # stap -g iostat-scsi.stp 1 -c "sleep 1" |
6a51196c | 2047 | |
54ff5e0c | 2048 | |
3e4444ed WC |
2049 | io/iostats.stp - List Executables Reading and Writing the Most Data |
2050 | keywords: io profiling | |
2051 | ||
2052 | The iostat.stp script measures the amount of data successfully read | |
2053 | and written by all the executables on the system. The output is | |
2054 | sorted from most greatest sum of bytes read and written by an | |
2055 | executable to the least. The output contains the count of operations | |
2056 | (opens, reads, and writes), the totals and averages for the number of | |
2057 | bytes read and written. | |
2058 | ||
1f115be7 | 2059 | # stap iostats.stp -c "sleep 1" |
6a51196c | 2060 | |
3e4444ed | 2061 | |
cf29c85e FCE |
2062 | io/iotime.stp - Trace Time Spent in Read and Write for Files |
2063 | keywords: profiling syscall io file | |
2064 | ||
2065 | The script watches each open, close, read, and write syscalls on the | |
2066 | system. For each file the scripts observes opened it accumulates the | |
2067 | amount of wall clock time spent in read and write operations and the | |
2068 | number of bytes read and written. When a file is closed the script | |
2069 | prints out a pair of lines for the file. Both lines begin with a | |
2070 | timestamp in microseconds, the PID number, and the executable name in | |
2071 | parentheses. The first line with the "access" keyword lists the file | |
2072 | name, the attempted number of bytes for the read and write | |
2073 | operations. The second line with the "iotime" keyword list the file | |
2074 | name and the number of microseconds accumulated in the read and write | |
2075 | syscalls. | |
2076 | ||
1f115be7 | 2077 | # stap iotime.stp -c "sleep 1" |
cf29c85e FCE |
2078 | |
2079 | ||
111dd9ac WC |
2080 | io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process |
2081 | keywords: io profiling | |
2082 | ||
2083 | The nfs_func_users.stp script counts the uses of NFS functions in the | |
2084 | kernel on a per process bases. The output is sorted from the process | |
2085 | with the greatest number of NFS functions called to the least. The | |
2086 | output contains the executable name, the process number, and the | |
2087 | total number of NFS functions called by the process. | |
2088 | ||
1f115be7 | 2089 | # stap nfs_func_users.stp -c "sleep 1" |
6a51196c | 2090 | |
111dd9ac | 2091 | |
b96bcce8 WC |
2092 | profiling/errno.stp - Show Which Processes and System Calls Return Errors Most Frequently |
2093 | keywords: profiling | |
2094 | ||
2095 | On exit the errno.stp script provides a sorted list showing which | |
2096 | combination of PID, system call, and error occur most frequently. | |
2097 | ||
1f115be7 | 2098 | # stap errno.stp -c "sleep 1" |
6a51196c | 2099 | |
b96bcce8 | 2100 | |
0a9d3bc8 FCE |
2101 | profiling/fileline-profile.stp - Profile Kernel/User Functions |
2102 | keywords: profiling | |
2103 | ||
2104 | The fileline-profile.stp script ends by printing out a sorted list of | |
2105 | the top twenty kernel and/or user processes providing file:line | |
2106 | information, if available, from the samples addresses gathered over | |
2107 | the time period the script is run. Use any of --ldd, --all-modules, | |
2108 | -d MODULE, -d /PATH/TO/EXEC to add more symbolic info. To include the | |
2109 | symbol name in the output, specify guru mode (-g) and add | |
2110 | symbolname="yes" to the stap command. | |
2111 | ||
2112 | # stap fileline-profile.stp -c "sleep 6" --all-modules --ldd | |
2113 | ||
2114 | ||
cf29c85e | 2115 | profiling/fntimes.stp - Show Functions Taking Longer Than Usual |
b9ae0bc4 | 2116 | keywords: _best profiling |
f503d3c0 FCE |
2117 | |
2118 | The fntimes.stp script monitors the execution time history of a given | |
2119 | function family (assumed non-recursive). Each time (beyond a warmup | |
2120 | interval) is then compared to the historical maximum. If it exceeds | |
2121 | a certain threshold (250%), a message is printed. | |
2122 | ||
6a51196c FCE |
2123 | # stap fntimes.stp 'kernel.function("sys_*")' -c "sleep 7" |
2124 | ||
f503d3c0 | 2125 | |
cf29c85e FCE |
2126 | profiling/functioncallcount.stp - Count Times Functions Are Called |
2127 | keywords: profiling function | |
210ff7bf FCE |
2128 | |
2129 | The functioncallcount.stp script takes one argument, a list of | |
2130 | functions to probe. The script will run and count the number of times | |
2131 | that each of the functions on the list is called. On exit the script | |
2132 | will print a sorted list from most frequently to least frequently | |
2133 | called function. | |
2134 | ||
ca5d5a88 | 2135 | # stap -w functioncallcount.stp "*@mm/*.c" -c "sleep 1" |
a4072787 WC |
2136 | |
2137 | ||
cf29c85e | 2138 | profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping |
b9ae0bc4 | 2139 | keywords: _best profiling |
a4072787 WC |
2140 | |
2141 | The latencytap.stp script collects data on the intervals processes | |
2142 | are deactivated (sleeping). The script categorizes the reasons for | |
2143 | the sleeps by analyzing the backtraces and displays a sorted list of | |
2144 | the top 20 causes from largest total sum time sleeping to smallest. | |
2145 | The output is updated every 30 seconds. The script needs to be | |
2146 | compiled with the '--all-modules' option to produce reasons for | |
2147 | sleeps caused by modules. Optionally, this script can be used with | |
2148 | the '-c' or '-x' options to focus on a specific PID. | |
2149 | ||
1f115be7 | 2150 | # stap latencytap.stp --all-modules -c "sleep 1" |
6a51196c | 2151 | |
210ff7bf | 2152 | |
cf29c85e | 2153 | profiling/linetimes.stp - Show Time Spent on Each Line of a Function |
0a9d3bc8 | 2154 | keywords: profiling _best |
e985df18 WC |
2155 | |
2156 | The linetimes.stp script takes two arguments: where to find the | |
2157 | function and the function name. linetimes.stp will instrument each | |
2158 | line in the function. It will print out the number of times that the | |
2159 | function is called, a table with the average and maximum time each | |
2160 | line takes, and control flow information when the script exits. | |
2161 | ||
1f115be7 | 2162 | # stap linetimes.stp kernel sys_nanosleep -c "sleep 1" |
6a51196c | 2163 | |
e985df18 | 2164 | |
eb45f1f9 FCE |
2165 | profiling/perf.stp - Show performance ratios using perf.counter to access performance counters |
2166 | keywords: profiling | |
2167 | ||
2168 | On exit the perf.stp script provides a sorted list showing cycles per | |
2169 | insn, branches per insn, and cache refs per insn | |
2170 | ||
70f3005e | 2171 | # stap -w perf.stp -c "find /usr/bin -name \"l*\" -printf \"%h/%f %s %Cx \ |
eb45f1f9 FCE |
2172 | %Ck%CM %Y\n\"" |
2173 | ||
2174 | ||
cf29c85e | 2175 | profiling/periodic.stp - Show the Period of the Various Timers on the System |
7bec2ae9 WC |
2176 | keywords: profiling |
2177 | ||
2178 | The periodic.stp script uses the kernel.trace("timer_expire_entry") | |
cf29c85e | 2179 | tracepoint to collect data on period and frequency of the various |
7bec2ae9 WC |
2180 | timers on the system. The script displays a sorted list of the |
2181 | timers observed on the system from most frequent to least frequent. | |
2182 | The script needs to be compiled with the '--all-modules' option to | |
2183 | produce list the function names. Optionally, this script can be used | |
2184 | with a numerical argument to indicate the interval in seconds between | |
2185 | printing output. | |
2186 | ||
1f115be7 | 2187 | # stap periodic.stp --all-modules -c "sleep 1" |
7bec2ae9 WC |
2188 | |
2189 | ||
9ebb5a4e FCE |
2190 | profiling/pf2.stp - Profile Kernel Functions |
2191 | keywords: profiling | |
2192 | ||
2193 | The pf2.stp script sets up time-based sampling. Every five seconds it | |
2194 | prints out a sorted list with the top ten kernel functions with | |
2195 | samples. | |
2196 | ||
1f115be7 | 2197 | # stap pf2.stp -c "sleep 1" |
9ebb5a4e FCE |
2198 | |
2199 | ||
d7dd78f0 | 2200 | profiling/pf3.stp - Profile Kernel/User Functions |
9ebb5a4e FCE |
2201 | keywords: profiling |
2202 | ||
2203 | The pf3.stp script sets up time-based sampling. Every five seconds it | |
2204 | prints out a sorted list with the top twenty kernel and/or user | |
2205 | functions with samples. Use any of --ldd, --all-modules, -d MODULE, | |
2206 | -d /PATH/TO/EXEC to add more symbolic info. | |
2207 | ||
2208 | # stap pf3.stp -c "sleep 6" --all-modules --ldd | |
2209 | ||
2210 | ||
16fc2650 | 2211 | profiling/pf4.stp - Profile Kernel/User Backtraces |
14a364c1 | 2212 | keywords: _best profiling backtrace |
16fc2650 FCE |
2213 | |
2214 | The pf4.stp script sets up time-based sampling. Every five seconds it | |
2215 | prints out a sorted list with the top twenty kernel and/or user stack | |
2216 | backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d | |
2217 | MODULE, -d /PATH/TO/EXEC to add more symbolic info. | |
2218 | ||
2219 | # stap pf4.stp -c "sleep 6" --all-modules --ldd | |
2220 | ||
2221 | ||
cf29c85e FCE |
2222 | profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler |
2223 | keywords: profiling function | |
0449af03 JS |
2224 | |
2225 | The sched_switch.stp script takes two arguments, first argument can | |
2226 | be "pid" or "name" to indicate what is being passed as second | |
2227 | argument. The script will trace the process based on pid/name and | |
2228 | print the scheduler switches happening with the process. If no | |
2229 | arguments are passed, it displays all the scheduler switches. This | |
2230 | can be used to understand which tasks schedule out the current | |
2231 | process being traced, and when it gets scheduled in again. | |
2232 | ||
6a51196c FCE |
2233 | # stap sched_switch.stp -c "sleep 1" |
2234 | ||
0449af03 | 2235 | |
cf29c85e | 2236 | profiling/thread-times.stp - Profile Kernel Functions |
96ef4c2a | 2237 | keywords: _best profiling |
210ff7bf | 2238 | |
1ae72757 JS |
2239 | The thread-times.stp script sets up time-based sampling. Every five |
2240 | seconds it prints out a sorted list with the top twenty threads | |
2241 | occupying the CPUs, broken down as a percentage of user and kernel | |
2242 | time. | |
210ff7bf | 2243 | |
1f115be7 | 2244 | # stap thread-times.stp -c "sleep 1" |
6a51196c | 2245 | |
210ff7bf | 2246 | |
ff90b297 WC |
2247 | profiling/timeout.stp - Show Processes Doing Polling Operations |
2248 | keywords: profiling | |
2249 | ||
2250 | The timeout.stp script is based on a blog entry | |
2251 | (http://udrepper.livejournal.com/19041.html) mentioning a need for a | |
2252 | tool to help developers find applications that are polling. The | |
2253 | timeout.stp script monitors systemcall used for polling and records | |
2254 | the systemcalls that timed out rather than returned because some | |
2255 | action occurred. The script updates the screen once a second with the | |
2256 | top twenty processes. | |
2257 | ||
1f115be7 | 2258 | # stap timeout.stp -c "sleep 1" |
6a51196c | 2259 | |
ff90b297 | 2260 | |
1bf72dfe WC |
2261 | profiling/topsys.stp - Show Processes Doing Polling Operations |
2262 | keywords: profiling | |
2263 | ||
2264 | The topsys.stp script lists out the top twenty systemcalls for the | |
2265 | previous 5 seconds. The output is sorted from most frequent to least | |
2266 | frequent. | |
2267 | ||
1f115be7 | 2268 | # stap topsys.stp -c "sleep 1" |
6a51196c | 2269 | |
1bf72dfe | 2270 | |
8c6bb289 WC |
2271 | = QEMU = |
2272 | ||
cf29c85e | 2273 | virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events |
8c6bb289 WC |
2274 | keywords: virtualization qemu kvm |
2275 | ||
2276 | The qemu_count.stp script tallies the number of times each of the | |
2277 | user-space qemu probepoints is encountered. When the script exits, it | |
2278 | prints a list of the number of times each user-space qemu probepoint | |
2279 | is encountered. | |
2280 | ||
1f115be7 | 2281 | # stap qemu_count.stp -c "sleep 1" |
8c6bb289 WC |
2282 | |
2283 | ||
cf29c85e | 2284 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
2285 | keywords: virtualization qemu kvm io |
2286 | ||
2287 | The qemu_io.stp script tallies the number of times each of the IO | |
2288 | port on the guest virtual machines is touched by a input or output | |
2289 | operation. When the script exits, it prints a count of the number of | |
2290 | times each IO port read and written. | |
2291 | ||
1f115be7 | 2292 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
2293 | |
2294 | ||
fefcef82 FCE |
2295 | = REGEX = |
2296 | ||
2297 | general/regex.stp - Report opened files whose names match a given regex | |
2298 | keywords: regex | |
2299 | ||
2300 | Uses the regex functionality to detect opened files whose names match | |
2301 | a pattern given on the command line. If no command line parameter is | |
2302 | given, demonstrate by filtering for files that end with an extension | |
2303 | showing them to be an archive. | |
2304 | ||
1f115be7 | 2305 | # stap regex.stp -c "sleep 1" |
fefcef82 FCE |
2306 | |
2307 | ||
210ff7bf FCE |
2308 | = SCHEDULER = |
2309 | ||
deb63545 WC |
2310 | process/chng_cpu.stp - Monitor Changes in Processor Executing a Task |
2311 | keywords: scheduler | |
2312 | ||
2313 | The chng_cpu.stp script takes an argument which is the executable | |
2314 | name of the task it should monitor. Each time a task with that | |
2315 | executable name is found running on a different processor, the script | |
2316 | prints out the thread id (tid), the executable name, the processor | |
2317 | now running the task, the thread state, and a backtrace showing the | |
2318 | kernel functions that triggered the running of the task on the | |
2319 | processor. | |
2320 | ||
1f115be7 | 2321 | # stap chng_cpu.stp -c "sleep 1" bash |
6a51196c | 2322 | |
deb63545 | 2323 | |
cf29c85e | 2324 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 2325 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
2326 | |
2327 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
2328 | to determine which processes and interrupts are competing with the | |
2329 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
2330 | options to focus on a specific task. The script output the number of | |
2331 | times the task migrates between processors, histograms showing the | |
2332 | length of time on and off processor, lists of processes running while | |
2333 | the task is off the processor, and the interrupts that occurred while | |
2334 | the task was running. | |
2335 | ||
1f115be7 | 2336 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 2337 | |
e01bc08e | 2338 | |
e6b653c8 WC |
2339 | process/forktracker.stp - Trace Creation of Processes |
2340 | keywords: process scheduler | |
2341 | ||
2342 | The forktracker.stp script prints out a time-stamped entry showing | |
f3c4da44 | 2343 | each fork and exec operation on the machine. This can be useful to |
e6b653c8 WC |
2344 | determine what process is creating a flurry of short-lived processes. |
2345 | ||
1f115be7 | 2346 | # stap forktracker.stp -c "sleep 1" |
6a51196c | 2347 | |
e6b653c8 | 2348 | |
deb63545 WC |
2349 | process/migrate.stp - Track the Migration of Specific Executables |
2350 | keywords: scheduler | |
2351 | ||
2352 | The migrate.stp script takes an argument which is the executable name | |
2353 | of the task it should monitor. Each time a task with that executable | |
2354 | name migrates between processors an entry is printed with the process | |
2355 | id (pid), the executable name, the processor off loading the task, | |
2356 | and the process taking the task. Note that the task may or may not be | |
2357 | executing at the time of the migration. | |
2358 | ||
1f115be7 | 2359 | # stap migrate.stp -c "sleep 1" bash |
6a51196c | 2360 | |
deb63545 | 2361 | |
944d282f AJ |
2362 | process/proctop.stp - Periodically Print Process Information With History |
2363 | keywords: process scheduler _best | |
2364 | ||
2365 | Every 5 seconds, print out a list of 25 processes that took the most | |
2366 | system time with information about the processes. Includes | |
2367 | information on processes that may have exited while the script was | |
2368 | running. The script contains configuration options listed in the | |
2369 | script source. | |
2370 | ||
2371 | # stap proctop.stp -c "sleep 1" | |
2372 | ||
2373 | ||
cf29c85e | 2374 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
2375 | keywords: process scheduler time tracepoint |
2376 | ||
2377 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
2378 | amount of time that each process spends in running, sleeping, |
2379 | queuing, and waiting for io. On exit the script prints out the | |
2380 | accumulated time for each state of processes observed. Optionally, | |
2381 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 2382 | specific PID and its children. |
34029cd3 | 2383 | |
1f115be7 | 2384 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 2385 | |
34029cd3 | 2386 | |
cf29c85e | 2387 | process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations |
09a98ae0 | 2388 | keywords: io scheduler backtrace |
cae71dd3 | 2389 | |
f3c4da44 | 2390 | The script monitors the time that threads spend in waiting for IO |
cae71dd3 FCE |
2391 | operations (in "D" state) in the wait_for_completion function. If a |
2392 | thread spends over 10ms, its name and backtrace is printed, and later | |
2393 | so is the total delay. | |
210ff7bf | 2394 | |
1f115be7 | 2395 | # stap sleepingBeauties.stp -c "sleep 1" |
6a51196c | 2396 | |
210ff7bf | 2397 | |
b9ae0bc4 FCE |
2398 | process/spawn_seeker.stp - Track Creation of Processes by process and execname |
2399 | keywords: process scheduler | |
2400 | ||
2401 | The spawn_seeker.stp script every minute (and on exit) prints out the | |
2402 | local time and sorted lists of which processes and executables | |
2403 | spawned tasks during the previous minute. This can be useful to | |
2404 | determine what process is creating a flurry of short-lived processes. | |
2405 | When a process exits its count of tasks created is added to its | |
2406 | parent's count to better account for the indirect task creation by | |
2407 | children processes. For more detailed examination of task creation | |
2408 | consider using forktracker.stp. | |
2409 | ||
2410 | # stap spawn_seeker.stp -c "sleep 1" | |
2411 | ||
2412 | ||
54ff5e0c FCE |
2413 | = SCSI = |
2414 | ||
cf29c85e | 2415 | io/iostat-scsi.stp - IO Statistics for SCSI Devices |
54ff5e0c FCE |
2416 | keywords: io profiling scsi |
2417 | ||
2418 | The iostat-scsi.stp script provides a breakdown of the number of blks | |
ad7e33d7 | 2419 | read and written on the machine's various SCSI devices. The script |
54ff5e0c FCE |
2420 | takes one argument which is the number of seconds between reports. |
2421 | ||
1f115be7 | 2422 | # stap -g iostat-scsi.stp 1 -c "sleep 1" |
6a51196c | 2423 | |
54ff5e0c | 2424 | |
5481d677 FCE |
2425 | = SECURITY = |
2426 | ||
7080b0c3 | 2427 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
2428 | keywords: monitoring security audit backtrace |
2429 | ||
7080b0c3 FCE |
2430 | Attaches to the kernel audit-log paths (also used by libaudit), and |
2431 | log every record being sent, along with a user-space backtrace of the | |
2432 | process that caused it. | |
a8a7b9f4 | 2433 | |
54a385da | 2434 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
2435 | |
2436 | ||
cf29c85e | 2437 | process/noptrace.stp - Disable ptrace from Hierarchies of Processes |
1ca36a99 | 2438 | keywords: process security guru |
5481d677 | 2439 | |
cf29c85e FCE |
2440 | Blocks ptrace(2) attempts from processes identified by stap -c/-x, as |
2441 | also specifiable from /proc/systemtap/stap_XXX/ control files. | |
2442 | Processes may be added or removed from the blocked list. | |
5481d677 | 2443 | |
15ecb1fe | 2444 | # stap -g noptrace.stp -c 'strace ls || true' |
6a51196c | 2445 | |
5481d677 | 2446 | |
1ca36a99 | 2447 | = SIGNALS = |
f1c6c6ed MW |
2448 | |
2449 | process/psig.stp - Print Process File Descriptors | |
1ca36a99 | 2450 | keywords: process signals |
f1c6c6ed MW |
2451 | |
2452 | Run psig.stp to produce a human-readable summary of the signal | |
2453 | handling configuration of a given process. Specify the process-id as | |
2454 | -x PID for fastest performance. | |
2455 | ||
2456 | # stap -DMAXACTION=10000 -g psig.stp -x $$ | |
2457 | ||
2458 | ||
210ff7bf | 2459 | process/sig_by_pid.stp - Signal Counts by Process ID |
09a98ae0 | 2460 | keywords: signals |
210ff7bf FCE |
2461 | |
2462 | Print signal counts by process ID in descending order. | |
2463 | ||
1f115be7 | 2464 | # stap sig_by_pid.stp -c "sleep 1" |
6a51196c | 2465 | |
210ff7bf FCE |
2466 | |
2467 | process/sig_by_proc.stp - Signal Counts by Process Name | |
09a98ae0 | 2468 | keywords: signals |
210ff7bf FCE |
2469 | |
2470 | Print signal counts by process name in descending order. | |
2471 | ||
1f115be7 | 2472 | # stap sig_by_proc.stp -c "sleep 1" |
6a51196c | 2473 | |
210ff7bf FCE |
2474 | |
2475 | process/sigkill.stp - Track SIGKILL Signals | |
09a98ae0 | 2476 | keywords: signals |
210ff7bf FCE |
2477 | |
2478 | The script traces any SIGKILL signals. When that SIGKILL signal is | |
2479 | sent to a process, the script prints out the signal name, the | |
f3c4da44 MW |
2480 | destination executable and process ID, the executable name and user |
2481 | ID that sents the signal. | |
210ff7bf | 2482 | |
1f115be7 | 2483 | # stap sigkill.stp -c "sleep 1" |
6a51196c | 2484 | |
210ff7bf | 2485 | |
cf29c85e | 2486 | process/sigmon.stp - Track a Particular Signal to a Specific Process |
09a98ae0 | 2487 | keywords: signals |
210ff7bf FCE |
2488 | |
2489 | The script watches for a particular signal sent to a specific | |
2490 | process. When that signal is sent to the specified process, the | |
2491 | script prints out the PID and executable of the process sending the | |
2492 | signal, the PID and executable name of the process receiving the | |
2493 | signal, and the signal number and name. | |
2494 | ||
1f115be7 | 2495 | # stap sigmon.stp -c "sleep 1" SIGKILL |
6a51196c | 2496 | |
210ff7bf FCE |
2497 | |
2498 | = SIMPLE = | |
2499 | ||
81a1a9e8 FCE |
2500 | general/callgraph.stp - Callgraph Tracing |
2501 | keywords: simple trace callgraph | |
2502 | ||
2503 | Print a timed per-thread microsecond-timed nested callgraph. The | |
2504 | first parameter names the function probe points to trace. | |
2505 | ||
2506 | # stap callgraph.stp 'kernel.function("*@fs/proc*.c")' -c "cat \ | |
2507 | /proc/sys/vm/* || true" | |
2508 | ||
2509 | ||
210ff7bf | 2510 | general/helloworld.stp - SystemTap "Hello World" Program |
b9ae0bc4 | 2511 | keywords: _best simple |
210ff7bf FCE |
2512 | |
2513 | A basic "Hello World" program implemented in SystemTap script. It | |
2514 | prints out "hello world" message and then immediately exits. | |
2515 | ||
6a51196c FCE |
2516 | # stap helloworld.stp |
2517 | ||
210ff7bf | 2518 | |
7955f6fd FCE |
2519 | general/py2example.stp - SystemTap python 2 support tapset |
2520 | keywords: simple | |
2521 | ||
2522 | A python support tapset that displays backtraces and variable values | |
2523 | ||
2524 | # stap -I tapset -c '/usr/bin/python2 pyexample.py 35' py2example.stp | |
2525 | ||
2526 | ||
2527 | general/py3example.stp - SystemTap python 3 support tapset | |
2528 | keywords: simple | |
2529 | ||
2530 | A python support tapset that displays backtraces and variable values | |
2531 | ||
2532 | # stap -g --suppress-time-limits -I tapset -c '/usr/bin/python3 \ | |
2533 | pyexample.py 35' py3example.stp | |
2534 | ||
2535 | ||
47300c64 | 2536 | io/eatmydata.stp - disable fsync |
1ca36a99 FCE |
2537 | keywords: io guru simple |
2538 | ||
2539 | Suppresses fsync() syscalls from processes identified by stap -c/-x | |
00b943ad FCE |
2540 | by turning them into presumed-faster fsync() on some dummy or other |
2541 | file descriptor | |
1ca36a99 FCE |
2542 | |
2543 | # stap -g eatmydata.stp -c 'strace ls || true' | |
2544 | ||
2545 | ||
42c55668 MW |
2546 | = SLAB = |
2547 | ||
cf29c85e FCE |
2548 | memory/vm.tracepoints.stp - Collect Slab Allocation Statistics |
2549 | keywords: memory slab statistics | |
42c55668 MW |
2550 | |
2551 | The script will probe all memory slab/slub allocations and collects | |
2552 | information about the size of the object (bytes requested) and | |
2553 | user-space process in execution. When run over a period of time, it | |
2554 | helps to correlate kernel-space memory consumption owing to | |
2555 | user-space processes. | |
2556 | ||
6a51196c FCE |
2557 | # stap vm.tracepoints.stp -c "sleep 10" |
2558 | ||
42c55668 | 2559 | |
cf29c85e | 2560 | = SOCKET = |
210ff7bf | 2561 | |
0e81b7f5 FCE |
2562 | lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo) |
2563 | keywords: net socket | |
2564 | ||
2565 | This traces socket duration from the accept() syscall to close(), and | |
2566 | provides details on the lifespan of these passive connections, | |
2567 | showing the distribution as a histogram. | |
2568 | ||
2569 | # stap accept2close-nd.stp -c "sleep 1" | |
2570 | ||
2571 | ||
9e749e50 JS |
2572 | network/connect_stat.stp - Show Process Ancestry for IP Connections |
2573 | keywords: network socket process | |
2574 | ||
2575 | The connect_stat.stp script prints a task's entire ancestry (parent | |
2576 | process name/uid/gid) whenever it attempts an outgoing socket | |
2577 | connection to a given IP address. | |
2578 | ||
1f115be7 | 2579 | # stap connect_stat.stp 127.0.0.1 -c "sleep 1" |
9e749e50 JS |
2580 | |
2581 | ||
cf29c85e FCE |
2582 | network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel |
2583 | keywords: network tracepoint socket | |
210ff7bf | 2584 | |
cf29c85e FCE |
2585 | Every five seconds the dropwatch.stp script lists the number of |
2586 | socket buffers freed at locations in the kernel. | |
6a51196c | 2587 | |
1f115be7 | 2588 | # stap dropwatch.stp -c "sleep 1" |
210ff7bf | 2589 | |
210ff7bf | 2590 | |
cf29c85e | 2591 | network/socket-trace.stp - Trace Functions Called in Network Socket Code |
09a98ae0 | 2592 | keywords: network socket |
210ff7bf | 2593 | |
f3c4da44 MW |
2594 | The script instruments each of the functions in the Linux kernel's |
2595 | net/socket.c file. The script prints out trace data. The first | |
2596 | element of a line is time delta in microseconds from the previous | |
2597 | entry. This is followed by the command name and the PID. The "->" and | |
2598 | "<-" indicates function entry and function exit, respectively. The | |
2599 | last element of the line is the function name. | |
210ff7bf | 2600 | |
1f115be7 | 2601 | # stap socket-trace.stp -c "sleep 1" |
6a51196c | 2602 | |
210ff7bf | 2603 | |
9141644d WC |
2604 | network/socktop - Periodically Summarize Socket Activity on the System |
2605 | keywords: network socket | |
2606 | ||
2607 | The socktop script periodically prints out a list of the processes | |
2608 | with the highest socket activity. Command line options for the | |
2609 | script allow filtering to focus on particular types of sockets. The | |
2610 | "-h" option lists socktop script's filtering options. | |
2611 | ||
6a51196c FCE |
2612 | # ./socktop -c 1 |
2613 | ||
9141644d | 2614 | |
492d227f WC |
2615 | network/tcp_connections.stp - Track Creation of Incoming TCP Connections |
2616 | keywords: network tcp socket | |
2617 | ||
2618 | The tcp_connections.stp script prints information for each new | |
2619 | incoming TCP connection accepted by the computer. The information | |
2620 | includes the UID, the command accepting the connection, the PID of | |
2621 | the command, the port the connection is on, and the IP address of the | |
2622 | originator of the request. | |
2623 | ||
1f115be7 | 2624 | # stap tcp_connections.stp -c "sleep 1" |
6a51196c | 2625 | |
492d227f | 2626 | |
cf29c85e | 2627 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 |
1ca36a99 | 2628 | keywords: network tcp socket guru |
65bdbf09 FCE |
2629 | |
2630 | Run the tcp_init_cwnd.stp script in the background to override a | |
2631 | kernel's default tcp cwnd value to 10, which has been found to | |
2632 | improve latency for web server type workloads. The script prints a | |
2633 | count of cwnd value changes when it is stopped. | |
2634 | ||
1f115be7 | 2635 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
65bdbf09 FCE |
2636 | |
2637 | ||
75ed3b3c FCE |
2638 | = SPECULATION = |
2639 | ||
2640 | general/whythefail.stp - Why did the function fail? | |
2641 | keywords: speculation monitoring function trace _best | |
2642 | ||
2643 | The whythefail.stp script prints a statement-execution trace for a | |
2644 | given function, but only for those runs of the function that ended up | |
2645 | with a (configurable) post-return condition. | |
2646 | ||
2647 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
2648 | /root/no-such-file || true' | |
2649 | ||
2650 | ||
875e1b27 FCE |
2651 | = STAPGAMES = |
2652 | ||
7955f6fd | 2653 | stapgames/2048.stp - 2048 |
b9ae0bc4 | 2654 | keywords: _best stapgames |
7955f6fd FCE |
2655 | |
2656 | The modern classic 2048 sliding-tiles game, using local keyboard and | |
2657 | ansi animation. | |
2658 | ||
2659 | # stap -p4 2048.stp | |
2660 | ||
2661 | ||
875e1b27 FCE |
2662 | stapgames/block.stp - block breaker game |
2663 | keywords: stapgames | |
2664 | ||
2665 | A block game where you progressively break the ceiling blocks until | |
2666 | clearing the level | |
2667 | ||
2668 | # stap -p4 -Itapset/ block.stp | |
2669 | ||
2670 | ||
2671 | stapgames/eater.stp - eater game | |
2672 | keywords: stapgames | |
2673 | ||
2674 | walk through a maze, eat stuff | |
2675 | ||
2676 | # stap -p4 -Itapset/ eater.stp | |
2677 | ||
2678 | ||
2679 | stapgames/lifegame.stp - life game | |
2680 | keywords: stapgames | |
2681 | ||
2682 | watch as your creation morphes into different forms | |
2683 | ||
2684 | # stap -p4 -Itapset/ lifegame.stp | |
2685 | ||
2686 | ||
2687 | stapgames/pingpong.stp - pingpong game | |
2688 | keywords: stapgames | |
2689 | ||
2690 | A simulated ball bounces around the terminal reflecting at the edges | |
2691 | ||
2692 | # stap -p4 -Itapset/ pingpong.stp | |
2693 | ||
2694 | ||
4bb6522c WC |
2695 | = STATISTICS = |
2696 | ||
beb59e38 FCE |
2697 | general/alias_suffixes.stp - Count I/O Syscalls using Alias Suffixes |
2698 | keywords: io statistics | |
2699 | ||
2700 | alias_suffixes.stp is a demonstration of how alias suffixes in the | |
2701 | systemtap language might be used. The script tracks the wall clock | |
2702 | time for each invocation of the system calls open, close, read, and | |
2703 | write. When the script exists it prints out the minimum, average, and | |
2704 | maximum times in microseconds for each system call, followed by a | |
2705 | count of times that each syscall was invoked and a histogram showing | |
2706 | the distributions of times. | |
2707 | ||
1f115be7 | 2708 | # stap alias_suffixes.stp -c "sleep 1" |
beb59e38 FCE |
2709 | |
2710 | ||
cf29c85e | 2711 | general/eventcount.stp - Count Specified Events |
b9ae0bc4 | 2712 | keywords: _best statistics thread process |
cf29c85e | 2713 | |
875e1b27 FCE |
2714 | The script periodically prints a count of specified events and their |
2715 | related tid's over the course of execution. Numerous configuration | |
2716 | options exist to control filtering / reporting, see the script | |
2717 | source. | |
cf29c85e | 2718 | |
1f115be7 | 2719 | # stap eventcount.stp syscall.* -c 'sleep 1' |
cf29c85e FCE |
2720 | |
2721 | ||
2722 | general/func_time_stats.stp - Function Time Statistics | |
2723 | keywords: function statistics | |
2724 | ||
2725 | The func_time_stats.stp script tracks the wall clock time for each | |
2726 | invocation of the function probe listed as the first command line | |
2727 | argument. When the script exits it prints out the minimum, average, | |
2728 | and maximum times in microseconds followed by a count of times that | |
2729 | the function was called and a histogram showing the distributions of | |
2730 | times. | |
2731 | ||
1f115be7 | 2732 | # stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1" |
cf29c85e FCE |
2733 | |
2734 | ||
2735 | general/sizeof.stp - Print the Size of a C Type | |
2736 | keywords: statistics memory | |
3cfd512b FCE |
2737 | |
2738 | This script prints the size of a type, based on dwarf debuginfo for | |
2739 | any kernel or userspace module, or trial-compilation of a given | |
2740 | header file name. | |
2741 | ||
6a51196c FCE |
2742 | # stap sizeof.stp FILE '</usr/include/stdio.h>' |
2743 | ||
3cfd512b | 2744 | |
cf29c85e FCE |
2745 | memory/vm.tracepoints.stp - Collect Slab Allocation Statistics |
2746 | keywords: memory slab statistics | |
2747 | ||
2748 | The script will probe all memory slab/slub allocations and collects | |
2749 | information about the size of the object (bytes requested) and | |
2750 | user-space process in execution. When run over a period of time, it | |
2751 | helps to correlate kernel-space memory consumption owing to | |
2752 | user-space processes. | |
2753 | ||
2754 | # stap vm.tracepoints.stp -c "sleep 10" | |
2755 | ||
2756 | ||
a89c2e98 FL |
2757 | network/net_xmit_json.stp - Tracks time between packet queue and transmit. |
2758 | keywords: network statistics json | |
2759 | ||
2760 | This script tracks time between packet queue and transmit. The | |
2761 | information is provided to userspace via procfs in JSON format. | |
2762 | ||
2763 | # stap net_xmit_json.stp -c "sleep 1" | |
2764 | ||
2765 | ||
d12c1a86 FCE |
2766 | network/nfsd-recent.stp - Keep track of NFS server statistics |
2767 | keywords: nfs statistics | |
2768 | ||
2769 | This script tracks all nfsd server operations by client_ip address, | |
2770 | and periodically lists those clients that have made recent requests. | |
2771 | It's a way of finding out which nfs clients might be considered still | |
2772 | connected. | |
2773 | ||
1f115be7 | 2774 | # stap nfsd-recent.stp -c "sleep 1" |
d12c1a86 FCE |
2775 | |
2776 | ||
5882f170 DS |
2777 | network/nfsdtop.stp - Keep track of NFS server statistics |
2778 | keywords: nfs statistics | |
2779 | ||
2780 | The nfsdtop.stp script gathers and displays NFS lookups, | |
2781 | ||
3b33d8c4 | 2782 | # stap nfsdtop.stp -c "sleep 1" |
5882f170 DS |
2783 | |
2784 | ||
cf29c85e | 2785 | network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets |
4bb6522c WC |
2786 | keywords: network statistics |
2787 | ||
cf29c85e FCE |
2788 | The tcpipstat script collects and displays network statistics related |
2789 | to individual TCP sockets or groups of sockets. The statistics that | |
2790 | are collected are simular to that of the command netstat -s, only | |
2791 | sorted and grouped by individual sockets. | |
4bb6522c | 2792 | |
6a51196c FCE |
2793 | # stap tcpipstat.stp timeout=1 |
2794 | ||
4bb6522c | 2795 | |
210ff7bf FCE |
2796 | = SYSCALL = |
2797 | ||
2798 | io/iotime.stp - Trace Time Spent in Read and Write for Files | |
cf29c85e | 2799 | keywords: profiling syscall io file |
210ff7bf FCE |
2800 | |
2801 | The script watches each open, close, read, and write syscalls on the | |
2802 | system. For each file the scripts observes opened it accumulates the | |
f3c4da44 | 2803 | amount of wall clock time spent in read and write operations and the |
210ff7bf FCE |
2804 | number of bytes read and written. When a file is closed the script |
2805 | prints out a pair of lines for the file. Both lines begin with a | |
2806 | timestamp in microseconds, the PID number, and the executable name in | |
ad7e33d7 | 2807 | parentheses. The first line with the "access" keyword lists the file |
210ff7bf FCE |
2808 | name, the attempted number of bytes for the read and write |
2809 | operations. The second line with the "iotime" keyword list the file | |
2810 | name and the number of microseconds accumulated in the read and write | |
2811 | syscalls. | |
2812 | ||
1f115be7 | 2813 | # stap iotime.stp -c "sleep 1" |
6a51196c | 2814 | |
210ff7bf | 2815 | |
cf29c85e | 2816 | process/errsnoop.stp - Tabulate System Call Errors |
cc20d853 ET |
2817 | keywords: process syscall |
2818 | ||
cf29c85e FCE |
2819 | Prints a periodic tabular report about failing system calls, by |
2820 | process and by syscall failure. The first optional argument | |
2821 | specifies the reporting interval (in seconds, default 5); the second | |
2822 | optional argument gives a screen height (number of lines in the | |
2823 | report, default 20). | |
cc20d853 | 2824 | |
6a51196c FCE |
2825 | # stap errsnoop.stp 1 10 -c "sleep 1" |
2826 | ||
cc20d853 | 2827 | |
210ff7bf | 2828 | process/futexes.stp - System-Wide Futex Contention |
09a98ae0 | 2829 | keywords: syscall locking futex |
210ff7bf FCE |
2830 | |
2831 | The script watches the futex syscall on the system. On exit the | |
0fb74e68 | 2832 | futex's address, the number of contentions, and the average time for |
210ff7bf FCE |
2833 | each contention on the futex are printed from lowest pid number to |
2834 | highest. | |
2835 | ||
1f115be7 | 2836 | # stap futexes.stp -c "sleep 1" |
6a51196c | 2837 | |
210ff7bf | 2838 | |
0fb74e68 JS |
2839 | process/futexes2.stp - System-Wide Shared Futex Contention |
2840 | keywords: syscall locking futex | |
2841 | ||
2842 | The script watches just shared futex syscalls on the system. On exit | |
2843 | the futex's key, the number of contentions, and the average time for | |
2844 | each contention on the futex are printed from lowest pid number to | |
2845 | highest. | |
2846 | ||
1f115be7 | 2847 | # stap futexes2.stp -c "sleep 1" |
0fb74e68 JS |
2848 | |
2849 | ||
677b471e JL |
2850 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
2851 | keywords: process monitoring syscall tracepoint | |
2852 | ||
2853 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
2854 | exit(), init_module(), and delete_module(). Event-specific details | |
2855 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
2856 | script does not require debuginfo. | |
2857 | ||
1f115be7 | 2858 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
2859 | |
2860 | ||
cf29c85e FCE |
2861 | process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls |
2862 | keywords: syscall nanosleep | |
210ff7bf FCE |
2863 | |
2864 | The script watches each nanosleep syscall on the system. At the end | |
2865 | of each nanosleep syscall the script prints out a line with a | |
2866 | timestamp in microseconds, the pid, the executable name in | |
ad7e33d7 | 2867 | parentheses, the "nanosleep:" key, and the duration of the sleep in |
210ff7bf FCE |
2868 | microseconds. |
2869 | ||
1f115be7 | 2870 | # stap sleeptime.stp -c "sleep 1" |
6a51196c | 2871 | |
210ff7bf | 2872 | |
cbaf758e | 2873 | process/strace.stp - Trace system calls |
b9ae0bc4 | 2874 | keywords: _best process syscall |
cbaf758e FCE |
2875 | |
2876 | The script loosely emulates strace, when applied to individual | |
2877 | processes or hierarchies (via -c/-x), or the entire system (without | |
2878 | -c/-x). A few output configuration parameters may be set with -G. | |
2879 | ||
2880 | # stap strace.stp -c "sleep 1" | |
2881 | ||
2882 | ||
210ff7bf | 2883 | process/syscalls_by_pid.stp - System-Wide Count of Syscalls by PID |
09a98ae0 | 2884 | keywords: syscall |
210ff7bf FCE |
2885 | |
2886 | The script watches all syscall on the system. On exit the script | |
2887 | prints a list showing the number of systemcalls executed by each PID | |
2888 | ordered from greatest to least number of syscalls. | |
2889 | ||
1f115be7 | 2890 | # stap syscalls_by_pid.stp -c "sleep 1" |
6a51196c | 2891 | |
210ff7bf FCE |
2892 | |
2893 | process/syscalls_by_proc.stp - System-Wide Count of Syscalls by Executable | |
09a98ae0 | 2894 | keywords: syscall |
210ff7bf FCE |
2895 | |
2896 | The script watches all syscall on the system. On exit the script | |
2897 | prints a list showing the number of systemcalls executed by each | |
ad7e33d7 | 2898 | executable ordered from greatest to least number of syscalls. |
210ff7bf | 2899 | |
1f115be7 | 2900 | # stap syscalls_by_proc.stp -c "sleep 1" |
6a51196c | 2901 | |
210ff7bf | 2902 | |
cf29c85e | 2903 | process/syscalltimes - System-Wide Syscall Statistics with Filtering |
04ae04b5 MW |
2904 | keywords: syscall |
2905 | ||
2906 | Combination shell/systemtap script to measure system call counts and | |
2907 | times. Can be filtered by process IDs, process names and users. | |
2908 | ||
1f115be7 | 2909 | # ./syscalltimes -c 'sleep 1' |
04ae04b5 MW |
2910 | |
2911 | ||
c29c6f83 | 2912 | process/thread-business.stp - monitor syscall history |
b9ae0bc4 | 2913 | keywords: _best process syscall |
c29c6f83 FCE |
2914 | |
2915 | Prints a periodic tabular report about the counts of syscall activity | |
2916 | of all threads on the system, along with a textual | |
2917 | recent-syscall-history for each | |
2918 | ||
2919 | # stap thread-business.stp -c "sleep 10" | |
2920 | ||
2921 | ||
210ff7bf | 2922 | process/wait4time.stp - Trace Time Spent in wait4 Syscalls |
74e5c5cf | 2923 | keywords: syscall process |
210ff7bf FCE |
2924 | |
2925 | The script watches each wait4 syscall on the system. At the end of | |
2926 | each wait4 syscall the script prints out a line with a timestamp in | |
ad7e33d7 | 2927 | microseconds, the pid, the executable name in parentheses, the |
210ff7bf FCE |
2928 | "wait4:" key, the duration of the wait and the PID that the wait4 was |
2929 | waiting for. If the waited for PID is not specified , it is "-1". | |
2930 | ||
1f115be7 | 2931 | # stap wait4time.stp -c "sleep 1" |
6a51196c | 2932 | |
210ff7bf | 2933 | |
492d227f WC |
2934 | = TCP = |
2935 | ||
905728a0 | 2936 | network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space |
cf29c85e | 2937 | keywords: network tcp process |
905728a0 WC |
2938 | |
2939 | The sk_stream-wait_memory.stp prints a time stamp, executable, and | |
2940 | pid each time a process blocks due to the send buffer being full. A | |
2941 | similar entry is printed each time a process continues because there | |
2942 | is room in the buffer. | |
2943 | ||
1f115be7 | 2944 | # stap sk_stream_wait_memory.stp -c "sleep 1" |
6a51196c | 2945 | |
905728a0 | 2946 | |
492d227f WC |
2947 | network/tcp_connections.stp - Track Creation of Incoming TCP Connections |
2948 | keywords: network tcp socket | |
2949 | ||
2950 | The tcp_connections.stp script prints information for each new | |
2951 | incoming TCP connection accepted by the computer. The information | |
2952 | includes the UID, the command accepting the connection, the PID of | |
2953 | the command, the port the connection is on, and the IP address of the | |
2954 | originator of the request. | |
2955 | ||
1f115be7 | 2956 | # stap tcp_connections.stp -c "sleep 1" |
6a51196c | 2957 | |
492d227f | 2958 | |
cf29c85e | 2959 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 |
1ca36a99 | 2960 | keywords: network tcp socket guru |
65bdbf09 FCE |
2961 | |
2962 | Run the tcp_init_cwnd.stp script in the background to override a | |
2963 | kernel's default tcp cwnd value to 10, which has been found to | |
2964 | improve latency for web server type workloads. The script prints a | |
2965 | count of cwnd value changes when it is stopped. | |
2966 | ||
1f115be7 | 2967 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
65bdbf09 FCE |
2968 | |
2969 | ||
cf29c85e | 2970 | = THREAD = |
db5bfb14 | 2971 | |
cf29c85e | 2972 | general/eventcount.stp - Count Specified Events |
b9ae0bc4 | 2973 | keywords: _best statistics thread process |
db5bfb14 | 2974 | |
875e1b27 FCE |
2975 | The script periodically prints a count of specified events and their |
2976 | related tid's over the course of execution. Numerous configuration | |
2977 | options exist to control filtering / reporting, see the script | |
2978 | source. | |
db5bfb14 | 2979 | |
1f115be7 | 2980 | # stap eventcount.stp syscall.* -c 'sleep 1' |
db5bfb14 FCE |
2981 | |
2982 | ||
875e1b27 | 2983 | process/threadstacks.stp - Override default new-pthread stack sizes |
1ca36a99 | 2984 | keywords: thread guru |
875e1b27 FCE |
2985 | |
2986 | Overrides default NPTL pthread_create stack size for all new threads | |
2987 | created by target processes. Reports one line per process when the | |
2988 | related glibc variable __default_stacksize is updated. Moot for | |
2989 | glibc versions that support $LIBC_PTHREAD_DEFAULT_STACKSIZE_NP. | |
2990 | ||
f70a0bc5 | 2991 | # stap -g threadstacks.stp -Gsize=65536 -c "sleep 1" -d `which stap` |
875e1b27 FCE |
2992 | |
2993 | ||
210ff7bf FCE |
2994 | = TIME = |
2995 | ||
1345075d WC |
2996 | general/stopwatches.stp - See the amount of wall clock time a process spends in various states |
2997 | keywords: time | |
2998 | ||
2999 | The stopwatch.stp script illustrates how to use multiple stopwatches | |
3000 | record how much wallclock time a process spends in kernel- and | |
3001 | user-space. On exit the script prints out the time in seconds, | |
3002 | milliseconds, microseconds, and nanoseconds. Note that this output of | |
3003 | this script is not directly comparable to the time command because | |
3004 | time records the time that the process is actually active in kernel- | |
3005 | and user-space. | |
3006 | ||
1f115be7 | 3007 | # stap stopwatches.stp -c "sleep 1" |
1345075d WC |
3008 | |
3009 | ||
cf29c85e | 3010 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 3011 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
3012 | |
3013 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
3014 | to determine which processes and interrupts are competing with the | |
3015 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
3016 | options to focus on a specific task. The script output the number of | |
3017 | times the task migrates between processors, histograms showing the | |
3018 | length of time on and off processor, lists of processes running while | |
3019 | the task is off the processor, and the interrupts that occurred while | |
3020 | the task was running. | |
3021 | ||
1f115be7 | 3022 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 3023 | |
e01bc08e | 3024 | |
cf29c85e | 3025 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
3026 | keywords: process scheduler time tracepoint |
3027 | ||
3028 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
3029 | amount of time that each process spends in running, sleeping, |
3030 | queuing, and waiting for io. On exit the script prints out the | |
3031 | accumulated time for each state of processes observed. Optionally, | |
3032 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 3033 | specific PID and its children. |
34029cd3 | 3034 | |
1f115be7 | 3035 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 3036 | |
34029cd3 | 3037 | |
210ff7bf FCE |
3038 | = TRACE = |
3039 | ||
81a1a9e8 FCE |
3040 | general/callgraph.stp - Callgraph Tracing |
3041 | keywords: simple trace callgraph | |
3042 | ||
3043 | Print a timed per-thread microsecond-timed nested callgraph. The | |
3044 | first parameter names the function probe points to trace. | |
3045 | ||
3046 | # stap callgraph.stp 'kernel.function("*@fs/proc*.c")' -c "cat \ | |
3047 | /proc/sys/vm/* || true" | |
3048 | ||
3049 | ||
cf29c85e | 3050 | general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments |
2515d897 JS |
3051 | keywords: trace callgraph |
3052 | ||
03568589 FCE |
3053 | Print a timed per-thread microsecond-timed callgraph, complete with |
3054 | pretty-printed function parameters and return values. The first | |
3055 | parameter names the function probe points to trace. The optional | |
3056 | second parameter names the probe points for trigger functions, which | |
3057 | acts to enable tracing for only those functions that occur while the | |
3058 | current thread is nested within the trigger. | |
2515d897 | 3059 | |
6a51196c | 3060 | # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 3061 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 3062 | |
2515d897 | 3063 | |
cf29c85e | 3064 | general/para-callgraph.stp - Callgraph Tracing with Arguments |
b9ae0bc4 | 3065 | keywords: _best trace callgraph |
210ff7bf | 3066 | |
03568589 FCE |
3067 | Print a timed per-thread microsecond-timed callgraph, complete with |
3068 | function parameters and return values. The first parameter names the | |
3069 | function probe points to trace. The optional second parameter names | |
3070 | the probe points for trigger functions, which acts to enable tracing | |
3071 | for only those functions that occur while the current thread is | |
3072 | nested within the trigger. | |
210ff7bf | 3073 | |
6a51196c | 3074 | # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 3075 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 3076 | |
210ff7bf | 3077 | |
75ed3b3c FCE |
3078 | general/whythefail.stp - Why did the function fail? |
3079 | keywords: speculation monitoring function trace _best | |
3080 | ||
3081 | The whythefail.stp script prints a statement-execution trace for a | |
3082 | given function, but only for those runs of the function that ended up | |
3083 | with a (configurable) post-return condition. | |
3084 | ||
3085 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
3086 | /root/no-such-file || true' | |
3087 | ||
3088 | ||
cf29c85e | 3089 | network/tcp_trace.stp - TCP Connection Tracing Utility |
2e251678 DW |
3090 | keywords: network trace |
3091 | ||
cf29c85e | 3092 | This scripts traces a given TCP connection based on the filter |
2e251678 DW |
3093 | parameters given by the user. The indexing is done by the 4 tuples |
3094 | local address, remote address, local port, remote port. | |
3095 | ||
6a51196c FCE |
3096 | # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1 |
3097 | ||
2e251678 | 3098 | |
0e4901b0 WC |
3099 | = TRACEPOINT = |
3100 | ||
cf29c85e FCE |
3101 | network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel |
3102 | keywords: network tracepoint socket | |
0e4901b0 WC |
3103 | |
3104 | Every five seconds the dropwatch.stp script lists the number of | |
3105 | socket buffers freed at locations in the kernel. | |
3106 | ||
1f115be7 | 3107 | # stap dropwatch.stp -c "sleep 1" |
6a51196c | 3108 | |
0e4901b0 | 3109 | |
cf29c85e | 3110 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 3111 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
3112 | |
3113 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
3114 | to determine which processes and interrupts are competing with the | |
3115 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
3116 | options to focus on a specific task. The script output the number of | |
3117 | times the task migrates between processors, histograms showing the | |
3118 | length of time on and off processor, lists of processes running while | |
3119 | the task is off the processor, and the interrupts that occurred while | |
3120 | the task was running. | |
3121 | ||
1f115be7 | 3122 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 3123 | |
e01bc08e | 3124 | |
677b471e JL |
3125 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
3126 | keywords: process monitoring syscall tracepoint | |
3127 | ||
3128 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
3129 | exit(), init_module(), and delete_module(). Event-specific details | |
3130 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
3131 | script does not require debuginfo. | |
3132 | ||
1f115be7 | 3133 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
3134 | |
3135 | ||
cf29c85e | 3136 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
3137 | keywords: process scheduler time tracepoint |
3138 | ||
3139 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
3140 | amount of time that each process spends in running, sleeping, |
3141 | queuing, and waiting for io. On exit the script prints out the | |
3142 | accumulated time for each state of processes observed. Optionally, | |
3143 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 3144 | specific PID and its children. |
34029cd3 | 3145 | |
1f115be7 | 3146 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 3147 | |
34029cd3 | 3148 | |
210ff7bf FCE |
3149 | = TRAFFIC = |
3150 | ||
53e5699f JS |
3151 | network/netdev.stp - Trace Activity on Network Devices |
3152 | keywords: network device traffic | |
3153 | ||
3154 | The netdev.stp script traces configuration and transmit/receive | |
3155 | activity on network devices. | |
3156 | ||
1f115be7 | 3157 | # stap netdev.stp -c "sleep 1" |
6a51196c | 3158 | |
53e5699f | 3159 | |
57db0e6f | 3160 | network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs |
b9ae0bc4 | 3161 | keywords: _best network traffic |
57db0e6f MW |
3162 | |
3163 | The script watches all IPv4 network traffic on the system. On exit | |
3164 | the script prints a list showing the number of packets sent along | |
3165 | source IP address / destination IP address pair encountered, and the | |
3166 | total number of bytes sent among the pair. The list is ordered from | |
3167 | greatest to least number of packets seen among the source/destination | |
3168 | pairs. | |
3169 | ||
1f115be7 | 3170 | # stap netfilter_summary.stp -c "sleep 1" |
57db0e6f MW |
3171 | |
3172 | ||
a89c2e98 FL |
3173 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs |
3174 | keywords: _best network traffic json | |
3175 | ||
3176 | The script watches all IPv4 network traffic on the system. The data | |
3177 | is output in JSON format and includes the number of packets sent | |
3178 | along source IP address / destination IP address pair encountered, | |
3179 | and the total number of bytes sent among the pair. | |
3180 | ||
3181 | # stap netfilter_summary_json.stp -c "sleep 1" | |
3182 | ||
3183 | ||
210ff7bf | 3184 | network/nettop.stp - Periodic Listing of Processes Using Network Interfaces |
74e5c5cf | 3185 | keywords: network traffic |
210ff7bf FCE |
3186 | |
3187 | Every five seconds the nettop.stp script prints out a list of | |
3188 | processed (PID and command) with the number of packets sent/received | |
3189 | and the amount of data sent/received by the process during that | |
3190 | interval. | |
3191 | ||
1f115be7 | 3192 | # stap nettop.stp -c "sleep 1" |
6a51196c | 3193 | |
210ff7bf | 3194 | |
a89c2e98 FL |
3195 | network/stp_dump.stp - Dump of STP packets |
3196 | keywords: network traffic | |
3197 | ||
3198 | The stp_dump.stp prints out the packet contents. Each block contains | |
3199 | the STP protocol ID, version ID, flags, root and bridge MAC | |
3200 | addresses, and various times. | |
3201 | ||
9e6206fd | 3202 | # stap stp_dump.stp -c "sleep 5" |
a89c2e98 FL |
3203 | |
3204 | ||
5cf50309 | 3205 | network/tcpdumplike.stp - Dump of Received UDP/TCP Packets |
7a51212c WC |
3206 | keywords: network traffic |
3207 | ||
5cf50309 FCE |
3208 | The tcpdumplike.stp prints out a line for each TCP & UDP packet |
3209 | received. Each line includes the source and destination IP addresses, | |
3210 | the source and destination ports, and flags. | |
7a51212c | 3211 | |
1f115be7 | 3212 | # stap tcpdumplike.stp -c "sleep 1" |
6a51196c | 3213 | |
7a51212c | 3214 | |
b7f6cfc5 FCE |
3215 | = TTY = |
3216 | ||
cf29c85e | 3217 | io/ttyspy.stp - Monitor TTY Typing |
1ca36a99 | 3218 | keywords: io tty monitoring guru |
b7f6cfc5 FCE |
3219 | |
3220 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
3221 | activity on the system, printing a scrolling record of recent | |
3222 | keystrokes, on a per-tty basis. | |
3223 | ||
6a51196c FCE |
3224 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" |
3225 | ||
b7f6cfc5 | 3226 | |
cf29c85e | 3227 | = UTILIZATION = |
210ff7bf FCE |
3228 | |
3229 | general/graphs.stp - Graphing Disk and CPU Utilization | |
cf29c85e | 3230 | keywords: disk cpu utilization |
210ff7bf FCE |
3231 | |
3232 | The script tracks the disk and CPU utilization. The resulting output | |
3233 | of the script can be piped into gnuplot to generate a graph of disk | |
3234 | and CPU USE. | |
3235 | ||
1f115be7 | 3236 | # stap graphs.stp -c "sleep 1" |
6a51196c | 3237 | |
210ff7bf | 3238 | |
8c6bb289 WC |
3239 | = VIRTUALIZATION = |
3240 | ||
cf29c85e | 3241 | virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons |
b9ae0bc4 | 3242 | keywords: _best virtualization kvm |
98ec0359 FCE |
3243 | |
3244 | The kvm_service_time.stp script tracks the statistics about the | |
3245 | amount of time that the processor left the guest virtual machine for | |
3246 | each exit reason (for example fixing up a page table or handling an | |
3247 | IO operation). When the script exits it prints out the number of | |
3248 | times each exit reason was encountered, the total duration of time it | |
3249 | left the guest VM, the minimum time, the average time, and the | |
3250 | maximum time in microseconds for that exit reason. On Linux 2.6.38 | |
3251 | and newer kernel the script can automatically determine whether it is | |
3252 | running on Intel or AMD processors. For older kernels with a | |
3253 | kernel.trace("kvm_exit") tracepoint that does not have the $isa | |
3254 | parameter you can explicitly state the kvm type with a "-G kvm=intel" | |
3255 | or "-G kvm=amd" on the command line. | |
3256 | ||
1f115be7 | 3257 | # stap kvm_service_time.stp -c "sleep 1" |
98ec0359 FCE |
3258 | |
3259 | ||
cf29c85e | 3260 | virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events |
8c6bb289 WC |
3261 | keywords: virtualization qemu kvm |
3262 | ||
3263 | The qemu_count.stp script tallies the number of times each of the | |
3264 | user-space qemu probepoints is encountered. When the script exits, it | |
3265 | prints a list of the number of times each user-space qemu probepoint | |
3266 | is encountered. | |
3267 | ||
1f115be7 | 3268 | # stap qemu_count.stp -c "sleep 1" |
8c6bb289 WC |
3269 | |
3270 | ||
cf29c85e | 3271 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
3272 | keywords: virtualization qemu kvm io |
3273 | ||
3274 | The qemu_io.stp script tallies the number of times each of the IO | |
3275 | port on the guest virtual machines is touched by a input or output | |
3276 | operation. When the script exits, it prints a count of the number of | |
3277 | times each IO port read and written. | |
3278 | ||
1f115be7 | 3279 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
3280 | |
3281 | ||
2dfeb3bf WC |
3282 | = WATCHDOG = |
3283 | ||
cf29c85e | 3284 | general/watchdog.stp - Watchdog Timer for Arbitrary Events |
2dfeb3bf WC |
3285 | keywords: watchdog backtrace |
3286 | ||
3287 | The watchdog.stp script provides a watchdog timer mechanism for | |
3288 | arbitrary events. The script takes three arguments: the events to | |
3289 | start watchdog timer, the event to stop the watchdog timer, and the | |
3290 | time in millseconds for the watchdog. If the watchdog timer is | |
3291 | exceed, the script will trigger a stack backtrace of the user-process | |
3292 | that timed out using pstack. This script can be used to diagnose what | |
3293 | the userspace application is doing when a slower than expected | |
3294 | operation occurs. | |
3295 | ||
3296 | # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \ | |
1f115be7 | 3297 | -c "sleep 1" |
2dfeb3bf WC |
3298 | |
3299 | ||
54d84647 WC |
3300 | = WATCHPOINT = |
3301 | ||
cf29c85e | 3302 | memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware |
54d84647 WC |
3303 | keywords: memory watchpoint |
3304 | ||
3305 | The script will watch accesses to a single kernel address and prints | |
3306 | a traceback each time the address is accessed. This script needs to | |
3307 | be run as root to allow access to the breakpoint hardware. | |
3308 | ||
23063de1 | 3309 | # stap --all-modules hw_watch_addr.stp 0x`grep "vm_dirty_ratio" \ |
0a9d3bc8 | 3310 | /proc/kallsyms | awk '{print $1}'` -c "sleep 5" |
54d84647 WC |
3311 | |
3312 | ||
cf29c85e | 3313 | memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware |
54d84647 WC |
3314 | keywords: memory watchpoint |
3315 | ||
3316 | The script will watch accesses to the starting address of a single | |
3317 | kernel symbol and prints a traceback each time the symbol is | |
3318 | accessed. This script needs to be run as root to allow access to the | |
3319 | breakpoint hardware. | |
3320 | ||
23063de1 | 3321 | # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -c "sleep 5" |
54d84647 WC |
3322 | |
3323 |