]>
Commit | Line | Data |
---|---|---|
210ff7bf | 1 | SYSTEMTAP EXAMPLES INDEX BY KEYWORD |
09a98ae0 | 2 | (see also index.txt) |
210ff7bf | 3 | |
3b33d8c4 FCE |
4 | |
5 | For systemtap version 2.9. | |
6 | ||
b9ae0bc4 FCE |
7 | = _BEST = |
8 | ||
9 | general/eventcount.stp - Count Specified Events | |
10 | keywords: _best statistics thread process | |
11 | ||
12 | The script periodically prints a count of specified events and their | |
13 | related tid's over the course of execution. Numerous configuration | |
14 | options exist to control filtering / reporting, see the script | |
15 | source. | |
16 | ||
17 | # stap eventcount.stp syscall.* -c 'sleep 1' | |
18 | ||
19 | ||
20 | general/helloworld.stp - SystemTap "Hello World" Program | |
21 | keywords: _best simple | |
22 | ||
23 | A basic "Hello World" program implemented in SystemTap script. It | |
24 | prints out "hello world" message and then immediately exits. | |
25 | ||
26 | # stap helloworld.stp | |
27 | ||
28 | ||
29 | general/para-callgraph.stp - Callgraph Tracing with Arguments | |
30 | keywords: _best trace callgraph | |
31 | ||
32 | Print a timed per-thread microsecond-timed callgraph, complete with | |
33 | function parameters and return values. The first parameter names the | |
34 | function probe points to trace. The optional second parameter names | |
35 | the probe points for trigger functions, which acts to enable tracing | |
36 | for only those functions that occur while the current thread is | |
37 | nested within the trigger. | |
38 | ||
39 | # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \ | |
40 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" | |
41 | ||
42 | ||
43 | general/varwatch.stp - Watch a Variable Changing Value in a Thread | |
44 | keywords: _best monitoring | |
45 | ||
46 | This script places a set of probes (specified by $1), each of which | |
47 | monitors the state of some context $variable expression (specified by | |
48 | $2). Whenever the value changes, with respect to the active thread, | |
49 | the event is traced. | |
50 | ||
51 | # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \ | |
52 | '$$vars' -c "sleep 1" | |
53 | ||
54 | ||
75ed3b3c FCE |
55 | general/whythefail.stp - Why did the function fail? |
56 | keywords: speculation monitoring function trace _best | |
57 | ||
58 | The whythefail.stp script prints a statement-execution trace for a | |
59 | given function, but only for those runs of the function that ended up | |
60 | with a (configurable) post-return condition. | |
61 | ||
62 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
63 | /root/no-such-file || true' | |
64 | ||
65 | ||
b9ae0bc4 FCE |
66 | network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs |
67 | keywords: _best network traffic | |
68 | ||
69 | The script watches all IPv4 network traffic on the system. On exit | |
70 | the script prints a list showing the number of packets sent along | |
71 | source IP address / destination IP address pair encountered, and the | |
72 | total number of bytes sent among the pair. The list is ordered from | |
73 | greatest to least number of packets seen among the source/destination | |
74 | pairs. | |
75 | ||
76 | # stap netfilter_summary.stp -c "sleep 1" | |
77 | ||
78 | ||
a89c2e98 FL |
79 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs |
80 | keywords: _best network traffic json | |
81 | ||
82 | The script watches all IPv4 network traffic on the system. The data | |
83 | is output in JSON format and includes the number of packets sent | |
84 | along source IP address / destination IP address pair encountered, | |
85 | and the total number of bytes sent among the pair. | |
86 | ||
87 | # stap netfilter_summary_json.stp -c "sleep 1" | |
88 | ||
89 | ||
b9ae0bc4 FCE |
90 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
91 | keywords: _best process scheduler time tracepoint interrupt | |
92 | ||
93 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
94 | to determine which processes and interrupts are competing with the | |
95 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
96 | options to focus on a specific task. The script output the number of | |
97 | times the task migrates between processors, histograms showing the | |
98 | length of time on and off processor, lists of processes running while | |
99 | the task is off the processor, and the interrupts that occurred while | |
100 | the task was running. | |
101 | ||
102 | # stap cycle_thief.stp -c "sleep 1" | |
103 | ||
104 | ||
944d282f AJ |
105 | process/proctop.stp - Periodically Print Process Information With History |
106 | keywords: process scheduler _best | |
107 | ||
108 | Every 5 seconds, print out a list of 25 processes that took the most | |
109 | system time with information about the processes. Includes | |
110 | information on processes that may have exited while the script was | |
111 | running. The script contains configuration options listed in the | |
112 | script source. | |
113 | ||
114 | # stap proctop.stp -c "sleep 1" | |
115 | ||
116 | ||
b9ae0bc4 FCE |
117 | process/strace.stp - Trace system calls |
118 | keywords: _best process syscall | |
119 | ||
120 | The script loosely emulates strace, when applied to individual | |
121 | processes or hierarchies (via -c/-x), or the entire system (without | |
122 | -c/-x). A few output configuration parameters may be set with -G. | |
123 | ||
124 | # stap strace.stp -c "sleep 1" | |
125 | ||
126 | ||
127 | process/thread-business.stp - monitor syscall history | |
128 | keywords: _best process syscall | |
129 | ||
130 | Prints a periodic tabular report about the counts of syscall activity | |
131 | of all threads on the system, along with a textual | |
132 | recent-syscall-history for each | |
133 | ||
134 | # stap thread-business.stp -c "sleep 10" | |
135 | ||
136 | ||
137 | profiling/fntimes.stp - Show Functions Taking Longer Than Usual | |
138 | keywords: _best profiling | |
139 | ||
140 | The fntimes.stp script monitors the execution time history of a given | |
141 | function family (assumed non-recursive). Each time (beyond a warmup | |
142 | interval) is then compared to the historical maximum. If it exceeds | |
143 | a certain threshold (250%), a message is printed. | |
144 | ||
145 | # stap fntimes.stp 'kernel.function("sys_*")' -c "sleep 7" | |
146 | ||
147 | ||
148 | profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping | |
149 | keywords: _best profiling | |
150 | ||
151 | The latencytap.stp script collects data on the intervals processes | |
152 | are deactivated (sleeping). The script categorizes the reasons for | |
153 | the sleeps by analyzing the backtraces and displays a sorted list of | |
154 | the top 20 causes from largest total sum time sleeping to smallest. | |
155 | The output is updated every 30 seconds. The script needs to be | |
156 | compiled with the '--all-modules' option to produce reasons for | |
157 | sleeps caused by modules. Optionally, this script can be used with | |
158 | the '-c' or '-x' options to focus on a specific PID. | |
159 | ||
160 | # stap latencytap.stp --all-modules -c "sleep 1" | |
161 | ||
162 | ||
0a9d3bc8 FCE |
163 | profiling/linetimes.stp - Show Time Spent on Each Line of a Function |
164 | keywords: profiling _best | |
165 | ||
166 | The linetimes.stp script takes two arguments: where to find the | |
167 | function and the function name. linetimes.stp will instrument each | |
168 | line in the function. It will print out the number of times that the | |
169 | function is called, a table with the average and maximum time each | |
170 | line takes, and control flow information when the script exits. | |
171 | ||
172 | # stap linetimes.stp kernel sys_nanosleep -c "sleep 1" | |
173 | ||
174 | ||
b9ae0bc4 | 175 | profiling/pf4.stp - Profile Kernel/User Backtraces |
14a364c1 | 176 | keywords: _best profiling backtrace |
b9ae0bc4 FCE |
177 | |
178 | The pf4.stp script sets up time-based sampling. Every five seconds it | |
179 | prints out a sorted list with the top twenty kernel and/or user stack | |
180 | backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d | |
181 | MODULE, -d /PATH/TO/EXEC to add more symbolic info. | |
182 | ||
183 | # stap pf4.stp -c "sleep 6" --all-modules --ldd | |
184 | ||
185 | ||
96ef4c2a FCE |
186 | profiling/thread-times.stp - Profile Kernel Functions |
187 | keywords: _best profiling | |
188 | ||
189 | The thread-times.stp script sets up time-based sampling. Every five | |
190 | seconds it prints out a sorted list with the top twenty threads | |
191 | occupying the CPUs, broken down as a percentage of user and kernel | |
192 | time. | |
193 | ||
194 | # stap thread-times.stp -c "sleep 1" | |
195 | ||
196 | ||
b9ae0bc4 FCE |
197 | stapgames/2048.stp - 2048 |
198 | keywords: _best stapgames | |
199 | ||
200 | The modern classic 2048 sliding-tiles game, using local keyboard and | |
201 | ansi animation. | |
202 | ||
203 | # stap -p4 2048.stp | |
204 | ||
205 | ||
206 | virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons | |
207 | keywords: _best virtualization kvm | |
208 | ||
209 | The kvm_service_time.stp script tracks the statistics about the | |
210 | amount of time that the processor left the guest virtual machine for | |
211 | each exit reason (for example fixing up a page table or handling an | |
212 | IO operation). When the script exits it prints out the number of | |
213 | times each exit reason was encountered, the total duration of time it | |
214 | left the guest VM, the minimum time, the average time, and the | |
215 | maximum time in microseconds for that exit reason. On Linux 2.6.38 | |
216 | and newer kernel the script can automatically determine whether it is | |
217 | running on Intel or AMD processors. For older kernels with a | |
218 | kernel.trace("kvm_exit") tracepoint that does not have the $isa | |
219 | parameter you can explicitly state the kvm type with a "-G kvm=intel" | |
220 | or "-G kvm=amd" on the command line. | |
221 | ||
222 | # stap kvm_service_time.stp -c "sleep 1" | |
223 | ||
224 | ||
a8a7b9f4 FCE |
225 | = AUDIT = |
226 | ||
7080b0c3 | 227 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
228 | keywords: monitoring security audit backtrace |
229 | ||
7080b0c3 FCE |
230 | Attaches to the kernel audit-log paths (also used by libaudit), and |
231 | log every record being sent, along with a user-space backtrace of the | |
232 | process that caused it. | |
a8a7b9f4 | 233 | |
54a385da | 234 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
235 | |
236 | ||
8b88e771 FCE |
237 | = AUTOFS = |
238 | ||
cf29c85e | 239 | network/autofs4.stp - Watch autofs4 Operations |
8b88e771 FCE |
240 | keywords: network autofs nfs |
241 | ||
242 | Trace key autofs4 operations such as mounting or unmounting remote | |
243 | filesystems. | |
244 | ||
6a51196c FCE |
245 | # stap autofs4.stp -c "sleep 1" |
246 | ||
8b88e771 | 247 | |
210ff7bf FCE |
248 | = BACKTRACE = |
249 | ||
cf29c85e | 250 | general/watchdog.stp - Watchdog Timer for Arbitrary Events |
2dfeb3bf WC |
251 | keywords: watchdog backtrace |
252 | ||
253 | The watchdog.stp script provides a watchdog timer mechanism for | |
254 | arbitrary events. The script takes three arguments: the events to | |
255 | start watchdog timer, the event to stop the watchdog timer, and the | |
256 | time in millseconds for the watchdog. If the watchdog timer is | |
257 | exceed, the script will trigger a stack backtrace of the user-process | |
258 | that timed out using pstack. This script can be used to diagnose what | |
259 | the userspace application is doing when a slower than expected | |
260 | operation occurs. | |
261 | ||
262 | # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \ | |
1f115be7 | 263 | -c "sleep 1" |
2dfeb3bf WC |
264 | |
265 | ||
cf29c85e | 266 | interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt |
1cc8a4c7 WC |
267 | keywords: interrupt backtrace |
268 | ||
269 | The Linux kernel function smp_call_function causes expensive | |
270 | inter-processor interrupts (IPIs). The scf.stp script tallies the | |
271 | processes and backtraces causing the interprocessor interrupts to | |
272 | identify the cause of the expensive IPI. On exit the script prints | |
273 | the tallies in descending frequency. | |
274 | ||
1f115be7 | 275 | # stap scf.stp -c "sleep 1" |
6a51196c | 276 | |
1cc8a4c7 | 277 | |
210ff7bf | 278 | io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call |
09a98ae0 | 279 | keywords: io backtrace |
210ff7bf FCE |
280 | |
281 | When a reschedule occurs during an AIO io_submit call, accumulate the | |
282 | traceback in a histogram. When the script exits prints out a sorted | |
283 | list from most common to least common backtrace. | |
284 | ||
1f115be7 | 285 | # stap io_submit.stp -c "sleep 1" |
6a51196c | 286 | |
210ff7bf | 287 | |
f70a0bc5 FCE |
288 | memory/last_100_frees.stp - Log recent free(3) calls. |
289 | keywords: memory process backtrace | |
290 | ||
291 | This script reports on the last few free(3) libc calls done by | |
292 | processes (possibly restricted by stap -x/-c), along with a userspace | |
293 | backtrace at those moments. | |
294 | ||
295 | # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd | |
296 | ||
297 | ||
7080b0c3 | 298 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
299 | keywords: monitoring security audit backtrace |
300 | ||
7080b0c3 FCE |
301 | Attaches to the kernel audit-log paths (also used by libaudit), and |
302 | log every record being sent, along with a user-space backtrace of the | |
303 | process that caused it. | |
a8a7b9f4 | 304 | |
54a385da | 305 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
306 | |
307 | ||
04c36a84 MW |
308 | process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands |
309 | keywords: process backtrace | |
310 | ||
311 | The pstrace_exec.stp script watches each exec operation. If the exec | |
312 | contains a substring that matches the script's command-line argument, | |
313 | it prints out that process and all of its ancestors. | |
314 | ||
1f115be7 | 315 | # stap pstrace_exec.stp -c "sleep 1" bash |
04c36a84 MW |
316 | |
317 | ||
cf29c85e | 318 | process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations |
09a98ae0 | 319 | keywords: io scheduler backtrace |
cae71dd3 | 320 | |
f3c4da44 | 321 | The script monitors the time that threads spend in waiting for IO |
cae71dd3 FCE |
322 | operations (in "D" state) in the wait_for_completion function. If a |
323 | thread spends over 10ms, its name and backtrace is printed, and later | |
324 | so is the total delay. | |
325 | ||
1f115be7 | 326 | # stap sleepingBeauties.stp -c "sleep 1" |
6a51196c | 327 | |
cae71dd3 | 328 | |
14a364c1 FCE |
329 | profiling/pf4.stp - Profile Kernel/User Backtraces |
330 | keywords: _best profiling backtrace | |
331 | ||
332 | The pf4.stp script sets up time-based sampling. Every five seconds it | |
333 | prints out a sorted list with the top twenty kernel and/or user stack | |
334 | backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d | |
335 | MODULE, -d /PATH/TO/EXEC to add more symbolic info. | |
336 | ||
337 | # stap pf4.stp -c "sleep 6" --all-modules --ldd | |
338 | ||
339 | ||
210ff7bf FCE |
340 | = CALLGRAPH = |
341 | ||
cf29c85e | 342 | general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments |
2515d897 JS |
343 | keywords: trace callgraph |
344 | ||
03568589 FCE |
345 | Print a timed per-thread microsecond-timed callgraph, complete with |
346 | pretty-printed function parameters and return values. The first | |
347 | parameter names the function probe points to trace. The optional | |
348 | second parameter names the probe points for trigger functions, which | |
349 | acts to enable tracing for only those functions that occur while the | |
350 | current thread is nested within the trigger. | |
2515d897 | 351 | |
6a51196c | 352 | # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 353 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 354 | |
2515d897 | 355 | |
cf29c85e | 356 | general/para-callgraph.stp - Callgraph Tracing with Arguments |
b9ae0bc4 | 357 | keywords: _best trace callgraph |
210ff7bf | 358 | |
03568589 FCE |
359 | Print a timed per-thread microsecond-timed callgraph, complete with |
360 | function parameters and return values. The first parameter names the | |
361 | function probe points to trace. The optional second parameter names | |
362 | the probe points for trigger functions, which acts to enable tracing | |
363 | for only those functions that occur while the current thread is | |
364 | nested within the trigger. | |
210ff7bf | 365 | |
6a51196c | 366 | # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 367 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 368 | |
210ff7bf FCE |
369 | |
370 | = CPU = | |
371 | ||
372 | general/graphs.stp - Graphing Disk and CPU Utilization | |
cf29c85e | 373 | keywords: disk cpu utilization |
210ff7bf FCE |
374 | |
375 | The script tracks the disk and CPU utilization. The resulting output | |
376 | of the script can be piped into gnuplot to generate a graph of disk | |
377 | and CPU USE. | |
378 | ||
1f115be7 | 379 | # stap graphs.stp -c "sleep 1" |
6a51196c | 380 | |
210ff7bf | 381 | |
53e5699f JS |
382 | = DEVICE = |
383 | ||
384 | network/netdev.stp - Trace Activity on Network Devices | |
385 | keywords: network device traffic | |
386 | ||
387 | The netdev.stp script traces configuration and transmit/receive | |
388 | activity on network devices. | |
389 | ||
1f115be7 | 390 | # stap netdev.stp -c "sleep 1" |
6a51196c | 391 | |
53e5699f | 392 | |
9e749e50 JS |
393 | = DIAGRAM = |
394 | ||
395 | process/pstree.stp - Generates a process diagram in DOT form. | |
396 | keywords: process diagram | |
397 | ||
398 | The pstree.stp script generates a process diagram in DOT form. For | |
399 | instance, it may be useful on a 'make' command to see all the | |
400 | processes that are started. | |
401 | ||
1f115be7 | 402 | # stap pstree.stp -c "sleep 1" |
9e749e50 JS |
403 | |
404 | ||
210ff7bf FCE |
405 | = DISK = |
406 | ||
407 | general/graphs.stp - Graphing Disk and CPU Utilization | |
cf29c85e | 408 | keywords: disk cpu utilization |
210ff7bf FCE |
409 | |
410 | The script tracks the disk and CPU utilization. The resulting output | |
411 | of the script can be piped into gnuplot to generate a graph of disk | |
412 | and CPU USE. | |
413 | ||
1f115be7 | 414 | # stap graphs.stp -c "sleep 1" |
6a51196c | 415 | |
210ff7bf | 416 | |
b96bcce8 WC |
417 | io/deviceseeks.stp - Histograms of Seek Behavior for Each Device |
418 | keywords: disk | |
419 | ||
420 | The deviceseeks.stp script generates a histogram showing the | |
875e1b27 | 421 | frequency of different sized seeks (in sectors) on each device. |
b96bcce8 | 422 | |
1f115be7 | 423 | # stap deviceseeks.stp -c "sleep 1" |
6a51196c | 424 | |
b96bcce8 | 425 | |
210ff7bf | 426 | io/disktop.stp - Summarize Disk Read/Write Traffic |
09a98ae0 | 427 | keywords: disk |
210ff7bf FCE |
428 | |
429 | Get the status of reading/writing disk every 5 seconds, output top | |
430 | ten entries during that period. | |
431 | ||
1f115be7 | 432 | # stap disktop.stp -c "sleep 1" |
6a51196c | 433 | |
210ff7bf | 434 | |
7fcecf19 MW |
435 | io/enospc.stp - Report an disk out-of-space condition. |
436 | keywords: disk filesystem | |
437 | ||
438 | This script monitors a filesystem implementations for early internal | |
439 | indications of ENOSPC, and reports these to the system logger and the | |
440 | systemtap console. | |
441 | ||
1f115be7 | 442 | # stap enospc.stp -c "sleep 1" |
7fcecf19 MW |
443 | |
444 | ||
cf29c85e | 445 | io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices |
74e5c5cf FCE |
446 | keywords: io monitoring disk |
447 | ||
448 | The mbrwatch.stp script reports any attempted reads/writes of the | |
449 | first few sectors of a raw block device. | |
450 | ||
451 | # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \ | |
199197b8 FCE |
452 | /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \ |
453 | head -1`" | |
74e5c5cf FCE |
454 | |
455 | ||
cf29c85e | 456 | network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server |
b7edd338 WC |
457 | keywords: nfs disk |
458 | ||
459 | The nfsd_unlink.stp script lists the ip address and file name each | |
460 | time time a file is being removed or unlinked by the nfsd. This | |
461 | script is run on the nfs server. | |
462 | ||
1f115be7 | 463 | # stap nfsd_unlink.stp -c "sleep 1" |
b7edd338 WC |
464 | |
465 | ||
cf29c85e | 466 | = FILE = |
db5bfb14 | 467 | |
cf29c85e FCE |
468 | io/iotime.stp - Trace Time Spent in Read and Write for Files |
469 | keywords: profiling syscall io file | |
db5bfb14 | 470 | |
cf29c85e FCE |
471 | The script watches each open, close, read, and write syscalls on the |
472 | system. For each file the scripts observes opened it accumulates the | |
473 | amount of wall clock time spent in read and write operations and the | |
474 | number of bytes read and written. When a file is closed the script | |
475 | prints out a pair of lines for the file. Both lines begin with a | |
476 | timestamp in microseconds, the PID number, and the executable name in | |
477 | parentheses. The first line with the "access" keyword lists the file | |
478 | name, the attempted number of bytes for the read and write | |
479 | operations. The second line with the "iotime" keyword list the file | |
480 | name and the number of microseconds accumulated in the read and write | |
481 | syscalls. | |
db5bfb14 | 482 | |
1f115be7 | 483 | # stap iotime.stp -c "sleep 1" |
db5bfb14 | 484 | |
ecf33ff4 | 485 | |
cf29c85e FCE |
486 | process/pfiles.stp - Print Process File Descriptors |
487 | keywords: process file | |
ecf33ff4 ET |
488 | |
489 | Run pfiles.stp to produce a human-readable summary of all open file | |
490 | descriptors of a given process. Specify the process-id as -x PID for | |
491 | fastest performance. | |
492 | ||
6a51196c FCE |
493 | # stap -g pfiles.stp -x $$ |
494 | ||
ecf33ff4 | 495 | |
ac505f97 JS |
496 | = FILESYSTEM = |
497 | ||
498 | general/badname.stp - Bad Filename Filter | |
1ca36a99 | 499 | keywords: filesystem guru |
ac505f97 JS |
500 | |
501 | The badname.stp script shows how one could prevent the creation of | |
6287a9e6 | 502 | files with undesirable names using guru mode. |
ac505f97 | 503 | |
15ecb1fe FCE |
504 | # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \ |
505 | denied" | |
6a51196c | 506 | |
ac505f97 | 507 | |
7fcecf19 MW |
508 | io/enospc.stp - Report an disk out-of-space condition. |
509 | keywords: disk filesystem | |
510 | ||
511 | This script monitors a filesystem implementations for early internal | |
512 | indications of ENOSPC, and reports these to the system logger and the | |
513 | systemtap console. | |
514 | ||
1f115be7 | 515 | # stap enospc.stp -c "sleep 1" |
7fcecf19 MW |
516 | |
517 | ||
0e81b7f5 FCE |
518 | lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo) |
519 | keywords: io filesystem | |
520 | ||
521 | This dynamically traces two common file system functions: | |
522 | do_sync_read() and do_sync_write(), and reports a histogram | |
523 | distribution of latency. Many, but not all, file systems and | |
524 | workloads use these functions. Tracing their time provides one view | |
525 | of suffered file system latency. | |
526 | ||
527 | # stap fslatency-nd.stp 1 1 | |
528 | ||
529 | ||
530 | lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo) | |
531 | keywords: io filesystem | |
532 | ||
533 | This dynamically traces two common file system functions: | |
534 | do_sync_read() and do_sync_write(), and shows details of each call | |
535 | that is slower than a threshold. Many, but not all, file systems and | |
536 | workloads use these functions. Tracing their time provides one view | |
537 | of suffered file system latency. | |
538 | ||
539 | # stap fsslower-nd.stp -c "sleep 1" | |
540 | ||
541 | ||
0e0b566a WC |
542 | = FORMAT = |
543 | ||
544 | general/ansi_colors.stp - Color Table for ansi_set_color2() and ansi_set_color3() | |
545 | keywords: format | |
546 | ||
547 | The script prints a table showing the available color combinations | |
548 | for the ansi_set_color2() and ans_set_color3() functions in the | |
549 | ansi.stp tapset. | |
550 | ||
6a51196c FCE |
551 | # stap ansi_colors.stp |
552 | ||
0e0b566a WC |
553 | |
554 | general/ansi_colors2.stp - Show Attribues in Table for ansi_set_color3() | |
555 | keywords: format | |
556 | ||
557 | The script prints a table showing the available attributes (bold, | |
558 | underline, and inverse) with color combinations for the | |
559 | ans_set_color3() function in the ansi.stp tapset. | |
560 | ||
6a51196c FCE |
561 | # stap ansi_colors2.stp |
562 | ||
0e0b566a | 563 | |
fce7a922 WC |
564 | = FUNCTION = |
565 | ||
cf29c85e FCE |
566 | general/func_time_stats.stp - Function Time Statistics |
567 | keywords: function statistics | |
fce7a922 WC |
568 | |
569 | The func_time_stats.stp script tracks the wall clock time for each | |
570 | invocation of the function probe listed as the first command line | |
571 | argument. When the script exits it prints out the minimum, average, | |
572 | and maximum times in microseconds followed by a count of times that | |
573 | the function was called and a histogram showing the distributions of | |
574 | times. | |
575 | ||
1f115be7 | 576 | # stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1" |
fce7a922 WC |
577 | |
578 | ||
75ed3b3c FCE |
579 | general/whythefail.stp - Why did the function fail? |
580 | keywords: speculation monitoring function trace _best | |
581 | ||
582 | The whythefail.stp script prints a statement-execution trace for a | |
583 | given function, but only for those runs of the function that ended up | |
584 | with a (configurable) post-return condition. | |
585 | ||
586 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
587 | /root/no-such-file || true' | |
588 | ||
589 | ||
cf29c85e FCE |
590 | profiling/functioncallcount.stp - Count Times Functions Are Called |
591 | keywords: profiling function | |
210ff7bf FCE |
592 | |
593 | The functioncallcount.stp script takes one argument, a list of | |
594 | functions to probe. The script will run and count the number of times | |
595 | that each of the functions on the list is called. On exit the script | |
596 | will print a sorted list from most frequently to least frequently | |
597 | called function. | |
598 | ||
ca5d5a88 | 599 | # stap -w functioncallcount.stp "*@mm/*.c" -c "sleep 1" |
6a51196c | 600 | |
210ff7bf | 601 | |
cf29c85e FCE |
602 | profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler |
603 | keywords: profiling function | |
0449af03 JS |
604 | |
605 | The sched_switch.stp script takes two arguments, first argument can | |
606 | be "pid" or "name" to indicate what is being passed as second | |
607 | argument. The script will trace the process based on pid/name and | |
608 | print the scheduler switches happening with the process. If no | |
609 | arguments are passed, it displays all the scheduler switches. This | |
610 | can be used to understand which tasks schedule out the current | |
611 | process being traced, and when it gets scheduled in again. | |
612 | ||
6a51196c FCE |
613 | # stap sched_switch.stp -c "sleep 1" |
614 | ||
0449af03 | 615 | |
210ff7bf FCE |
616 | = FUTEX = |
617 | ||
618 | process/futexes.stp - System-Wide Futex Contention | |
09a98ae0 | 619 | keywords: syscall locking futex |
210ff7bf FCE |
620 | |
621 | The script watches the futex syscall on the system. On exit the | |
0fb74e68 | 622 | futex's address, the number of contentions, and the average time for |
210ff7bf FCE |
623 | each contention on the futex are printed from lowest pid number to |
624 | highest. | |
625 | ||
1f115be7 | 626 | # stap futexes.stp -c "sleep 1" |
6a51196c | 627 | |
210ff7bf | 628 | |
0fb74e68 JS |
629 | process/futexes2.stp - System-Wide Shared Futex Contention |
630 | keywords: syscall locking futex | |
631 | ||
632 | The script watches just shared futex syscalls on the system. On exit | |
633 | the futex's key, the number of contentions, and the average time for | |
634 | each contention on the futex are printed from lowest pid number to | |
635 | highest. | |
636 | ||
1f115be7 | 637 | # stap futexes2.stp -c "sleep 1" |
0fb74e68 JS |
638 | |
639 | ||
1ca36a99 FCE |
640 | = GURU = |
641 | ||
642 | general/badname.stp - Bad Filename Filter | |
643 | keywords: filesystem guru | |
644 | ||
645 | The badname.stp script shows how one could prevent the creation of | |
646 | files with undesirable names using guru mode. | |
647 | ||
648 | # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \ | |
649 | denied" | |
650 | ||
651 | ||
47300c64 | 652 | io/eatmydata.stp - disable fsync |
1ca36a99 FCE |
653 | keywords: io guru simple |
654 | ||
655 | Suppresses fsync() syscalls from processes identified by stap -c/-x | |
00b943ad FCE |
656 | by turning them into presumed-faster fsync() on some dummy or other |
657 | file descriptor | |
1ca36a99 FCE |
658 | |
659 | # stap -g eatmydata.stp -c 'strace ls || true' | |
660 | ||
661 | ||
662 | io/ttyspy.stp - Monitor TTY Typing | |
663 | keywords: io tty monitoring guru | |
664 | ||
665 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
666 | activity on the system, printing a scrolling record of recent | |
667 | keystrokes, on a per-tty basis. | |
668 | ||
669 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" | |
670 | ||
671 | ||
672 | network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool | |
673 | keywords: network packets guru | |
674 | ||
675 | The script drops the specified number of packets of the specified | |
676 | protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, | |
677 | all incoming packets are dropped. The number of packets to drop can | |
678 | be specified with a positive integer. A value of 0 indicates that | |
679 | packets should be dropped until the user manually exits. | |
680 | ||
681 | # stap -g netfilter_drop.stp TCP 1 -c "sleep 2" | |
682 | ||
683 | ||
684 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 | |
685 | keywords: network tcp socket guru | |
686 | ||
687 | Run the tcp_init_cwnd.stp script in the background to override a | |
688 | kernel's default tcp cwnd value to 10, which has been found to | |
689 | improve latency for web server type workloads. The script prints a | |
690 | count of cwnd value changes when it is stopped. | |
691 | ||
1f115be7 | 692 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
1ca36a99 FCE |
693 | |
694 | ||
695 | process/noptrace.stp - Disable ptrace from Hierarchies of Processes | |
696 | keywords: process security guru | |
697 | ||
698 | Blocks ptrace(2) attempts from processes identified by stap -c/-x, as | |
699 | also specifiable from /proc/systemtap/stap_XXX/ control files. | |
700 | Processes may be added or removed from the blocked list. | |
701 | ||
702 | # stap -g noptrace.stp -c 'strace ls || true' | |
703 | ||
704 | ||
705 | process/threadstacks.stp - Override default new-pthread stack sizes | |
706 | keywords: thread guru | |
707 | ||
708 | Overrides default NPTL pthread_create stack size for all new threads | |
709 | created by target processes. Reports one line per process when the | |
710 | related glibc variable __default_stacksize is updated. Moot for | |
711 | glibc versions that support $LIBC_PTHREAD_DEFAULT_STACKSIZE_NP. | |
712 | ||
f70a0bc5 | 713 | # stap -g threadstacks.stp -Gsize=65536 -c "sleep 1" -d `which stap` |
1ca36a99 FCE |
714 | |
715 | ||
1cc8a4c7 WC |
716 | = INTERRUPT = |
717 | ||
cf29c85e | 718 | interrupt/interrupts-by-dev.stp - Record Interrupts on a Per-Device Basis |
6b95efe9 JS |
719 | keywords: interrupt |
720 | ||
721 | The interrupts-by-dev.stp script profiles interrupts received by each | |
722 | device per 100 ms. | |
723 | ||
1f115be7 | 724 | # stap interrupts-by-dev.stp -c "sleep 1" |
6a51196c | 725 | |
6b95efe9 | 726 | |
cf29c85e | 727 | interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt |
1cc8a4c7 WC |
728 | keywords: interrupt backtrace |
729 | ||
730 | The Linux kernel function smp_call_function causes expensive | |
731 | inter-processor interrupts (IPIs). The scf.stp script tallies the | |
732 | processes and backtraces causing the interprocessor interrupts to | |
733 | identify the cause of the expensive IPI. On exit the script prints | |
734 | the tallies in descending frequency. | |
735 | ||
1f115be7 | 736 | # stap scf.stp -c "sleep 1" |
6a51196c | 737 | |
1cc8a4c7 | 738 | |
cf29c85e | 739 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 740 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
741 | |
742 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
743 | to determine which processes and interrupts are competing with the | |
744 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
745 | options to focus on a specific task. The script output the number of | |
746 | times the task migrates between processors, histograms showing the | |
747 | length of time on and off processor, lists of processes running while | |
748 | the task is off the processor, and the interrupts that occurred while | |
749 | the task was running. | |
750 | ||
1f115be7 | 751 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 752 | |
e01bc08e | 753 | |
210ff7bf FCE |
754 | = IO = |
755 | ||
beb59e38 FCE |
756 | general/alias_suffixes.stp - Count I/O Syscalls using Alias Suffixes |
757 | keywords: io statistics | |
758 | ||
759 | alias_suffixes.stp is a demonstration of how alias suffixes in the | |
760 | systemtap language might be used. The script tracks the wall clock | |
761 | time for each invocation of the system calls open, close, read, and | |
762 | write. When the script exists it prints out the minimum, average, and | |
763 | maximum times in microseconds for each system call, followed by a | |
764 | count of times that each syscall was invoked and a histogram showing | |
765 | the distributions of times. | |
766 | ||
1f115be7 | 767 | # stap alias_suffixes.stp -c "sleep 1" |
beb59e38 FCE |
768 | |
769 | ||
47300c64 | 770 | io/eatmydata.stp - disable fsync |
1ca36a99 FCE |
771 | keywords: io guru simple |
772 | ||
773 | Suppresses fsync() syscalls from processes identified by stap -c/-x | |
00b943ad FCE |
774 | by turning them into presumed-faster fsync() on some dummy or other |
775 | file descriptor | |
1ca36a99 FCE |
776 | |
777 | # stap -g eatmydata.stp -c 'strace ls || true' | |
778 | ||
779 | ||
d8878d94 WC |
780 | io/inodewatch.stp - Monitoring Reads and Writes to a File |
781 | keywords: io | |
782 | ||
783 | The inodewatch.stp outputs the executable name and process id each | |
784 | time a read or write occurs to the specified inode on the specified | |
785 | major/minor device. | |
786 | ||
1f115be7 | 787 | # stap inodewatch.stp 0x08 0x01 100 -c "sleep 1" |
6a51196c | 788 | |
d8878d94 | 789 | |
7ad79dbe WC |
790 | io/inodewatch2.stp - Monitoring Attribute Changes to a File |
791 | keywords: io | |
792 | ||
793 | The inodewatch2.stp script outputs the executable name, process id, | |
794 | and attributes each time the attributes are changed on the specified | |
795 | inode on the specified major/minor device. | |
796 | ||
1f115be7 | 797 | # stap inodewatch2.stp 0x08 0x01 100 -c "sleep 1" |
6a51196c | 798 | |
7ad79dbe | 799 | |
210ff7bf | 800 | io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call |
09a98ae0 | 801 | keywords: io backtrace |
210ff7bf FCE |
802 | |
803 | When a reschedule occurs during an AIO io_submit call, accumulate the | |
804 | traceback in a histogram. When the script exits prints out a sorted | |
805 | list from most common to least common backtrace. | |
806 | ||
1f115be7 | 807 | # stap io_submit.stp -c "sleep 1" |
6a51196c | 808 | |
210ff7bf | 809 | |
67f8611b WC |
810 | io/ioblktime.stp - Average Time Block IO Requests Spend in Queue |
811 | keywords: io | |
812 | ||
813 | The ioblktime.stp script tracks the amount of time that each block IO | |
cc20d853 | 814 | requests spend waiting for completion. The script computes the |
f3c4da44 MW |
815 | average waiting time for block IO per device and prints list every 10 |
816 | seconds. In some cases there can be too many outstanding block IO | |
817 | operations and the script may exceed the default number of | |
67f8611b WC |
818 | MAXMAPENTRIES allowed. In this case the allowed number can be |
819 | increased with "-DMAXMAPENTRIES=10000" option on the stap command | |
820 | line. | |
821 | ||
1f115be7 | 822 | # stap ioblktime.stp -c "sleep 1" |
6a51196c | 823 | |
67f8611b | 824 | |
a4f3198f WC |
825 | io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device |
826 | keywords: io profiling | |
827 | ||
828 | The iodevstats.stp script measures the amount of data successfully | |
829 | read and written by all the executables for each io device on the | |
830 | system. The output is sorted from greatest sum of bytes read and | |
831 | written to a device by an executable to the least. The output | |
832 | contains device major/minor number, the count of operations (reads | |
833 | and writes), the totals and averages for the number of bytes read and | |
834 | written. | |
835 | ||
1f115be7 | 836 | # stap iodevstats.stp -c "sleep 1" |
6a51196c | 837 | |
a4f3198f | 838 | |
cf29c85e | 839 | io/iostat-scsi.stp - IO Statistics for SCSI Devices |
54ff5e0c FCE |
840 | keywords: io profiling scsi |
841 | ||
842 | The iostat-scsi.stp script provides a breakdown of the number of blks | |
ad7e33d7 | 843 | read and written on the machine's various SCSI devices. The script |
54ff5e0c FCE |
844 | takes one argument which is the number of seconds between reports. |
845 | ||
1f115be7 | 846 | # stap -g iostat-scsi.stp 1 -c "sleep 1" |
6a51196c | 847 | |
54ff5e0c | 848 | |
3e4444ed WC |
849 | io/iostats.stp - List Executables Reading and Writing the Most Data |
850 | keywords: io profiling | |
851 | ||
852 | The iostat.stp script measures the amount of data successfully read | |
853 | and written by all the executables on the system. The output is | |
854 | sorted from most greatest sum of bytes read and written by an | |
855 | executable to the least. The output contains the count of operations | |
856 | (opens, reads, and writes), the totals and averages for the number of | |
857 | bytes read and written. | |
858 | ||
1f115be7 | 859 | # stap iostats.stp -c "sleep 1" |
6a51196c | 860 | |
3e4444ed | 861 | |
210ff7bf | 862 | io/iotime.stp - Trace Time Spent in Read and Write for Files |
cf29c85e | 863 | keywords: profiling syscall io file |
210ff7bf FCE |
864 | |
865 | The script watches each open, close, read, and write syscalls on the | |
866 | system. For each file the scripts observes opened it accumulates the | |
f3c4da44 | 867 | amount of wall clock time spent in read and write operations and the |
210ff7bf FCE |
868 | number of bytes read and written. When a file is closed the script |
869 | prints out a pair of lines for the file. Both lines begin with a | |
870 | timestamp in microseconds, the PID number, and the executable name in | |
ad7e33d7 | 871 | parentheses. The first line with the "access" keyword lists the file |
210ff7bf FCE |
872 | name, the attempted number of bytes for the read and write |
873 | operations. The second line with the "iotime" keyword list the file | |
874 | name and the number of microseconds accumulated in the read and write | |
875 | syscalls. | |
876 | ||
1f115be7 | 877 | # stap iotime.stp -c "sleep 1" |
6a51196c | 878 | |
210ff7bf | 879 | |
cf29c85e | 880 | io/iotop.stp - Periodically Print IO Activity by Process Name |
09a98ae0 | 881 | keywords: io |
210ff7bf FCE |
882 | |
883 | Every five seconds print out the top ten executables generating I/O | |
884 | traffic during that interval sorted in descending order. | |
885 | ||
1f115be7 | 886 | # stap iotop.stp -c "sleep 1" |
6a51196c | 887 | |
210ff7bf | 888 | |
cf29c85e | 889 | io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices |
74e5c5cf | 890 | keywords: io monitoring disk |
1beb5089 FCE |
891 | |
892 | The mbrwatch.stp script reports any attempted reads/writes of the | |
893 | first few sectors of a raw block device. | |
894 | ||
6a51196c | 895 | # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \ |
199197b8 FCE |
896 | /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \ |
897 | head -1`" | |
6a51196c | 898 | |
1beb5089 | 899 | |
111dd9ac WC |
900 | io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process |
901 | keywords: io profiling | |
902 | ||
903 | The nfs_func_users.stp script counts the uses of NFS functions in the | |
904 | kernel on a per process bases. The output is sorted from the process | |
905 | with the greatest number of NFS functions called to the least. The | |
906 | output contains the executable name, the process number, and the | |
907 | total number of NFS functions called by the process. | |
908 | ||
1f115be7 | 909 | # stap nfs_func_users.stp -c "sleep 1" |
6a51196c | 910 | |
111dd9ac | 911 | |
a11f34b0 FCE |
912 | io/slowvfs.stp - Trace slow vfs opens. |
913 | keywords: io | |
914 | ||
915 | This script prints a line for every kernel vfs_open operation that | |
916 | takes longer than a configurable number of microseconds. Highly | |
917 | contended or remote filesystems are likelier to hit this. | |
918 | ||
919 | # stap slowvfs.stp -G sloth=10 -c 'find /proc >/dev/null' | |
920 | ||
921 | ||
922 | io/switchfile.stp - Switch log files | |
923 | keywords: io | |
924 | ||
925 | Every second print a log message and switch log files every 5 | |
926 | seconds. | |
927 | ||
928 | # stap -o switchfile.stp.out switchfile.stp -c "sleep 8" | |
929 | ||
930 | ||
cf29c85e | 931 | io/traceio.stp - Track Cumulative IO Activity by Process Name |
09a98ae0 | 932 | keywords: io |
210ff7bf FCE |
933 | |
934 | Every second print out the top ten executables sorted in descending | |
935 | order based on cumulative I/O traffic observed. | |
936 | ||
1f115be7 | 937 | # stap traceio.stp -c "sleep 1" |
6a51196c | 938 | |
210ff7bf | 939 | |
cf29c85e | 940 | io/traceio2.stp - Watch IO Activity on a Particular Device |
09a98ae0 | 941 | keywords: io |
210ff7bf FCE |
942 | |
943 | Print out the executable name and process number as reads and writes | |
944 | to the specified device occur. | |
945 | ||
1f115be7 | 946 | # stap traceio2.stp 0x0801 -c "sleep 1" |
6a51196c | 947 | |
210ff7bf | 948 | |
cf29c85e | 949 | io/ttyspy.stp - Monitor TTY Typing |
1ca36a99 | 950 | keywords: io tty monitoring guru |
b7f6cfc5 FCE |
951 | |
952 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
953 | activity on the system, printing a scrolling record of recent | |
954 | keystrokes, on a per-tty basis. | |
955 | ||
6a51196c FCE |
956 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" |
957 | ||
b7f6cfc5 | 958 | |
0e81b7f5 FCE |
959 | lwtools/biolatency-nd.stp - Measure block I/O latency distribution (non-debuginfo) |
960 | keywords: io | |
961 | ||
962 | This measures block I/O latency (storage I/O, ie, disk I/O), and | |
963 | shows the distribution as a histogram. This can be useful to identify | |
964 | the characteristics of I/O latency, beyond the averages shown by | |
965 | iostat(1). For example, to study I/O latency outliers, or multi-modal | |
966 | distributions. | |
967 | ||
968 | # stap biolatency-nd.stp 1 1 | |
969 | ||
970 | ||
971 | lwtools/bitesize-nd.stp - Measure block I/O size distribution (non-debuginfo) | |
972 | keywords: io | |
973 | ||
974 | This uses the kernel tracepoint block_rq_insert to read the size of | |
975 | I/O. The output includes the name of the process or thread that was | |
976 | on-CPU when the I/O request was inserted on the issue queue. | |
977 | ||
978 | # stap bitesize-nd.stp -c "sleep 1" | |
979 | ||
980 | ||
981 | lwtools/execsnoop-nd.stp - Trace process exec() with command line argument details (non-debuginfo) | |
982 | keywords: io | |
983 | ||
984 | This can identify if CPU is consumed by short-lived processes, by | |
985 | tracing new process execution. It works by tracing exec() from the | |
986 | fork()->exec() sequence, which means it will not catch new processes | |
987 | that only fork(). It will also show every exec(), including those if | |
988 | a process re-execs. | |
989 | ||
990 | # stap execsnoop-nd.stp -c "sleep 1" | |
991 | ||
992 | ||
993 | lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo) | |
994 | keywords: io filesystem | |
995 | ||
996 | This dynamically traces two common file system functions: | |
997 | do_sync_read() and do_sync_write(), and reports a histogram | |
998 | distribution of latency. Many, but not all, file systems and | |
999 | workloads use these functions. Tracing their time provides one view | |
1000 | of suffered file system latency. | |
1001 | ||
1002 | # stap fslatency-nd.stp 1 1 | |
1003 | ||
1004 | ||
1005 | lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo) | |
1006 | keywords: io filesystem | |
1007 | ||
1008 | This dynamically traces two common file system functions: | |
1009 | do_sync_read() and do_sync_write(), and shows details of each call | |
1010 | that is slower than a threshold. Many, but not all, file systems and | |
1011 | workloads use these functions. Tracing their time provides one view | |
1012 | of suffered file system latency. | |
1013 | ||
1014 | # stap fsslower-nd.stp -c "sleep 1" | |
1015 | ||
1016 | ||
1017 | lwtools/killsnoop-nd.stp - Trace kill() signals showing process and signal details (non-debuginfo) | |
1018 | keywords: io | |
1019 | ||
1020 | This traces signals system-wide, including those sent by the kill(1) | |
1021 | command, and shows various details. | |
1022 | ||
1023 | # stap killsnoop-nd.stp -c "sleep 1" | |
1024 | ||
1025 | ||
1026 | lwtools/opensnoop-nd.stp - Trace open() syscalls showing filenames (non-debuginfo) | |
1027 | keywords: io | |
1028 | ||
1029 | This traces the open() syscall system-wide, to show which files are | |
1030 | being opened, and by who. | |
1031 | ||
1032 | # stap opensnoop-nd.stp -c "sleep 1" | |
1033 | ||
1034 | ||
1035 | lwtools/rwtime-nd.stp - Summarize read() and write() syscall latency (non-debuginfo) | |
1036 | keywords: io | |
1037 | ||
1038 | This traces read() and write() syscalls, producing a histogram | |
1039 | summary of their durations (aka latencies). | |
1040 | ||
1041 | # stap rwtime-nd.stp -c "sleep 1" | |
1042 | ||
1043 | ||
1044 | lwtools/syscallbypid-nd.stp - Count syscalls with process details (non-debuginfo) | |
1045 | keywords: io | |
1046 | ||
1047 | This traces syscalls system-wide, and produces a summary report | |
1048 | showing their counts by process ID, process name, and syscall types. | |
1049 | ||
1050 | # stap syscallbypid-nd.stp -c "sleep 1" | |
1051 | ||
1052 | ||
cf29c85e | 1053 | process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations |
09a98ae0 | 1054 | keywords: io scheduler backtrace |
210ff7bf | 1055 | |
f3c4da44 | 1056 | The script monitors the time that threads spend in waiting for IO |
cae71dd3 FCE |
1057 | operations (in "D" state) in the wait_for_completion function. If a |
1058 | thread spends over 10ms, its name and backtrace is printed, and later | |
1059 | so is the total delay. | |
210ff7bf | 1060 | |
1f115be7 | 1061 | # stap sleepingBeauties.stp -c "sleep 1" |
6a51196c | 1062 | |
210ff7bf | 1063 | |
cf29c85e | 1064 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
1065 | keywords: virtualization qemu kvm io |
1066 | ||
1067 | The qemu_io.stp script tallies the number of times each of the IO | |
1068 | port on the guest virtual machines is touched by a input or output | |
1069 | operation. When the script exits, it prints a count of the number of | |
1070 | times each IO port read and written. | |
1071 | ||
1f115be7 | 1072 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
1073 | |
1074 | ||
a89c2e98 FL |
1075 | = JSON = |
1076 | ||
1077 | network/net_xmit_json.stp - Tracks time between packet queue and transmit. | |
1078 | keywords: network statistics json | |
1079 | ||
1080 | This script tracks time between packet queue and transmit. The | |
1081 | information is provided to userspace via procfs in JSON format. | |
1082 | ||
1083 | # stap net_xmit_json.stp -c "sleep 1" | |
1084 | ||
1085 | ||
1086 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs | |
1087 | keywords: _best network traffic json | |
1088 | ||
1089 | The script watches all IPv4 network traffic on the system. The data | |
1090 | is output in JSON format and includes the number of packets sent | |
1091 | along source IP address / destination IP address pair encountered, | |
1092 | and the total number of bytes sent among the pair. | |
1093 | ||
1094 | # stap netfilter_summary_json.stp -c "sleep 1" | |
1095 | ||
1096 | ||
8c6bb289 WC |
1097 | = KVM = |
1098 | ||
cf29c85e | 1099 | virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons |
b9ae0bc4 | 1100 | keywords: _best virtualization kvm |
98ec0359 FCE |
1101 | |
1102 | The kvm_service_time.stp script tracks the statistics about the | |
1103 | amount of time that the processor left the guest virtual machine for | |
1104 | each exit reason (for example fixing up a page table or handling an | |
1105 | IO operation). When the script exits it prints out the number of | |
1106 | times each exit reason was encountered, the total duration of time it | |
1107 | left the guest VM, the minimum time, the average time, and the | |
1108 | maximum time in microseconds for that exit reason. On Linux 2.6.38 | |
1109 | and newer kernel the script can automatically determine whether it is | |
1110 | running on Intel or AMD processors. For older kernels with a | |
1111 | kernel.trace("kvm_exit") tracepoint that does not have the $isa | |
1112 | parameter you can explicitly state the kvm type with a "-G kvm=intel" | |
1113 | or "-G kvm=amd" on the command line. | |
1114 | ||
1f115be7 | 1115 | # stap kvm_service_time.stp -c "sleep 1" |
98ec0359 FCE |
1116 | |
1117 | ||
cf29c85e | 1118 | virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events |
8c6bb289 WC |
1119 | keywords: virtualization qemu kvm |
1120 | ||
1121 | The qemu_count.stp script tallies the number of times each of the | |
1122 | user-space qemu probepoints is encountered. When the script exits, it | |
1123 | prints a list of the number of times each user-space qemu probepoint | |
1124 | is encountered. | |
1125 | ||
1f115be7 | 1126 | # stap qemu_count.stp -c "sleep 1" |
8c6bb289 WC |
1127 | |
1128 | ||
cf29c85e | 1129 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
1130 | keywords: virtualization qemu kvm io |
1131 | ||
1132 | The qemu_io.stp script tallies the number of times each of the IO | |
1133 | port on the guest virtual machines is touched by a input or output | |
1134 | operation. When the script exits, it prints a count of the number of | |
1135 | times each IO port read and written. | |
1136 | ||
1f115be7 | 1137 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
1138 | |
1139 | ||
cf29c85e FCE |
1140 | = LIMITS = |
1141 | ||
1142 | memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits | |
1143 | keywords: memory limits | |
1144 | ||
1145 | The overcommit.stp script prints a line each time the kernel refuses | |
1146 | a memory allocation request from a process because of | |
1147 | /proc/sys/vm/overcommit* limits. | |
1148 | ||
1f115be7 FCE |
1149 | # stap overcommit.stp -c "sleep 1" |
1150 | ||
1151 | ||
1152 | process/rlimit_nofile.stp - Trace processes running out of file descriptors | |
1153 | keywords: limits | |
1154 | ||
1155 | This script watches processes being scheduled and which try to | |
1156 | allocate a file descriptor without luck. | |
1157 | ||
1158 | # stap rlimit_nofile.stp -c "sleep 1" | |
cf29c85e FCE |
1159 | |
1160 | ||
210ff7bf FCE |
1161 | = LOCKING = |
1162 | ||
cf29c85e | 1163 | locks/bkl.stp - Tracing Contention on Big Kernel Lock |
cf5023fb WC |
1164 | keywords: locking |
1165 | ||
1166 | The bkl.stp script can help determine whether the Big Kernel Lock | |
1167 | (BKL) is causing serialization on a multiprocessor system due to | |
a11f34b0 FCE |
1168 | excessive contention of the BKL. The bkl.stp script takes two |
1169 | arguments. The first one is optional, and used to enable backtraces, | |
1170 | and print them once a process has been holding the BKL for a user | |
1171 | specified number of nseconds is reached. The second option is | |
1172 | compulsory and is the number of processes waiting for the Big Kernel | |
cf5023fb WC |
1173 | Lock (BKL). When the number of processes waiting for the BKL is |
1174 | reached or exceeded, the script will print a time stamp, the number | |
1175 | of processes waiting for the BKL, the holder of the BKL, and the | |
a11f34b0 FCE |
1176 | amount of time the BKL was held. If backtraces are enabled, a |
1177 | backtrace will be printed as well. | |
cf5023fb | 1178 | |
1f115be7 | 1179 | # stap bkl.stp -c "sleep 1" 1 |
6a51196c | 1180 | |
cf5023fb | 1181 | |
cf29c85e | 1182 | locks/bkl_stats.stp - Per Process Statistics on Big Kernel Lock Use |
cf5023fb WC |
1183 | keywords: locking |
1184 | ||
1185 | The bkl_stats.stp script can indicate which processes have excessive | |
1186 | waits for the Big Kernel Lock (BKL) and which processes are taking | |
1187 | the BKL for long periods of time. The bkl_stats.stp script prints | |
1188 | lists of all the processes that require the BKL. Every five seconds | |
1189 | two tables are printed out. The first table lists the processes that | |
1190 | waited for the BKL followed by the number of times that the process | |
1191 | waited, the minimum time of the wait, the average and the maximum | |
1192 | time waited. The second table lists has similar information for the | |
f3c4da44 | 1193 | time spent in holding the lock for each of the processes. |
cf5023fb | 1194 | |
1f115be7 | 1195 | # stap bkl_stats.stp -c "sleep 1" |
6a51196c | 1196 | |
cf5023fb | 1197 | |
210ff7bf | 1198 | process/futexes.stp - System-Wide Futex Contention |
09a98ae0 | 1199 | keywords: syscall locking futex |
210ff7bf FCE |
1200 | |
1201 | The script watches the futex syscall on the system. On exit the | |
0fb74e68 | 1202 | futex's address, the number of contentions, and the average time for |
210ff7bf FCE |
1203 | each contention on the futex are printed from lowest pid number to |
1204 | highest. | |
1205 | ||
1f115be7 | 1206 | # stap futexes.stp -c "sleep 1" |
6a51196c | 1207 | |
210ff7bf | 1208 | |
0fb74e68 JS |
1209 | process/futexes2.stp - System-Wide Shared Futex Contention |
1210 | keywords: syscall locking futex | |
1211 | ||
1212 | The script watches just shared futex syscalls on the system. On exit | |
1213 | the futex's key, the number of contentions, and the average time for | |
1214 | each contention on the futex are printed from lowest pid number to | |
1215 | highest. | |
1216 | ||
1f115be7 | 1217 | # stap futexes2.stp -c "sleep 1" |
0fb74e68 JS |
1218 | |
1219 | ||
5a224324 FCE |
1220 | process/mutex-contention.stp - pthread mutex contention analysis |
1221 | keywords: locking | |
1222 | ||
1223 | Tracks pthread-mutex initialization/use and underlying futex | |
1224 | operations, to identify (with backtraces/symbol-names) the mutexes | |
f275b3f7 FCE |
1225 | suffering most contention. Invoke with "-d SHLIB --ldd", perhaps |
1226 | with -DMAXMAPENTRIES=NNNN for some large NNNN, if the arrays overflow | |
1227 | due to heavy activity. | |
5a224324 | 1228 | |
1f115be7 | 1229 | # stap mutex-contention.stp -c "sleep 1" |
5a224324 FCE |
1230 | |
1231 | ||
5cf50309 FCE |
1232 | process/semop-watch.stp - Watch semop(2)/semtimedop(2) operations |
1233 | keywords: process locking | |
1234 | ||
1235 | Prints a timed trace of semop(2)/semtimedop(2) syscalls | |
1236 | ||
1237 | # stap semop-watch.stp -c 'sleep 2' | |
1238 | ||
1239 | ||
22f971e8 WC |
1240 | = MEMORY = |
1241 | ||
cf29c85e FCE |
1242 | general/sizeof.stp - Print the Size of a C Type |
1243 | keywords: statistics memory | |
3cfd512b FCE |
1244 | |
1245 | This script prints the size of a type, based on dwarf debuginfo for | |
1246 | any kernel or userspace module, or trial-compilation of a given | |
1247 | header file name. | |
1248 | ||
6a51196c FCE |
1249 | # stap sizeof.stp FILE '</usr/include/stdio.h>' |
1250 | ||
3cfd512b | 1251 | |
c4abda95 SP |
1252 | memory/glibc-malloc.stp - Overview glibc malloc internal operations |
1253 | keywords: memory process | |
1254 | ||
1255 | This script reports on internal statistics of the glibc malloc | |
1256 | implementation, as used by a process restricted by stap -x/-c | |
1257 | ||
1258 | # stap glibc-malloc.stp -c 'stap --dump-functions' | |
1259 | ||
1260 | ||
cf29c85e | 1261 | memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware |
54d84647 WC |
1262 | keywords: memory watchpoint |
1263 | ||
1264 | The script will watch accesses to a single kernel address and prints | |
1265 | a traceback each time the address is accessed. This script needs to | |
1266 | be run as root to allow access to the breakpoint hardware. | |
1267 | ||
23063de1 | 1268 | # stap --all-modules hw_watch_addr.stp 0x`grep "vm_dirty_ratio" \ |
0a9d3bc8 | 1269 | /proc/kallsyms | awk '{print $1}'` -c "sleep 5" |
54d84647 WC |
1270 | |
1271 | ||
cf29c85e | 1272 | memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware |
54d84647 WC |
1273 | keywords: memory watchpoint |
1274 | ||
1275 | The script will watch accesses to the starting address of a single | |
1276 | kernel symbol and prints a traceback each time the symbol is | |
1277 | accessed. This script needs to be run as root to allow access to the | |
1278 | breakpoint hardware. | |
1279 | ||
23063de1 | 1280 | # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -c "sleep 5" |
54d84647 WC |
1281 | |
1282 | ||
cf29c85e | 1283 | memory/kmalloc-top - Show Paths to Kernel Malloc Invocations |
22f971e8 WC |
1284 | keywords: memory |
1285 | ||
1286 | The kmalloc-top perl program runs a small systemtap script to collect | |
1287 | stack traces for each call to the kmalloc function and counts the | |
1288 | time that each stack trace is observed. When kmalloc-top exits it | |
f3c4da44 MW |
1289 | prints out sorted list. The output can be filtered to print only the |
1290 | first N stack traces (-t), stack traces with a minimum counts (-m), | |
1291 | or exclude certain stack traces (-e). | |
22f971e8 | 1292 | |
1f115be7 | 1293 | # ./kmalloc-top -c "sleep 1" |
6a51196c | 1294 | |
22f971e8 | 1295 | |
f70a0bc5 FCE |
1296 | memory/last_100_frees.stp - Log recent free(3) calls. |
1297 | keywords: memory process backtrace | |
1298 | ||
1299 | This script reports on the last few free(3) libc calls done by | |
1300 | processes (possibly restricted by stap -x/-c), along with a userspace | |
1301 | backtrace at those moments. | |
1302 | ||
1303 | # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd | |
1304 | ||
1305 | ||
0dc23d1d WC |
1306 | memory/mmanonpage.stp - Track Virtual Memory System Actions on Anonymous Pages |
1307 | keywords: memory | |
1308 | ||
1309 | The mmanonpage.stp script uses the virtual memory tracepoints | |
1310 | available in some kernels to track the number of faults, user space | |
1311 | frees, page ins, copy on writes and unmaps for anonymous pages. When | |
1312 | the script is terminated the counts are printed for each process that | |
1313 | allocated pages while the script was running. This script displays | |
1314 | the anonymous page statistics for each process that ran while the | |
f3c4da44 | 1315 | script is active. It's useful in debugging leaks in the anonymous |
0dc23d1d WC |
1316 | regions of a process. |
1317 | ||
1f115be7 | 1318 | # stap mmanonpage.stp -c "sleep 1" |
6a51196c | 1319 | |
0dc23d1d WC |
1320 | |
1321 | memory/mmfilepage.stp - Track Virtual Memory System Actions on File Backed Pages | |
1322 | keywords: memory | |
1323 | ||
1324 | The mmfilepage.stp script uses the virtual memory tracepoints | |
1325 | available in some kernels to track the number of faults, copy on | |
1326 | writes mapping, and unmapping operations for file backed pages. When | |
1327 | the script is terminated the counts are printed for each process that | |
1328 | allocated pages while the script was running. The mmfilepage.stp | |
1329 | script is useful in debugging leaks in the mapped file regions of a | |
1330 | process. | |
1331 | ||
1f115be7 | 1332 | # stap mmfilepage.stp -c "sleep 1" |
6a51196c | 1333 | |
0dc23d1d WC |
1334 | |
1335 | memory/mmreclaim.stp - Track Virtual Memory System Page Reclamation | |
1336 | keywords: memory | |
1337 | ||
1338 | The mmreclaim.stp script uses the virtual memory tracepoints | |
ad7e33d7 | 1339 | available in some kernels to track page reclaim activity that |
f3c4da44 | 1340 | occurred while the script was running. It's useful in debugging |
ad7e33d7 | 1341 | performance problems that occur due to page reclamation. |
0dc23d1d | 1342 | |
1f115be7 | 1343 | # stap mmreclaim.stp -c "sleep 1" |
6a51196c | 1344 | |
0dc23d1d WC |
1345 | |
1346 | memory/mmwriteback.stp - Track Virtual Memory System Writing to Disk | |
1347 | keywords: memory | |
1348 | ||
1349 | The mmwriteback.stp script uses the virtual memory tracepoints | |
1350 | available in some kernels to report all of the file writebacks that | |
1351 | occur form kupdate, pdflush and kjournald while the script is | |
f3c4da44 MW |
1352 | running. It's useful in determining where writes are coming from on |
1353 | a supposedly idle system that is experiencing unexpected IO. | |
0dc23d1d | 1354 | |
1f115be7 | 1355 | # stap mmwriteback.stp -c "sleep 1" |
6a51196c | 1356 | |
0dc23d1d | 1357 | |
c728b7da WC |
1358 | memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes |
1359 | keywords: memory numa | |
1360 | ||
1361 | The numa_faults.stp script tracks the read and write pages faults for | |
1362 | each process. When the script exits it prints out the total read and | |
ad7e33d7 | 1363 | write pages faults for each process. The script also provide a break |
c728b7da WC |
1364 | down of page faults per node for each process. This script is useful |
1365 | for determining whether the program has good locality (page faults | |
1366 | limited to a single node) on a NUMA computer. | |
1367 | ||
1f115be7 | 1368 | # stap numa_faults.stp -c "sleep 1" |
6a51196c | 1369 | |
c728b7da | 1370 | |
cf29c85e FCE |
1371 | memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits |
1372 | keywords: memory limits | |
1ff1a65d FCE |
1373 | |
1374 | The overcommit.stp script prints a line each time the kernel refuses | |
1375 | a memory allocation request from a process because of | |
1376 | /proc/sys/vm/overcommit* limits. | |
1377 | ||
1f115be7 | 1378 | # stap overcommit.stp -c "sleep 1" |
6a51196c | 1379 | |
1ff1a65d | 1380 | |
413996e0 WC |
1381 | memory/pfaults.stp - Generate Log of Major and Minor Page Faults |
1382 | keywords: memory | |
1383 | ||
1384 | The pfaults.stp script generates a simple log for each major and | |
1385 | minor page fault that occurs on the system. Each line contains a | |
1386 | timestamp (in microseconds) when the page fault servicing was | |
1387 | completed, the pid of the process, the address of the page fault, the | |
1388 | type of access (read or write), the type of fault (major or minor), | |
1389 | and the elapsed time for page fault. This log can be examined to | |
ad7e33d7 | 1390 | determine where the page faults are occurring. |
413996e0 | 1391 | |
1f115be7 | 1392 | # stap pfaults.stp -c "sleep 1" |
6a51196c | 1393 | |
413996e0 | 1394 | |
cf29c85e FCE |
1395 | memory/vm.tracepoints.stp - Collect Slab Allocation Statistics |
1396 | keywords: memory slab statistics | |
42c55668 MW |
1397 | |
1398 | The script will probe all memory slab/slub allocations and collects | |
1399 | information about the size of the object (bytes requested) and | |
1400 | user-space process in execution. When run over a period of time, it | |
1401 | helps to correlate kernel-space memory consumption owing to | |
1402 | user-space processes. | |
1403 | ||
6a51196c FCE |
1404 | # stap vm.tracepoints.stp -c "sleep 10" |
1405 | ||
42c55668 | 1406 | |
14f0bb18 | 1407 | = MONITORING = |
b7f6cfc5 | 1408 | |
cf29c85e | 1409 | general/varwatch.stp - Watch a Variable Changing Value in a Thread |
b9ae0bc4 | 1410 | keywords: _best monitoring |
b7f6cfc5 | 1411 | |
14f0bb18 FCE |
1412 | This script places a set of probes (specified by $1), each of which |
1413 | monitors the state of some context $variable expression (specified by | |
1414 | $2). Whenever the value changes, with respect to the active thread, | |
1415 | the event is traced. | |
b7f6cfc5 | 1416 | |
6a51196c | 1417 | # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \ |
1f115be7 | 1418 | '$$vars' -c "sleep 1" |
6a51196c | 1419 | |
1beb5089 | 1420 | |
75ed3b3c FCE |
1421 | general/whythefail.stp - Why did the function fail? |
1422 | keywords: speculation monitoring function trace _best | |
1423 | ||
1424 | The whythefail.stp script prints a statement-execution trace for a | |
1425 | given function, but only for those runs of the function that ended up | |
1426 | with a (configurable) post-return condition. | |
1427 | ||
1428 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
1429 | /root/no-such-file || true' | |
1430 | ||
1431 | ||
cf29c85e | 1432 | io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices |
74e5c5cf | 1433 | keywords: io monitoring disk |
1beb5089 FCE |
1434 | |
1435 | The mbrwatch.stp script reports any attempted reads/writes of the | |
1436 | first few sectors of a raw block device. | |
1437 | ||
6a51196c | 1438 | # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \ |
199197b8 FCE |
1439 | /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \ |
1440 | head -1`" | |
6a51196c | 1441 | |
1beb5089 | 1442 | |
cf29c85e | 1443 | io/ttyspy.stp - Monitor TTY Typing |
1ca36a99 | 1444 | keywords: io tty monitoring guru |
14f0bb18 FCE |
1445 | |
1446 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
1447 | activity on the system, printing a scrolling record of recent | |
1448 | keystrokes, on a per-tty basis. | |
1449 | ||
6a51196c FCE |
1450 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" |
1451 | ||
14f0bb18 | 1452 | |
7080b0c3 | 1453 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
1454 | keywords: monitoring security audit backtrace |
1455 | ||
7080b0c3 FCE |
1456 | Attaches to the kernel audit-log paths (also used by libaudit), and |
1457 | log every record being sent, along with a user-space backtrace of the | |
1458 | process that caused it. | |
a8a7b9f4 | 1459 | |
54a385da | 1460 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
1461 | |
1462 | ||
677b471e JL |
1463 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
1464 | keywords: process monitoring syscall tracepoint | |
1465 | ||
1466 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
1467 | exit(), init_module(), and delete_module(). Event-specific details | |
1468 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
1469 | script does not require debuginfo. | |
1470 | ||
1f115be7 | 1471 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
1472 | |
1473 | ||
cf29c85e FCE |
1474 | = NANOSLEEP = |
1475 | ||
1476 | process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls | |
1477 | keywords: syscall nanosleep | |
1478 | ||
1479 | The script watches each nanosleep syscall on the system. At the end | |
1480 | of each nanosleep syscall the script prints out a line with a | |
1481 | timestamp in microseconds, the pid, the executable name in | |
1482 | parentheses, the "nanosleep:" key, and the duration of the sleep in | |
1483 | microseconds. | |
1484 | ||
1f115be7 | 1485 | # stap sleeptime.stp -c "sleep 1" |
cf29c85e FCE |
1486 | |
1487 | ||
0e81b7f5 FCE |
1488 | = NET = |
1489 | ||
1490 | lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo) | |
1491 | keywords: net socket | |
1492 | ||
1493 | This traces socket duration from the accept() syscall to close(), and | |
1494 | provides details on the lifespan of these passive connections, | |
1495 | showing the distribution as a histogram. | |
1496 | ||
1497 | # stap accept2close-nd.stp -c "sleep 1" | |
1498 | ||
1499 | ||
210ff7bf FCE |
1500 | = NETWORK = |
1501 | ||
cf29c85e | 1502 | network/autofs4.stp - Watch autofs4 Operations |
8b88e771 FCE |
1503 | keywords: network autofs nfs |
1504 | ||
1505 | Trace key autofs4 operations such as mounting or unmounting remote | |
1506 | filesystems. | |
1507 | ||
6a51196c FCE |
1508 | # stap autofs4.stp -c "sleep 1" |
1509 | ||
8b88e771 | 1510 | |
9e749e50 JS |
1511 | network/connect_stat.stp - Show Process Ancestry for IP Connections |
1512 | keywords: network socket process | |
1513 | ||
1514 | The connect_stat.stp script prints a task's entire ancestry (parent | |
1515 | process name/uid/gid) whenever it attempts an outgoing socket | |
1516 | connection to a given IP address. | |
1517 | ||
1f115be7 | 1518 | # stap connect_stat.stp 127.0.0.1 -c "sleep 1" |
9e749e50 JS |
1519 | |
1520 | ||
cf29c85e FCE |
1521 | network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel |
1522 | keywords: network tracepoint socket | |
0e4901b0 WC |
1523 | |
1524 | Every five seconds the dropwatch.stp script lists the number of | |
1525 | socket buffers freed at locations in the kernel. | |
1526 | ||
1f115be7 | 1527 | # stap dropwatch.stp -c "sleep 1" |
6a51196c | 1528 | |
0e4901b0 | 1529 | |
a89c2e98 FL |
1530 | network/net_xmit_json.stp - Tracks time between packet queue and transmit. |
1531 | keywords: network statistics json | |
1532 | ||
1533 | This script tracks time between packet queue and transmit. The | |
1534 | information is provided to userspace via procfs in JSON format. | |
1535 | ||
1536 | # stap net_xmit_json.stp -c "sleep 1" | |
1537 | ||
1538 | ||
53e5699f JS |
1539 | network/netdev.stp - Trace Activity on Network Devices |
1540 | keywords: network device traffic | |
1541 | ||
1542 | The netdev.stp script traces configuration and transmit/receive | |
1543 | activity on network devices. | |
1544 | ||
1f115be7 | 1545 | # stap netdev.stp -c "sleep 1" |
6a51196c | 1546 | |
53e5699f | 1547 | |
f1c6c6ed | 1548 | network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool |
1ca36a99 | 1549 | keywords: network packets guru |
f1c6c6ed MW |
1550 | |
1551 | The script drops the specified number of packets of the specified | |
1552 | protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, | |
1553 | all incoming packets are dropped. The number of packets to drop can | |
1554 | be specified with a positive integer. A value of 0 indicates that | |
1555 | packets should be dropped until the user manually exits. | |
1556 | ||
1557 | # stap -g netfilter_drop.stp TCP 1 -c "sleep 2" | |
1558 | ||
1559 | ||
57db0e6f | 1560 | network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs |
b9ae0bc4 | 1561 | keywords: _best network traffic |
57db0e6f MW |
1562 | |
1563 | The script watches all IPv4 network traffic on the system. On exit | |
1564 | the script prints a list showing the number of packets sent along | |
1565 | source IP address / destination IP address pair encountered, and the | |
1566 | total number of bytes sent among the pair. The list is ordered from | |
1567 | greatest to least number of packets seen among the source/destination | |
1568 | pairs. | |
1569 | ||
1f115be7 | 1570 | # stap netfilter_summary.stp -c "sleep 1" |
57db0e6f MW |
1571 | |
1572 | ||
a89c2e98 FL |
1573 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs |
1574 | keywords: _best network traffic json | |
1575 | ||
1576 | The script watches all IPv4 network traffic on the system. The data | |
1577 | is output in JSON format and includes the number of packets sent | |
1578 | along source IP address / destination IP address pair encountered, | |
1579 | and the total number of bytes sent among the pair. | |
1580 | ||
1581 | # stap netfilter_summary_json.stp -c "sleep 1" | |
1582 | ||
1583 | ||
210ff7bf | 1584 | network/nettop.stp - Periodic Listing of Processes Using Network Interfaces |
74e5c5cf | 1585 | keywords: network traffic |
210ff7bf FCE |
1586 | |
1587 | Every five seconds the nettop.stp script prints out a list of | |
1588 | processed (PID and command) with the number of packets sent/received | |
1589 | and the amount of data sent/received by the process during that | |
1590 | interval. | |
1591 | ||
1f115be7 | 1592 | # stap nettop.stp -c "sleep 1" |
6a51196c | 1593 | |
210ff7bf | 1594 | |
905728a0 | 1595 | network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space |
cf29c85e | 1596 | keywords: network tcp process |
905728a0 WC |
1597 | |
1598 | The sk_stream-wait_memory.stp prints a time stamp, executable, and | |
1599 | pid each time a process blocks due to the send buffer being full. A | |
1600 | similar entry is printed each time a process continues because there | |
1601 | is room in the buffer. | |
1602 | ||
1f115be7 | 1603 | # stap sk_stream_wait_memory.stp -c "sleep 1" |
6a51196c | 1604 | |
905728a0 | 1605 | |
cf29c85e | 1606 | network/socket-trace.stp - Trace Functions Called in Network Socket Code |
09a98ae0 | 1607 | keywords: network socket |
210ff7bf | 1608 | |
f3c4da44 MW |
1609 | The script instruments each of the functions in the Linux kernel's |
1610 | net/socket.c file. The script prints out trace data. The first | |
1611 | element of a line is time delta in microseconds from the previous | |
1612 | entry. This is followed by the command name and the PID. The "->" and | |
1613 | "<-" indicates function entry and function exit, respectively. The | |
1614 | last element of the line is the function name. | |
210ff7bf | 1615 | |
1f115be7 | 1616 | # stap socket-trace.stp -c "sleep 1" |
6a51196c | 1617 | |
210ff7bf | 1618 | |
9141644d WC |
1619 | network/socktop - Periodically Summarize Socket Activity on the System |
1620 | keywords: network socket | |
1621 | ||
1622 | The socktop script periodically prints out a list of the processes | |
1623 | with the highest socket activity. Command line options for the | |
1624 | script allow filtering to focus on particular types of sockets. The | |
1625 | "-h" option lists socktop script's filtering options. | |
1626 | ||
6a51196c FCE |
1627 | # ./socktop -c 1 |
1628 | ||
9141644d | 1629 | |
a89c2e98 FL |
1630 | network/stp_dump.stp - Dump of STP packets |
1631 | keywords: network traffic | |
1632 | ||
1633 | The stp_dump.stp prints out the packet contents. Each block contains | |
1634 | the STP protocol ID, version ID, flags, root and bridge MAC | |
1635 | addresses, and various times. | |
1636 | ||
9e6206fd | 1637 | # stap stp_dump.stp -c "sleep 5" |
a89c2e98 FL |
1638 | |
1639 | ||
492d227f WC |
1640 | network/tcp_connections.stp - Track Creation of Incoming TCP Connections |
1641 | keywords: network tcp socket | |
1642 | ||
1643 | The tcp_connections.stp script prints information for each new | |
1644 | incoming TCP connection accepted by the computer. The information | |
1645 | includes the UID, the command accepting the connection, the PID of | |
1646 | the command, the port the connection is on, and the IP address of the | |
1647 | originator of the request. | |
1648 | ||
1f115be7 | 1649 | # stap tcp_connections.stp -c "sleep 1" |
6a51196c | 1650 | |
492d227f | 1651 | |
cf29c85e | 1652 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 |
1ca36a99 | 1653 | keywords: network tcp socket guru |
65bdbf09 FCE |
1654 | |
1655 | Run the tcp_init_cwnd.stp script in the background to override a | |
1656 | kernel's default tcp cwnd value to 10, which has been found to | |
1657 | improve latency for web server type workloads. The script prints a | |
1658 | count of cwnd value changes when it is stopped. | |
1659 | ||
1f115be7 | 1660 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
65bdbf09 FCE |
1661 | |
1662 | ||
cf29c85e | 1663 | network/tcp_trace.stp - TCP Connection Tracing Utility |
2e251678 DW |
1664 | keywords: network trace |
1665 | ||
cf29c85e | 1666 | This scripts traces a given TCP connection based on the filter |
2e251678 DW |
1667 | parameters given by the user. The indexing is done by the 4 tuples |
1668 | local address, remote address, local port, remote port. | |
1669 | ||
6a51196c FCE |
1670 | # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1 |
1671 | ||
2e251678 | 1672 | |
5cf50309 | 1673 | network/tcpdumplike.stp - Dump of Received UDP/TCP Packets |
7a51212c WC |
1674 | keywords: network traffic |
1675 | ||
5cf50309 FCE |
1676 | The tcpdumplike.stp prints out a line for each TCP & UDP packet |
1677 | received. Each line includes the source and destination IP addresses, | |
1678 | the source and destination ports, and flags. | |
7a51212c | 1679 | |
1f115be7 | 1680 | # stap tcpdumplike.stp -c "sleep 1" |
6a51196c | 1681 | |
7a51212c | 1682 | |
cf29c85e | 1683 | network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets |
4bb6522c WC |
1684 | keywords: network statistics |
1685 | ||
cf29c85e FCE |
1686 | The tcpipstat script collects and displays network statistics related |
1687 | to individual TCP sockets or groups of sockets. The statistics that | |
1688 | are collected are simular to that of the command netstat -s, only | |
1689 | sorted and grouped by individual sockets. | |
4bb6522c | 1690 | |
6a51196c FCE |
1691 | # stap tcpipstat.stp timeout=1 |
1692 | ||
4bb6522c | 1693 | |
8b88e771 FCE |
1694 | = NFS = |
1695 | ||
cf29c85e | 1696 | network/autofs4.stp - Watch autofs4 Operations |
8b88e771 FCE |
1697 | keywords: network autofs nfs |
1698 | ||
1699 | Trace key autofs4 operations such as mounting or unmounting remote | |
1700 | filesystems. | |
1701 | ||
6a51196c FCE |
1702 | # stap autofs4.stp -c "sleep 1" |
1703 | ||
8b88e771 | 1704 | |
d12c1a86 FCE |
1705 | network/nfsd-recent.stp - Keep track of NFS server statistics |
1706 | keywords: nfs statistics | |
1707 | ||
1708 | This script tracks all nfsd server operations by client_ip address, | |
1709 | and periodically lists those clients that have made recent requests. | |
1710 | It's a way of finding out which nfs clients might be considered still | |
1711 | connected. | |
1712 | ||
1f115be7 | 1713 | # stap nfsd-recent.stp -c "sleep 1" |
d12c1a86 FCE |
1714 | |
1715 | ||
cf29c85e | 1716 | network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server |
b7edd338 WC |
1717 | keywords: nfs disk |
1718 | ||
1719 | The nfsd_unlink.stp script lists the ip address and file name each | |
1720 | time time a file is being removed or unlinked by the nfsd. This | |
1721 | script is run on the nfs server. | |
1722 | ||
1f115be7 | 1723 | # stap nfsd_unlink.stp -c "sleep 1" |
b7edd338 WC |
1724 | |
1725 | ||
5882f170 DS |
1726 | network/nfsdtop.stp - Keep track of NFS server statistics |
1727 | keywords: nfs statistics | |
1728 | ||
1729 | The nfsdtop.stp script gathers and displays NFS lookups, | |
1730 | ||
3b33d8c4 | 1731 | # stap nfsdtop.stp -c "sleep 1" |
5882f170 DS |
1732 | |
1733 | ||
c728b7da WC |
1734 | = NUMA = |
1735 | ||
1736 | memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes | |
1737 | keywords: memory numa | |
1738 | ||
1739 | The numa_faults.stp script tracks the read and write pages faults for | |
1740 | each process. When the script exits it prints out the total read and | |
ad7e33d7 | 1741 | write pages faults for each process. The script also provide a break |
c728b7da WC |
1742 | down of page faults per node for each process. This script is useful |
1743 | for determining whether the program has good locality (page faults | |
1744 | limited to a single node) on a NUMA computer. | |
1745 | ||
1f115be7 | 1746 | # stap numa_faults.stp -c "sleep 1" |
6a51196c | 1747 | |
c728b7da | 1748 | |
f1c6c6ed MW |
1749 | = PACKETS = |
1750 | ||
1751 | network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool | |
1ca36a99 | 1752 | keywords: network packets guru |
f1c6c6ed MW |
1753 | |
1754 | The script drops the specified number of packets of the specified | |
1755 | protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, | |
1756 | all incoming packets are dropped. The number of packets to drop can | |
1757 | be specified with a positive integer. A value of 0 indicates that | |
1758 | packets should be dropped until the user manually exits. | |
1759 | ||
1760 | # stap -g netfilter_drop.stp TCP 1 -c "sleep 2" | |
1761 | ||
1762 | ||
cf29c85e | 1763 | = PROCESS = |
db5bfb14 | 1764 | |
cf29c85e | 1765 | general/eventcount.stp - Count Specified Events |
b9ae0bc4 | 1766 | keywords: _best statistics thread process |
db5bfb14 | 1767 | |
875e1b27 FCE |
1768 | The script periodically prints a count of specified events and their |
1769 | related tid's over the course of execution. Numerous configuration | |
1770 | options exist to control filtering / reporting, see the script | |
1771 | source. | |
db5bfb14 | 1772 | |
1f115be7 | 1773 | # stap eventcount.stp syscall.* -c 'sleep 1' |
db5bfb14 FCE |
1774 | |
1775 | ||
c4abda95 SP |
1776 | memory/glibc-malloc.stp - Overview glibc malloc internal operations |
1777 | keywords: memory process | |
1778 | ||
1779 | This script reports on internal statistics of the glibc malloc | |
1780 | implementation, as used by a process restricted by stap -x/-c | |
1781 | ||
1782 | # stap glibc-malloc.stp -c 'stap --dump-functions' | |
1783 | ||
1784 | ||
f70a0bc5 FCE |
1785 | memory/last_100_frees.stp - Log recent free(3) calls. |
1786 | keywords: memory process backtrace | |
1787 | ||
1788 | This script reports on the last few free(3) libc calls done by | |
1789 | processes (possibly restricted by stap -x/-c), along with a userspace | |
1790 | backtrace at those moments. | |
1791 | ||
1792 | # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd | |
1793 | ||
1794 | ||
9e749e50 JS |
1795 | network/connect_stat.stp - Show Process Ancestry for IP Connections |
1796 | keywords: network socket process | |
1797 | ||
1798 | The connect_stat.stp script prints a task's entire ancestry (parent | |
1799 | process name/uid/gid) whenever it attempts an outgoing socket | |
1800 | connection to a given IP address. | |
1801 | ||
1f115be7 | 1802 | # stap connect_stat.stp 127.0.0.1 -c "sleep 1" |
9e749e50 JS |
1803 | |
1804 | ||
cf29c85e FCE |
1805 | network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space |
1806 | keywords: network tcp process | |
1ff1a65d | 1807 | |
cf29c85e FCE |
1808 | The sk_stream-wait_memory.stp prints a time stamp, executable, and |
1809 | pid each time a process blocks due to the send buffer being full. A | |
1810 | similar entry is printed each time a process continues because there | |
1811 | is room in the buffer. | |
1ff1a65d | 1812 | |
1f115be7 | 1813 | # stap sk_stream_wait_memory.stp -c "sleep 1" |
6a51196c | 1814 | |
1ff1a65d | 1815 | |
cf29c85e | 1816 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 1817 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
1818 | |
1819 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
1820 | to determine which processes and interrupts are competing with the | |
1821 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
1822 | options to focus on a specific task. The script output the number of | |
1823 | times the task migrates between processors, histograms showing the | |
1824 | length of time on and off processor, lists of processes running while | |
1825 | the task is off the processor, and the interrupts that occurred while | |
1826 | the task was running. | |
1827 | ||
1f115be7 | 1828 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 1829 | |
e01bc08e | 1830 | |
cf29c85e | 1831 | process/errsnoop.stp - Tabulate System Call Errors |
cc20d853 ET |
1832 | keywords: process syscall |
1833 | ||
cf29c85e FCE |
1834 | Prints a periodic tabular report about failing system calls, by |
1835 | process and by syscall failure. The first optional argument | |
1836 | specifies the reporting interval (in seconds, default 5); the second | |
1837 | optional argument gives a screen height (number of lines in the | |
1838 | report, default 20). | |
cc20d853 | 1839 | |
6a51196c FCE |
1840 | # stap errsnoop.stp 1 10 -c "sleep 1" |
1841 | ||
cc20d853 | 1842 | |
e6b653c8 WC |
1843 | process/forktracker.stp - Trace Creation of Processes |
1844 | keywords: process scheduler | |
1845 | ||
1846 | The forktracker.stp script prints out a time-stamped entry showing | |
f3c4da44 | 1847 | each fork and exec operation on the machine. This can be useful to |
e6b653c8 WC |
1848 | determine what process is creating a flurry of short-lived processes. |
1849 | ||
1f115be7 | 1850 | # stap forktracker.stp -c "sleep 1" |
6a51196c | 1851 | |
e6b653c8 | 1852 | |
abb4cfbb FCE |
1853 | process/ltrace.stp - uprobes-based ltrace |
1854 | keywords: process | |
1855 | ||
1856 | The ltrace.stp script lists calls that the designated process makes | |
1857 | through PLTs (procedure linkage tables), generally into shared | |
1858 | libraries. | |
1859 | ||
1860 | # stap ltrace.stp -c ls || echo PR14738 | |
1861 | ||
1862 | ||
cf29c85e | 1863 | process/noptrace.stp - Disable ptrace from Hierarchies of Processes |
1ca36a99 | 1864 | keywords: process security guru |
5481d677 | 1865 | |
cf29c85e FCE |
1866 | Blocks ptrace(2) attempts from processes identified by stap -c/-x, as |
1867 | also specifiable from /proc/systemtap/stap_XXX/ control files. | |
1868 | Processes may be added or removed from the blocked list. | |
5481d677 | 1869 | |
15ecb1fe | 1870 | # stap -g noptrace.stp -c 'strace ls || true' |
6a51196c | 1871 | |
5481d677 | 1872 | |
cf29c85e FCE |
1873 | process/pfiles.stp - Print Process File Descriptors |
1874 | keywords: process file | |
ecf33ff4 ET |
1875 | |
1876 | Run pfiles.stp to produce a human-readable summary of all open file | |
1877 | descriptors of a given process. Specify the process-id as -x PID for | |
1878 | fastest performance. | |
1879 | ||
6a51196c FCE |
1880 | # stap -g pfiles.stp -x $$ |
1881 | ||
ecf33ff4 | 1882 | |
cf29c85e | 1883 | process/plimit.stp - Print Resource Limits of Process |
5b8642a2 FCE |
1884 | keywords: process |
1885 | ||
1886 | The script prints a variety of resource limits for a given pid, like | |
1887 | /proc/$$/limits on recent kernels. | |
1888 | ||
6a51196c FCE |
1889 | # stap -g plimit.stp $$ |
1890 | ||
5b8642a2 | 1891 | |
677b471e JL |
1892 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
1893 | keywords: process monitoring syscall tracepoint | |
1894 | ||
1895 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
1896 | exit(), init_module(), and delete_module(). Event-specific details | |
1897 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
1898 | script does not require debuginfo. | |
1899 | ||
1f115be7 | 1900 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
1901 | |
1902 | ||
944d282f AJ |
1903 | process/proctop.stp - Periodically Print Process Information With History |
1904 | keywords: process scheduler _best | |
1905 | ||
1906 | Every 5 seconds, print out a list of 25 processes that took the most | |
1907 | system time with information about the processes. Includes | |
1908 | information on processes that may have exited while the script was | |
1909 | running. The script contains configuration options listed in the | |
1910 | script source. | |
1911 | ||
1912 | # stap proctop.stp -c "sleep 1" | |
1913 | ||
1914 | ||
f1c6c6ed | 1915 | process/psig.stp - Print Process File Descriptors |
1ca36a99 | 1916 | keywords: process signals |
f1c6c6ed MW |
1917 | |
1918 | Run psig.stp to produce a human-readable summary of the signal | |
1919 | handling configuration of a given process. Specify the process-id as | |
1920 | -x PID for fastest performance. | |
1921 | ||
1922 | # stap -DMAXACTION=10000 -g psig.stp -x $$ | |
1923 | ||
1924 | ||
04c36a84 MW |
1925 | process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands |
1926 | keywords: process backtrace | |
1927 | ||
1928 | The pstrace_exec.stp script watches each exec operation. If the exec | |
1929 | contains a substring that matches the script's command-line argument, | |
1930 | it prints out that process and all of its ancestors. | |
1931 | ||
1f115be7 | 1932 | # stap pstrace_exec.stp -c "sleep 1" bash |
04c36a84 MW |
1933 | |
1934 | ||
9e749e50 JS |
1935 | process/pstree.stp - Generates a process diagram in DOT form. |
1936 | keywords: process diagram | |
1937 | ||
1938 | The pstree.stp script generates a process diagram in DOT form. For | |
1939 | instance, it may be useful on a 'make' command to see all the | |
1940 | processes that are started. | |
1941 | ||
1f115be7 | 1942 | # stap pstree.stp -c "sleep 1" |
9e749e50 JS |
1943 | |
1944 | ||
cf29c85e | 1945 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
1946 | keywords: process scheduler time tracepoint |
1947 | ||
1948 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
1949 | amount of time that each process spends in running, sleeping, |
1950 | queuing, and waiting for io. On exit the script prints out the | |
1951 | accumulated time for each state of processes observed. Optionally, | |
1952 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 1953 | specific PID and its children. |
34029cd3 | 1954 | |
1f115be7 | 1955 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 1956 | |
34029cd3 | 1957 | |
5cf50309 FCE |
1958 | process/semop-watch.stp - Watch semop(2)/semtimedop(2) operations |
1959 | keywords: process locking | |
1960 | ||
1961 | Prints a timed trace of semop(2)/semtimedop(2) syscalls | |
1962 | ||
1963 | # stap semop-watch.stp -c 'sleep 2' | |
1964 | ||
1965 | ||
b9ae0bc4 FCE |
1966 | process/spawn_seeker.stp - Track Creation of Processes by process and execname |
1967 | keywords: process scheduler | |
1968 | ||
1969 | The spawn_seeker.stp script every minute (and on exit) prints out the | |
1970 | local time and sorted lists of which processes and executables | |
1971 | spawned tasks during the previous minute. This can be useful to | |
1972 | determine what process is creating a flurry of short-lived processes. | |
1973 | When a process exits its count of tasks created is added to its | |
1974 | parent's count to better account for the indirect task creation by | |
1975 | children processes. For more detailed examination of task creation | |
1976 | consider using forktracker.stp. | |
1977 | ||
1978 | # stap spawn_seeker.stp -c "sleep 1" | |
1979 | ||
1980 | ||
5cf50309 | 1981 | process/strace.stp - Trace system calls |
b9ae0bc4 | 1982 | keywords: _best process syscall |
5cf50309 FCE |
1983 | |
1984 | The script loosely emulates strace, when applied to individual | |
1985 | processes or hierarchies (via -c/-x), or the entire system (without | |
1986 | -c/-x). A few output configuration parameters may be set with -G. | |
1987 | ||
1f115be7 | 1988 | # stap strace.stp -c "sleep 1" |
5cf50309 FCE |
1989 | |
1990 | ||
c29c6f83 | 1991 | process/thread-business.stp - monitor syscall history |
b9ae0bc4 | 1992 | keywords: _best process syscall |
c29c6f83 FCE |
1993 | |
1994 | Prints a periodic tabular report about the counts of syscall activity | |
1995 | of all threads on the system, along with a textual | |
1996 | recent-syscall-history for each | |
1997 | ||
1998 | # stap thread-business.stp -c "sleep 10" | |
1999 | ||
2000 | ||
74e5c5cf FCE |
2001 | process/wait4time.stp - Trace Time Spent in wait4 Syscalls |
2002 | keywords: syscall process | |
2003 | ||
2004 | The script watches each wait4 syscall on the system. At the end of | |
2005 | each wait4 syscall the script prints out a line with a timestamp in | |
2006 | microseconds, the pid, the executable name in parentheses, the | |
2007 | "wait4:" key, the duration of the wait and the PID that the wait4 was | |
2008 | waiting for. If the waited for PID is not specified , it is "-1". | |
2009 | ||
1f115be7 | 2010 | # stap wait4time.stp -c "sleep 1" |
74e5c5cf FCE |
2011 | |
2012 | ||
210ff7bf FCE |
2013 | = PROFILING = |
2014 | ||
a4f3198f WC |
2015 | io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device |
2016 | keywords: io profiling | |
2017 | ||
2018 | The iodevstats.stp script measures the amount of data successfully | |
2019 | read and written by all the executables for each io device on the | |
2020 | system. The output is sorted from greatest sum of bytes read and | |
2021 | written to a device by an executable to the least. The output | |
2022 | contains device major/minor number, the count of operations (reads | |
2023 | and writes), the totals and averages for the number of bytes read and | |
2024 | written. | |
2025 | ||
1f115be7 | 2026 | # stap iodevstats.stp -c "sleep 1" |
6a51196c | 2027 | |
a4f3198f | 2028 | |
cf29c85e | 2029 | io/iostat-scsi.stp - IO Statistics for SCSI Devices |
54ff5e0c FCE |
2030 | keywords: io profiling scsi |
2031 | ||
2032 | The iostat-scsi.stp script provides a breakdown of the number of blks | |
ad7e33d7 | 2033 | read and written on the machine's various SCSI devices. The script |
54ff5e0c FCE |
2034 | takes one argument which is the number of seconds between reports. |
2035 | ||
1f115be7 | 2036 | # stap -g iostat-scsi.stp 1 -c "sleep 1" |
6a51196c | 2037 | |
54ff5e0c | 2038 | |
3e4444ed WC |
2039 | io/iostats.stp - List Executables Reading and Writing the Most Data |
2040 | keywords: io profiling | |
2041 | ||
2042 | The iostat.stp script measures the amount of data successfully read | |
2043 | and written by all the executables on the system. The output is | |
2044 | sorted from most greatest sum of bytes read and written by an | |
2045 | executable to the least. The output contains the count of operations | |
2046 | (opens, reads, and writes), the totals and averages for the number of | |
2047 | bytes read and written. | |
2048 | ||
1f115be7 | 2049 | # stap iostats.stp -c "sleep 1" |
6a51196c | 2050 | |
3e4444ed | 2051 | |
cf29c85e FCE |
2052 | io/iotime.stp - Trace Time Spent in Read and Write for Files |
2053 | keywords: profiling syscall io file | |
2054 | ||
2055 | The script watches each open, close, read, and write syscalls on the | |
2056 | system. For each file the scripts observes opened it accumulates the | |
2057 | amount of wall clock time spent in read and write operations and the | |
2058 | number of bytes read and written. When a file is closed the script | |
2059 | prints out a pair of lines for the file. Both lines begin with a | |
2060 | timestamp in microseconds, the PID number, and the executable name in | |
2061 | parentheses. The first line with the "access" keyword lists the file | |
2062 | name, the attempted number of bytes for the read and write | |
2063 | operations. The second line with the "iotime" keyword list the file | |
2064 | name and the number of microseconds accumulated in the read and write | |
2065 | syscalls. | |
2066 | ||
1f115be7 | 2067 | # stap iotime.stp -c "sleep 1" |
cf29c85e FCE |
2068 | |
2069 | ||
111dd9ac WC |
2070 | io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process |
2071 | keywords: io profiling | |
2072 | ||
2073 | The nfs_func_users.stp script counts the uses of NFS functions in the | |
2074 | kernel on a per process bases. The output is sorted from the process | |
2075 | with the greatest number of NFS functions called to the least. The | |
2076 | output contains the executable name, the process number, and the | |
2077 | total number of NFS functions called by the process. | |
2078 | ||
1f115be7 | 2079 | # stap nfs_func_users.stp -c "sleep 1" |
6a51196c | 2080 | |
111dd9ac | 2081 | |
b96bcce8 WC |
2082 | profiling/errno.stp - Show Which Processes and System Calls Return Errors Most Frequently |
2083 | keywords: profiling | |
2084 | ||
2085 | On exit the errno.stp script provides a sorted list showing which | |
2086 | combination of PID, system call, and error occur most frequently. | |
2087 | ||
1f115be7 | 2088 | # stap errno.stp -c "sleep 1" |
6a51196c | 2089 | |
b96bcce8 | 2090 | |
0a9d3bc8 FCE |
2091 | profiling/fileline-profile.stp - Profile Kernel/User Functions |
2092 | keywords: profiling | |
2093 | ||
2094 | The fileline-profile.stp script ends by printing out a sorted list of | |
2095 | the top twenty kernel and/or user processes providing file:line | |
2096 | information, if available, from the samples addresses gathered over | |
2097 | the time period the script is run. Use any of --ldd, --all-modules, | |
2098 | -d MODULE, -d /PATH/TO/EXEC to add more symbolic info. To include the | |
2099 | symbol name in the output, specify guru mode (-g) and add | |
2100 | symbolname="yes" to the stap command. | |
2101 | ||
2102 | # stap fileline-profile.stp -c "sleep 6" --all-modules --ldd | |
2103 | ||
2104 | ||
cf29c85e | 2105 | profiling/fntimes.stp - Show Functions Taking Longer Than Usual |
b9ae0bc4 | 2106 | keywords: _best profiling |
f503d3c0 FCE |
2107 | |
2108 | The fntimes.stp script monitors the execution time history of a given | |
2109 | function family (assumed non-recursive). Each time (beyond a warmup | |
2110 | interval) is then compared to the historical maximum. If it exceeds | |
2111 | a certain threshold (250%), a message is printed. | |
2112 | ||
6a51196c FCE |
2113 | # stap fntimes.stp 'kernel.function("sys_*")' -c "sleep 7" |
2114 | ||
f503d3c0 | 2115 | |
cf29c85e FCE |
2116 | profiling/functioncallcount.stp - Count Times Functions Are Called |
2117 | keywords: profiling function | |
210ff7bf FCE |
2118 | |
2119 | The functioncallcount.stp script takes one argument, a list of | |
2120 | functions to probe. The script will run and count the number of times | |
2121 | that each of the functions on the list is called. On exit the script | |
2122 | will print a sorted list from most frequently to least frequently | |
2123 | called function. | |
2124 | ||
ca5d5a88 | 2125 | # stap -w functioncallcount.stp "*@mm/*.c" -c "sleep 1" |
a4072787 WC |
2126 | |
2127 | ||
cf29c85e | 2128 | profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping |
b9ae0bc4 | 2129 | keywords: _best profiling |
a4072787 WC |
2130 | |
2131 | The latencytap.stp script collects data on the intervals processes | |
2132 | are deactivated (sleeping). The script categorizes the reasons for | |
2133 | the sleeps by analyzing the backtraces and displays a sorted list of | |
2134 | the top 20 causes from largest total sum time sleeping to smallest. | |
2135 | The output is updated every 30 seconds. The script needs to be | |
2136 | compiled with the '--all-modules' option to produce reasons for | |
2137 | sleeps caused by modules. Optionally, this script can be used with | |
2138 | the '-c' or '-x' options to focus on a specific PID. | |
2139 | ||
1f115be7 | 2140 | # stap latencytap.stp --all-modules -c "sleep 1" |
6a51196c | 2141 | |
210ff7bf | 2142 | |
cf29c85e | 2143 | profiling/linetimes.stp - Show Time Spent on Each Line of a Function |
0a9d3bc8 | 2144 | keywords: profiling _best |
e985df18 WC |
2145 | |
2146 | The linetimes.stp script takes two arguments: where to find the | |
2147 | function and the function name. linetimes.stp will instrument each | |
2148 | line in the function. It will print out the number of times that the | |
2149 | function is called, a table with the average and maximum time each | |
2150 | line takes, and control flow information when the script exits. | |
2151 | ||
1f115be7 | 2152 | # stap linetimes.stp kernel sys_nanosleep -c "sleep 1" |
6a51196c | 2153 | |
e985df18 | 2154 | |
eb45f1f9 FCE |
2155 | profiling/perf.stp - Show performance ratios using perf.counter to access performance counters |
2156 | keywords: profiling | |
2157 | ||
2158 | On exit the perf.stp script provides a sorted list showing cycles per | |
2159 | insn, branches per insn, and cache refs per insn | |
2160 | ||
70f3005e | 2161 | # stap -w perf.stp -c "find /usr/bin -name \"l*\" -printf \"%h/%f %s %Cx \ |
eb45f1f9 FCE |
2162 | %Ck%CM %Y\n\"" |
2163 | ||
2164 | ||
cf29c85e | 2165 | profiling/periodic.stp - Show the Period of the Various Timers on the System |
7bec2ae9 WC |
2166 | keywords: profiling |
2167 | ||
2168 | The periodic.stp script uses the kernel.trace("timer_expire_entry") | |
cf29c85e | 2169 | tracepoint to collect data on period and frequency of the various |
7bec2ae9 WC |
2170 | timers on the system. The script displays a sorted list of the |
2171 | timers observed on the system from most frequent to least frequent. | |
2172 | The script needs to be compiled with the '--all-modules' option to | |
2173 | produce list the function names. Optionally, this script can be used | |
2174 | with a numerical argument to indicate the interval in seconds between | |
2175 | printing output. | |
2176 | ||
1f115be7 | 2177 | # stap periodic.stp --all-modules -c "sleep 1" |
7bec2ae9 WC |
2178 | |
2179 | ||
9ebb5a4e FCE |
2180 | profiling/pf2.stp - Profile Kernel Functions |
2181 | keywords: profiling | |
2182 | ||
2183 | The pf2.stp script sets up time-based sampling. Every five seconds it | |
2184 | prints out a sorted list with the top ten kernel functions with | |
2185 | samples. | |
2186 | ||
1f115be7 | 2187 | # stap pf2.stp -c "sleep 1" |
9ebb5a4e FCE |
2188 | |
2189 | ||
d7dd78f0 | 2190 | profiling/pf3.stp - Profile Kernel/User Functions |
9ebb5a4e FCE |
2191 | keywords: profiling |
2192 | ||
2193 | The pf3.stp script sets up time-based sampling. Every five seconds it | |
2194 | prints out a sorted list with the top twenty kernel and/or user | |
2195 | functions with samples. Use any of --ldd, --all-modules, -d MODULE, | |
2196 | -d /PATH/TO/EXEC to add more symbolic info. | |
2197 | ||
2198 | # stap pf3.stp -c "sleep 6" --all-modules --ldd | |
2199 | ||
2200 | ||
16fc2650 | 2201 | profiling/pf4.stp - Profile Kernel/User Backtraces |
14a364c1 | 2202 | keywords: _best profiling backtrace |
16fc2650 FCE |
2203 | |
2204 | The pf4.stp script sets up time-based sampling. Every five seconds it | |
2205 | prints out a sorted list with the top twenty kernel and/or user stack | |
2206 | backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d | |
2207 | MODULE, -d /PATH/TO/EXEC to add more symbolic info. | |
2208 | ||
2209 | # stap pf4.stp -c "sleep 6" --all-modules --ldd | |
2210 | ||
2211 | ||
cf29c85e FCE |
2212 | profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler |
2213 | keywords: profiling function | |
0449af03 JS |
2214 | |
2215 | The sched_switch.stp script takes two arguments, first argument can | |
2216 | be "pid" or "name" to indicate what is being passed as second | |
2217 | argument. The script will trace the process based on pid/name and | |
2218 | print the scheduler switches happening with the process. If no | |
2219 | arguments are passed, it displays all the scheduler switches. This | |
2220 | can be used to understand which tasks schedule out the current | |
2221 | process being traced, and when it gets scheduled in again. | |
2222 | ||
6a51196c FCE |
2223 | # stap sched_switch.stp -c "sleep 1" |
2224 | ||
0449af03 | 2225 | |
cf29c85e | 2226 | profiling/thread-times.stp - Profile Kernel Functions |
96ef4c2a | 2227 | keywords: _best profiling |
210ff7bf | 2228 | |
1ae72757 JS |
2229 | The thread-times.stp script sets up time-based sampling. Every five |
2230 | seconds it prints out a sorted list with the top twenty threads | |
2231 | occupying the CPUs, broken down as a percentage of user and kernel | |
2232 | time. | |
210ff7bf | 2233 | |
1f115be7 | 2234 | # stap thread-times.stp -c "sleep 1" |
6a51196c | 2235 | |
210ff7bf | 2236 | |
ff90b297 WC |
2237 | profiling/timeout.stp - Show Processes Doing Polling Operations |
2238 | keywords: profiling | |
2239 | ||
2240 | The timeout.stp script is based on a blog entry | |
2241 | (http://udrepper.livejournal.com/19041.html) mentioning a need for a | |
2242 | tool to help developers find applications that are polling. The | |
2243 | timeout.stp script monitors systemcall used for polling and records | |
2244 | the systemcalls that timed out rather than returned because some | |
2245 | action occurred. The script updates the screen once a second with the | |
2246 | top twenty processes. | |
2247 | ||
1f115be7 | 2248 | # stap timeout.stp -c "sleep 1" |
6a51196c | 2249 | |
ff90b297 | 2250 | |
1bf72dfe WC |
2251 | profiling/topsys.stp - Show Processes Doing Polling Operations |
2252 | keywords: profiling | |
2253 | ||
2254 | The topsys.stp script lists out the top twenty systemcalls for the | |
2255 | previous 5 seconds. The output is sorted from most frequent to least | |
2256 | frequent. | |
2257 | ||
1f115be7 | 2258 | # stap topsys.stp -c "sleep 1" |
6a51196c | 2259 | |
1bf72dfe | 2260 | |
8c6bb289 WC |
2261 | = QEMU = |
2262 | ||
cf29c85e | 2263 | virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events |
8c6bb289 WC |
2264 | keywords: virtualization qemu kvm |
2265 | ||
2266 | The qemu_count.stp script tallies the number of times each of the | |
2267 | user-space qemu probepoints is encountered. When the script exits, it | |
2268 | prints a list of the number of times each user-space qemu probepoint | |
2269 | is encountered. | |
2270 | ||
1f115be7 | 2271 | # stap qemu_count.stp -c "sleep 1" |
8c6bb289 WC |
2272 | |
2273 | ||
cf29c85e | 2274 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
2275 | keywords: virtualization qemu kvm io |
2276 | ||
2277 | The qemu_io.stp script tallies the number of times each of the IO | |
2278 | port on the guest virtual machines is touched by a input or output | |
2279 | operation. When the script exits, it prints a count of the number of | |
2280 | times each IO port read and written. | |
2281 | ||
1f115be7 | 2282 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
2283 | |
2284 | ||
fefcef82 FCE |
2285 | = REGEX = |
2286 | ||
2287 | general/regex.stp - Report opened files whose names match a given regex | |
2288 | keywords: regex | |
2289 | ||
2290 | Uses the regex functionality to detect opened files whose names match | |
2291 | a pattern given on the command line. If no command line parameter is | |
2292 | given, demonstrate by filtering for files that end with an extension | |
2293 | showing them to be an archive. | |
2294 | ||
1f115be7 | 2295 | # stap regex.stp -c "sleep 1" |
fefcef82 FCE |
2296 | |
2297 | ||
210ff7bf FCE |
2298 | = SCHEDULER = |
2299 | ||
deb63545 WC |
2300 | process/chng_cpu.stp - Monitor Changes in Processor Executing a Task |
2301 | keywords: scheduler | |
2302 | ||
2303 | The chng_cpu.stp script takes an argument which is the executable | |
2304 | name of the task it should monitor. Each time a task with that | |
2305 | executable name is found running on a different processor, the script | |
2306 | prints out the thread id (tid), the executable name, the processor | |
2307 | now running the task, the thread state, and a backtrace showing the | |
2308 | kernel functions that triggered the running of the task on the | |
2309 | processor. | |
2310 | ||
1f115be7 | 2311 | # stap chng_cpu.stp -c "sleep 1" bash |
6a51196c | 2312 | |
deb63545 | 2313 | |
cf29c85e | 2314 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 2315 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
2316 | |
2317 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
2318 | to determine which processes and interrupts are competing with the | |
2319 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
2320 | options to focus on a specific task. The script output the number of | |
2321 | times the task migrates between processors, histograms showing the | |
2322 | length of time on and off processor, lists of processes running while | |
2323 | the task is off the processor, and the interrupts that occurred while | |
2324 | the task was running. | |
2325 | ||
1f115be7 | 2326 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 2327 | |
e01bc08e | 2328 | |
e6b653c8 WC |
2329 | process/forktracker.stp - Trace Creation of Processes |
2330 | keywords: process scheduler | |
2331 | ||
2332 | The forktracker.stp script prints out a time-stamped entry showing | |
f3c4da44 | 2333 | each fork and exec operation on the machine. This can be useful to |
e6b653c8 WC |
2334 | determine what process is creating a flurry of short-lived processes. |
2335 | ||
1f115be7 | 2336 | # stap forktracker.stp -c "sleep 1" |
6a51196c | 2337 | |
e6b653c8 | 2338 | |
deb63545 WC |
2339 | process/migrate.stp - Track the Migration of Specific Executables |
2340 | keywords: scheduler | |
2341 | ||
2342 | The migrate.stp script takes an argument which is the executable name | |
2343 | of the task it should monitor. Each time a task with that executable | |
2344 | name migrates between processors an entry is printed with the process | |
2345 | id (pid), the executable name, the processor off loading the task, | |
2346 | and the process taking the task. Note that the task may or may not be | |
2347 | executing at the time of the migration. | |
2348 | ||
1f115be7 | 2349 | # stap migrate.stp -c "sleep 1" bash |
6a51196c | 2350 | |
deb63545 | 2351 | |
944d282f AJ |
2352 | process/proctop.stp - Periodically Print Process Information With History |
2353 | keywords: process scheduler _best | |
2354 | ||
2355 | Every 5 seconds, print out a list of 25 processes that took the most | |
2356 | system time with information about the processes. Includes | |
2357 | information on processes that may have exited while the script was | |
2358 | running. The script contains configuration options listed in the | |
2359 | script source. | |
2360 | ||
2361 | # stap proctop.stp -c "sleep 1" | |
2362 | ||
2363 | ||
cf29c85e | 2364 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
2365 | keywords: process scheduler time tracepoint |
2366 | ||
2367 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
2368 | amount of time that each process spends in running, sleeping, |
2369 | queuing, and waiting for io. On exit the script prints out the | |
2370 | accumulated time for each state of processes observed. Optionally, | |
2371 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 2372 | specific PID and its children. |
34029cd3 | 2373 | |
1f115be7 | 2374 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 2375 | |
34029cd3 | 2376 | |
cf29c85e | 2377 | process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations |
09a98ae0 | 2378 | keywords: io scheduler backtrace |
cae71dd3 | 2379 | |
f3c4da44 | 2380 | The script monitors the time that threads spend in waiting for IO |
cae71dd3 FCE |
2381 | operations (in "D" state) in the wait_for_completion function. If a |
2382 | thread spends over 10ms, its name and backtrace is printed, and later | |
2383 | so is the total delay. | |
210ff7bf | 2384 | |
1f115be7 | 2385 | # stap sleepingBeauties.stp -c "sleep 1" |
6a51196c | 2386 | |
210ff7bf | 2387 | |
b9ae0bc4 FCE |
2388 | process/spawn_seeker.stp - Track Creation of Processes by process and execname |
2389 | keywords: process scheduler | |
2390 | ||
2391 | The spawn_seeker.stp script every minute (and on exit) prints out the | |
2392 | local time and sorted lists of which processes and executables | |
2393 | spawned tasks during the previous minute. This can be useful to | |
2394 | determine what process is creating a flurry of short-lived processes. | |
2395 | When a process exits its count of tasks created is added to its | |
2396 | parent's count to better account for the indirect task creation by | |
2397 | children processes. For more detailed examination of task creation | |
2398 | consider using forktracker.stp. | |
2399 | ||
2400 | # stap spawn_seeker.stp -c "sleep 1" | |
2401 | ||
2402 | ||
54ff5e0c FCE |
2403 | = SCSI = |
2404 | ||
cf29c85e | 2405 | io/iostat-scsi.stp - IO Statistics for SCSI Devices |
54ff5e0c FCE |
2406 | keywords: io profiling scsi |
2407 | ||
2408 | The iostat-scsi.stp script provides a breakdown of the number of blks | |
ad7e33d7 | 2409 | read and written on the machine's various SCSI devices. The script |
54ff5e0c FCE |
2410 | takes one argument which is the number of seconds between reports. |
2411 | ||
1f115be7 | 2412 | # stap -g iostat-scsi.stp 1 -c "sleep 1" |
6a51196c | 2413 | |
54ff5e0c | 2414 | |
5481d677 FCE |
2415 | = SECURITY = |
2416 | ||
7080b0c3 | 2417 | process/auditbt.stp - Generate backtraces for kernel audit events |
a8a7b9f4 FCE |
2418 | keywords: monitoring security audit backtrace |
2419 | ||
7080b0c3 FCE |
2420 | Attaches to the kernel audit-log paths (also used by libaudit), and |
2421 | log every record being sent, along with a user-space backtrace of the | |
2422 | process that caused it. | |
a8a7b9f4 | 2423 | |
54a385da | 2424 | # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true" |
a8a7b9f4 FCE |
2425 | |
2426 | ||
cf29c85e | 2427 | process/noptrace.stp - Disable ptrace from Hierarchies of Processes |
1ca36a99 | 2428 | keywords: process security guru |
5481d677 | 2429 | |
cf29c85e FCE |
2430 | Blocks ptrace(2) attempts from processes identified by stap -c/-x, as |
2431 | also specifiable from /proc/systemtap/stap_XXX/ control files. | |
2432 | Processes may be added or removed from the blocked list. | |
5481d677 | 2433 | |
15ecb1fe | 2434 | # stap -g noptrace.stp -c 'strace ls || true' |
6a51196c | 2435 | |
5481d677 | 2436 | |
1ca36a99 | 2437 | = SIGNALS = |
f1c6c6ed MW |
2438 | |
2439 | process/psig.stp - Print Process File Descriptors | |
1ca36a99 | 2440 | keywords: process signals |
f1c6c6ed MW |
2441 | |
2442 | Run psig.stp to produce a human-readable summary of the signal | |
2443 | handling configuration of a given process. Specify the process-id as | |
2444 | -x PID for fastest performance. | |
2445 | ||
2446 | # stap -DMAXACTION=10000 -g psig.stp -x $$ | |
2447 | ||
2448 | ||
210ff7bf | 2449 | process/sig_by_pid.stp - Signal Counts by Process ID |
09a98ae0 | 2450 | keywords: signals |
210ff7bf FCE |
2451 | |
2452 | Print signal counts by process ID in descending order. | |
2453 | ||
1f115be7 | 2454 | # stap sig_by_pid.stp -c "sleep 1" |
6a51196c | 2455 | |
210ff7bf FCE |
2456 | |
2457 | process/sig_by_proc.stp - Signal Counts by Process Name | |
09a98ae0 | 2458 | keywords: signals |
210ff7bf FCE |
2459 | |
2460 | Print signal counts by process name in descending order. | |
2461 | ||
1f115be7 | 2462 | # stap sig_by_proc.stp -c "sleep 1" |
6a51196c | 2463 | |
210ff7bf FCE |
2464 | |
2465 | process/sigkill.stp - Track SIGKILL Signals | |
09a98ae0 | 2466 | keywords: signals |
210ff7bf FCE |
2467 | |
2468 | The script traces any SIGKILL signals. When that SIGKILL signal is | |
2469 | sent to a process, the script prints out the signal name, the | |
f3c4da44 MW |
2470 | destination executable and process ID, the executable name and user |
2471 | ID that sents the signal. | |
210ff7bf | 2472 | |
1f115be7 | 2473 | # stap sigkill.stp -c "sleep 1" |
6a51196c | 2474 | |
210ff7bf | 2475 | |
cf29c85e | 2476 | process/sigmon.stp - Track a Particular Signal to a Specific Process |
09a98ae0 | 2477 | keywords: signals |
210ff7bf FCE |
2478 | |
2479 | The script watches for a particular signal sent to a specific | |
2480 | process. When that signal is sent to the specified process, the | |
2481 | script prints out the PID and executable of the process sending the | |
2482 | signal, the PID and executable name of the process receiving the | |
2483 | signal, and the signal number and name. | |
2484 | ||
1f115be7 | 2485 | # stap sigmon.stp -c "sleep 1" SIGKILL |
6a51196c | 2486 | |
210ff7bf FCE |
2487 | |
2488 | = SIMPLE = | |
2489 | ||
2490 | general/helloworld.stp - SystemTap "Hello World" Program | |
b9ae0bc4 | 2491 | keywords: _best simple |
210ff7bf FCE |
2492 | |
2493 | A basic "Hello World" program implemented in SystemTap script. It | |
2494 | prints out "hello world" message and then immediately exits. | |
2495 | ||
6a51196c FCE |
2496 | # stap helloworld.stp |
2497 | ||
210ff7bf | 2498 | |
7955f6fd FCE |
2499 | general/py2example.stp - SystemTap python 2 support tapset |
2500 | keywords: simple | |
2501 | ||
2502 | A python support tapset that displays backtraces and variable values | |
2503 | ||
2504 | # stap -I tapset -c '/usr/bin/python2 pyexample.py 35' py2example.stp | |
2505 | ||
2506 | ||
2507 | general/py3example.stp - SystemTap python 3 support tapset | |
2508 | keywords: simple | |
2509 | ||
2510 | A python support tapset that displays backtraces and variable values | |
2511 | ||
2512 | # stap -g --suppress-time-limits -I tapset -c '/usr/bin/python3 \ | |
2513 | pyexample.py 35' py3example.stp | |
2514 | ||
2515 | ||
47300c64 | 2516 | io/eatmydata.stp - disable fsync |
1ca36a99 FCE |
2517 | keywords: io guru simple |
2518 | ||
2519 | Suppresses fsync() syscalls from processes identified by stap -c/-x | |
00b943ad FCE |
2520 | by turning them into presumed-faster fsync() on some dummy or other |
2521 | file descriptor | |
1ca36a99 FCE |
2522 | |
2523 | # stap -g eatmydata.stp -c 'strace ls || true' | |
2524 | ||
2525 | ||
42c55668 MW |
2526 | = SLAB = |
2527 | ||
cf29c85e FCE |
2528 | memory/vm.tracepoints.stp - Collect Slab Allocation Statistics |
2529 | keywords: memory slab statistics | |
42c55668 MW |
2530 | |
2531 | The script will probe all memory slab/slub allocations and collects | |
2532 | information about the size of the object (bytes requested) and | |
2533 | user-space process in execution. When run over a period of time, it | |
2534 | helps to correlate kernel-space memory consumption owing to | |
2535 | user-space processes. | |
2536 | ||
6a51196c FCE |
2537 | # stap vm.tracepoints.stp -c "sleep 10" |
2538 | ||
42c55668 | 2539 | |
cf29c85e | 2540 | = SOCKET = |
210ff7bf | 2541 | |
0e81b7f5 FCE |
2542 | lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo) |
2543 | keywords: net socket | |
2544 | ||
2545 | This traces socket duration from the accept() syscall to close(), and | |
2546 | provides details on the lifespan of these passive connections, | |
2547 | showing the distribution as a histogram. | |
2548 | ||
2549 | # stap accept2close-nd.stp -c "sleep 1" | |
2550 | ||
2551 | ||
9e749e50 JS |
2552 | network/connect_stat.stp - Show Process Ancestry for IP Connections |
2553 | keywords: network socket process | |
2554 | ||
2555 | The connect_stat.stp script prints a task's entire ancestry (parent | |
2556 | process name/uid/gid) whenever it attempts an outgoing socket | |
2557 | connection to a given IP address. | |
2558 | ||
1f115be7 | 2559 | # stap connect_stat.stp 127.0.0.1 -c "sleep 1" |
9e749e50 JS |
2560 | |
2561 | ||
cf29c85e FCE |
2562 | network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel |
2563 | keywords: network tracepoint socket | |
210ff7bf | 2564 | |
cf29c85e FCE |
2565 | Every five seconds the dropwatch.stp script lists the number of |
2566 | socket buffers freed at locations in the kernel. | |
6a51196c | 2567 | |
1f115be7 | 2568 | # stap dropwatch.stp -c "sleep 1" |
210ff7bf | 2569 | |
210ff7bf | 2570 | |
cf29c85e | 2571 | network/socket-trace.stp - Trace Functions Called in Network Socket Code |
09a98ae0 | 2572 | keywords: network socket |
210ff7bf | 2573 | |
f3c4da44 MW |
2574 | The script instruments each of the functions in the Linux kernel's |
2575 | net/socket.c file. The script prints out trace data. The first | |
2576 | element of a line is time delta in microseconds from the previous | |
2577 | entry. This is followed by the command name and the PID. The "->" and | |
2578 | "<-" indicates function entry and function exit, respectively. The | |
2579 | last element of the line is the function name. | |
210ff7bf | 2580 | |
1f115be7 | 2581 | # stap socket-trace.stp -c "sleep 1" |
6a51196c | 2582 | |
210ff7bf | 2583 | |
9141644d WC |
2584 | network/socktop - Periodically Summarize Socket Activity on the System |
2585 | keywords: network socket | |
2586 | ||
2587 | The socktop script periodically prints out a list of the processes | |
2588 | with the highest socket activity. Command line options for the | |
2589 | script allow filtering to focus on particular types of sockets. The | |
2590 | "-h" option lists socktop script's filtering options. | |
2591 | ||
6a51196c FCE |
2592 | # ./socktop -c 1 |
2593 | ||
9141644d | 2594 | |
492d227f WC |
2595 | network/tcp_connections.stp - Track Creation of Incoming TCP Connections |
2596 | keywords: network tcp socket | |
2597 | ||
2598 | The tcp_connections.stp script prints information for each new | |
2599 | incoming TCP connection accepted by the computer. The information | |
2600 | includes the UID, the command accepting the connection, the PID of | |
2601 | the command, the port the connection is on, and the IP address of the | |
2602 | originator of the request. | |
2603 | ||
1f115be7 | 2604 | # stap tcp_connections.stp -c "sleep 1" |
6a51196c | 2605 | |
492d227f | 2606 | |
cf29c85e | 2607 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 |
1ca36a99 | 2608 | keywords: network tcp socket guru |
65bdbf09 FCE |
2609 | |
2610 | Run the tcp_init_cwnd.stp script in the background to override a | |
2611 | kernel's default tcp cwnd value to 10, which has been found to | |
2612 | improve latency for web server type workloads. The script prints a | |
2613 | count of cwnd value changes when it is stopped. | |
2614 | ||
1f115be7 | 2615 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
65bdbf09 FCE |
2616 | |
2617 | ||
75ed3b3c FCE |
2618 | = SPECULATION = |
2619 | ||
2620 | general/whythefail.stp - Why did the function fail? | |
2621 | keywords: speculation monitoring function trace _best | |
2622 | ||
2623 | The whythefail.stp script prints a statement-execution trace for a | |
2624 | given function, but only for those runs of the function that ended up | |
2625 | with a (configurable) post-return condition. | |
2626 | ||
2627 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
2628 | /root/no-such-file || true' | |
2629 | ||
2630 | ||
875e1b27 FCE |
2631 | = STAPGAMES = |
2632 | ||
7955f6fd | 2633 | stapgames/2048.stp - 2048 |
b9ae0bc4 | 2634 | keywords: _best stapgames |
7955f6fd FCE |
2635 | |
2636 | The modern classic 2048 sliding-tiles game, using local keyboard and | |
2637 | ansi animation. | |
2638 | ||
2639 | # stap -p4 2048.stp | |
2640 | ||
2641 | ||
875e1b27 FCE |
2642 | stapgames/block.stp - block breaker game |
2643 | keywords: stapgames | |
2644 | ||
2645 | A block game where you progressively break the ceiling blocks until | |
2646 | clearing the level | |
2647 | ||
2648 | # stap -p4 -Itapset/ block.stp | |
2649 | ||
2650 | ||
2651 | stapgames/eater.stp - eater game | |
2652 | keywords: stapgames | |
2653 | ||
2654 | walk through a maze, eat stuff | |
2655 | ||
2656 | # stap -p4 -Itapset/ eater.stp | |
2657 | ||
2658 | ||
2659 | stapgames/lifegame.stp - life game | |
2660 | keywords: stapgames | |
2661 | ||
2662 | watch as your creation morphes into different forms | |
2663 | ||
2664 | # stap -p4 -Itapset/ lifegame.stp | |
2665 | ||
2666 | ||
2667 | stapgames/pingpong.stp - pingpong game | |
2668 | keywords: stapgames | |
2669 | ||
2670 | A simulated ball bounces around the terminal reflecting at the edges | |
2671 | ||
2672 | # stap -p4 -Itapset/ pingpong.stp | |
2673 | ||
2674 | ||
4bb6522c WC |
2675 | = STATISTICS = |
2676 | ||
beb59e38 FCE |
2677 | general/alias_suffixes.stp - Count I/O Syscalls using Alias Suffixes |
2678 | keywords: io statistics | |
2679 | ||
2680 | alias_suffixes.stp is a demonstration of how alias suffixes in the | |
2681 | systemtap language might be used. The script tracks the wall clock | |
2682 | time for each invocation of the system calls open, close, read, and | |
2683 | write. When the script exists it prints out the minimum, average, and | |
2684 | maximum times in microseconds for each system call, followed by a | |
2685 | count of times that each syscall was invoked and a histogram showing | |
2686 | the distributions of times. | |
2687 | ||
1f115be7 | 2688 | # stap alias_suffixes.stp -c "sleep 1" |
beb59e38 FCE |
2689 | |
2690 | ||
cf29c85e | 2691 | general/eventcount.stp - Count Specified Events |
b9ae0bc4 | 2692 | keywords: _best statistics thread process |
cf29c85e | 2693 | |
875e1b27 FCE |
2694 | The script periodically prints a count of specified events and their |
2695 | related tid's over the course of execution. Numerous configuration | |
2696 | options exist to control filtering / reporting, see the script | |
2697 | source. | |
cf29c85e | 2698 | |
1f115be7 | 2699 | # stap eventcount.stp syscall.* -c 'sleep 1' |
cf29c85e FCE |
2700 | |
2701 | ||
2702 | general/func_time_stats.stp - Function Time Statistics | |
2703 | keywords: function statistics | |
2704 | ||
2705 | The func_time_stats.stp script tracks the wall clock time for each | |
2706 | invocation of the function probe listed as the first command line | |
2707 | argument. When the script exits it prints out the minimum, average, | |
2708 | and maximum times in microseconds followed by a count of times that | |
2709 | the function was called and a histogram showing the distributions of | |
2710 | times. | |
2711 | ||
1f115be7 | 2712 | # stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1" |
cf29c85e FCE |
2713 | |
2714 | ||
2715 | general/sizeof.stp - Print the Size of a C Type | |
2716 | keywords: statistics memory | |
3cfd512b FCE |
2717 | |
2718 | This script prints the size of a type, based on dwarf debuginfo for | |
2719 | any kernel or userspace module, or trial-compilation of a given | |
2720 | header file name. | |
2721 | ||
6a51196c FCE |
2722 | # stap sizeof.stp FILE '</usr/include/stdio.h>' |
2723 | ||
3cfd512b | 2724 | |
cf29c85e FCE |
2725 | memory/vm.tracepoints.stp - Collect Slab Allocation Statistics |
2726 | keywords: memory slab statistics | |
2727 | ||
2728 | The script will probe all memory slab/slub allocations and collects | |
2729 | information about the size of the object (bytes requested) and | |
2730 | user-space process in execution. When run over a period of time, it | |
2731 | helps to correlate kernel-space memory consumption owing to | |
2732 | user-space processes. | |
2733 | ||
2734 | # stap vm.tracepoints.stp -c "sleep 10" | |
2735 | ||
2736 | ||
a89c2e98 FL |
2737 | network/net_xmit_json.stp - Tracks time between packet queue and transmit. |
2738 | keywords: network statistics json | |
2739 | ||
2740 | This script tracks time between packet queue and transmit. The | |
2741 | information is provided to userspace via procfs in JSON format. | |
2742 | ||
2743 | # stap net_xmit_json.stp -c "sleep 1" | |
2744 | ||
2745 | ||
d12c1a86 FCE |
2746 | network/nfsd-recent.stp - Keep track of NFS server statistics |
2747 | keywords: nfs statistics | |
2748 | ||
2749 | This script tracks all nfsd server operations by client_ip address, | |
2750 | and periodically lists those clients that have made recent requests. | |
2751 | It's a way of finding out which nfs clients might be considered still | |
2752 | connected. | |
2753 | ||
1f115be7 | 2754 | # stap nfsd-recent.stp -c "sleep 1" |
d12c1a86 FCE |
2755 | |
2756 | ||
5882f170 DS |
2757 | network/nfsdtop.stp - Keep track of NFS server statistics |
2758 | keywords: nfs statistics | |
2759 | ||
2760 | The nfsdtop.stp script gathers and displays NFS lookups, | |
2761 | ||
3b33d8c4 | 2762 | # stap nfsdtop.stp -c "sleep 1" |
5882f170 DS |
2763 | |
2764 | ||
cf29c85e | 2765 | network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets |
4bb6522c WC |
2766 | keywords: network statistics |
2767 | ||
cf29c85e FCE |
2768 | The tcpipstat script collects and displays network statistics related |
2769 | to individual TCP sockets or groups of sockets. The statistics that | |
2770 | are collected are simular to that of the command netstat -s, only | |
2771 | sorted and grouped by individual sockets. | |
4bb6522c | 2772 | |
6a51196c FCE |
2773 | # stap tcpipstat.stp timeout=1 |
2774 | ||
4bb6522c | 2775 | |
210ff7bf FCE |
2776 | = SYSCALL = |
2777 | ||
2778 | io/iotime.stp - Trace Time Spent in Read and Write for Files | |
cf29c85e | 2779 | keywords: profiling syscall io file |
210ff7bf FCE |
2780 | |
2781 | The script watches each open, close, read, and write syscalls on the | |
2782 | system. For each file the scripts observes opened it accumulates the | |
f3c4da44 | 2783 | amount of wall clock time spent in read and write operations and the |
210ff7bf FCE |
2784 | number of bytes read and written. When a file is closed the script |
2785 | prints out a pair of lines for the file. Both lines begin with a | |
2786 | timestamp in microseconds, the PID number, and the executable name in | |
ad7e33d7 | 2787 | parentheses. The first line with the "access" keyword lists the file |
210ff7bf FCE |
2788 | name, the attempted number of bytes for the read and write |
2789 | operations. The second line with the "iotime" keyword list the file | |
2790 | name and the number of microseconds accumulated in the read and write | |
2791 | syscalls. | |
2792 | ||
1f115be7 | 2793 | # stap iotime.stp -c "sleep 1" |
6a51196c | 2794 | |
210ff7bf | 2795 | |
cf29c85e | 2796 | process/errsnoop.stp - Tabulate System Call Errors |
cc20d853 ET |
2797 | keywords: process syscall |
2798 | ||
cf29c85e FCE |
2799 | Prints a periodic tabular report about failing system calls, by |
2800 | process and by syscall failure. The first optional argument | |
2801 | specifies the reporting interval (in seconds, default 5); the second | |
2802 | optional argument gives a screen height (number of lines in the | |
2803 | report, default 20). | |
cc20d853 | 2804 | |
6a51196c FCE |
2805 | # stap errsnoop.stp 1 10 -c "sleep 1" |
2806 | ||
cc20d853 | 2807 | |
210ff7bf | 2808 | process/futexes.stp - System-Wide Futex Contention |
09a98ae0 | 2809 | keywords: syscall locking futex |
210ff7bf FCE |
2810 | |
2811 | The script watches the futex syscall on the system. On exit the | |
0fb74e68 | 2812 | futex's address, the number of contentions, and the average time for |
210ff7bf FCE |
2813 | each contention on the futex are printed from lowest pid number to |
2814 | highest. | |
2815 | ||
1f115be7 | 2816 | # stap futexes.stp -c "sleep 1" |
6a51196c | 2817 | |
210ff7bf | 2818 | |
0fb74e68 JS |
2819 | process/futexes2.stp - System-Wide Shared Futex Contention |
2820 | keywords: syscall locking futex | |
2821 | ||
2822 | The script watches just shared futex syscalls on the system. On exit | |
2823 | the futex's key, the number of contentions, and the average time for | |
2824 | each contention on the futex are printed from lowest pid number to | |
2825 | highest. | |
2826 | ||
1f115be7 | 2827 | # stap futexes2.stp -c "sleep 1" |
0fb74e68 JS |
2828 | |
2829 | ||
677b471e JL |
2830 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
2831 | keywords: process monitoring syscall tracepoint | |
2832 | ||
2833 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
2834 | exit(), init_module(), and delete_module(). Event-specific details | |
2835 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
2836 | script does not require debuginfo. | |
2837 | ||
1f115be7 | 2838 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
2839 | |
2840 | ||
cf29c85e FCE |
2841 | process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls |
2842 | keywords: syscall nanosleep | |
210ff7bf FCE |
2843 | |
2844 | The script watches each nanosleep syscall on the system. At the end | |
2845 | of each nanosleep syscall the script prints out a line with a | |
2846 | timestamp in microseconds, the pid, the executable name in | |
ad7e33d7 | 2847 | parentheses, the "nanosleep:" key, and the duration of the sleep in |
210ff7bf FCE |
2848 | microseconds. |
2849 | ||
1f115be7 | 2850 | # stap sleeptime.stp -c "sleep 1" |
6a51196c | 2851 | |
210ff7bf | 2852 | |
cbaf758e | 2853 | process/strace.stp - Trace system calls |
b9ae0bc4 | 2854 | keywords: _best process syscall |
cbaf758e FCE |
2855 | |
2856 | The script loosely emulates strace, when applied to individual | |
2857 | processes or hierarchies (via -c/-x), or the entire system (without | |
2858 | -c/-x). A few output configuration parameters may be set with -G. | |
2859 | ||
2860 | # stap strace.stp -c "sleep 1" | |
2861 | ||
2862 | ||
210ff7bf | 2863 | process/syscalls_by_pid.stp - System-Wide Count of Syscalls by PID |
09a98ae0 | 2864 | keywords: syscall |
210ff7bf FCE |
2865 | |
2866 | The script watches all syscall on the system. On exit the script | |
2867 | prints a list showing the number of systemcalls executed by each PID | |
2868 | ordered from greatest to least number of syscalls. | |
2869 | ||
1f115be7 | 2870 | # stap syscalls_by_pid.stp -c "sleep 1" |
6a51196c | 2871 | |
210ff7bf FCE |
2872 | |
2873 | process/syscalls_by_proc.stp - System-Wide Count of Syscalls by Executable | |
09a98ae0 | 2874 | keywords: syscall |
210ff7bf FCE |
2875 | |
2876 | The script watches all syscall on the system. On exit the script | |
2877 | prints a list showing the number of systemcalls executed by each | |
ad7e33d7 | 2878 | executable ordered from greatest to least number of syscalls. |
210ff7bf | 2879 | |
1f115be7 | 2880 | # stap syscalls_by_proc.stp -c "sleep 1" |
6a51196c | 2881 | |
210ff7bf | 2882 | |
cf29c85e | 2883 | process/syscalltimes - System-Wide Syscall Statistics with Filtering |
04ae04b5 MW |
2884 | keywords: syscall |
2885 | ||
2886 | Combination shell/systemtap script to measure system call counts and | |
2887 | times. Can be filtered by process IDs, process names and users. | |
2888 | ||
1f115be7 | 2889 | # ./syscalltimes -c 'sleep 1' |
04ae04b5 MW |
2890 | |
2891 | ||
c29c6f83 | 2892 | process/thread-business.stp - monitor syscall history |
b9ae0bc4 | 2893 | keywords: _best process syscall |
c29c6f83 FCE |
2894 | |
2895 | Prints a periodic tabular report about the counts of syscall activity | |
2896 | of all threads on the system, along with a textual | |
2897 | recent-syscall-history for each | |
2898 | ||
2899 | # stap thread-business.stp -c "sleep 10" | |
2900 | ||
2901 | ||
210ff7bf | 2902 | process/wait4time.stp - Trace Time Spent in wait4 Syscalls |
74e5c5cf | 2903 | keywords: syscall process |
210ff7bf FCE |
2904 | |
2905 | The script watches each wait4 syscall on the system. At the end of | |
2906 | each wait4 syscall the script prints out a line with a timestamp in | |
ad7e33d7 | 2907 | microseconds, the pid, the executable name in parentheses, the |
210ff7bf FCE |
2908 | "wait4:" key, the duration of the wait and the PID that the wait4 was |
2909 | waiting for. If the waited for PID is not specified , it is "-1". | |
2910 | ||
1f115be7 | 2911 | # stap wait4time.stp -c "sleep 1" |
6a51196c | 2912 | |
210ff7bf | 2913 | |
492d227f WC |
2914 | = TCP = |
2915 | ||
905728a0 | 2916 | network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space |
cf29c85e | 2917 | keywords: network tcp process |
905728a0 WC |
2918 | |
2919 | The sk_stream-wait_memory.stp prints a time stamp, executable, and | |
2920 | pid each time a process blocks due to the send buffer being full. A | |
2921 | similar entry is printed each time a process continues because there | |
2922 | is room in the buffer. | |
2923 | ||
1f115be7 | 2924 | # stap sk_stream_wait_memory.stp -c "sleep 1" |
6a51196c | 2925 | |
905728a0 | 2926 | |
492d227f WC |
2927 | network/tcp_connections.stp - Track Creation of Incoming TCP Connections |
2928 | keywords: network tcp socket | |
2929 | ||
2930 | The tcp_connections.stp script prints information for each new | |
2931 | incoming TCP connection accepted by the computer. The information | |
2932 | includes the UID, the command accepting the connection, the PID of | |
2933 | the command, the port the connection is on, and the IP address of the | |
2934 | originator of the request. | |
2935 | ||
1f115be7 | 2936 | # stap tcp_connections.stp -c "sleep 1" |
6a51196c | 2937 | |
492d227f | 2938 | |
cf29c85e | 2939 | network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10 |
1ca36a99 | 2940 | keywords: network tcp socket guru |
65bdbf09 FCE |
2941 | |
2942 | Run the tcp_init_cwnd.stp script in the background to override a | |
2943 | kernel's default tcp cwnd value to 10, which has been found to | |
2944 | improve latency for web server type workloads. The script prints a | |
2945 | count of cwnd value changes when it is stopped. | |
2946 | ||
1f115be7 | 2947 | # stap -g tcp_init_cwnd.stp -c "sleep 1" |
65bdbf09 FCE |
2948 | |
2949 | ||
cf29c85e | 2950 | = THREAD = |
db5bfb14 | 2951 | |
cf29c85e | 2952 | general/eventcount.stp - Count Specified Events |
b9ae0bc4 | 2953 | keywords: _best statistics thread process |
db5bfb14 | 2954 | |
875e1b27 FCE |
2955 | The script periodically prints a count of specified events and their |
2956 | related tid's over the course of execution. Numerous configuration | |
2957 | options exist to control filtering / reporting, see the script | |
2958 | source. | |
db5bfb14 | 2959 | |
1f115be7 | 2960 | # stap eventcount.stp syscall.* -c 'sleep 1' |
db5bfb14 FCE |
2961 | |
2962 | ||
875e1b27 | 2963 | process/threadstacks.stp - Override default new-pthread stack sizes |
1ca36a99 | 2964 | keywords: thread guru |
875e1b27 FCE |
2965 | |
2966 | Overrides default NPTL pthread_create stack size for all new threads | |
2967 | created by target processes. Reports one line per process when the | |
2968 | related glibc variable __default_stacksize is updated. Moot for | |
2969 | glibc versions that support $LIBC_PTHREAD_DEFAULT_STACKSIZE_NP. | |
2970 | ||
f70a0bc5 | 2971 | # stap -g threadstacks.stp -Gsize=65536 -c "sleep 1" -d `which stap` |
875e1b27 FCE |
2972 | |
2973 | ||
210ff7bf FCE |
2974 | = TIME = |
2975 | ||
1345075d WC |
2976 | general/stopwatches.stp - See the amount of wall clock time a process spends in various states |
2977 | keywords: time | |
2978 | ||
2979 | The stopwatch.stp script illustrates how to use multiple stopwatches | |
2980 | record how much wallclock time a process spends in kernel- and | |
2981 | user-space. On exit the script prints out the time in seconds, | |
2982 | milliseconds, microseconds, and nanoseconds. Note that this output of | |
2983 | this script is not directly comparable to the time command because | |
2984 | time records the time that the process is actually active in kernel- | |
2985 | and user-space. | |
2986 | ||
1f115be7 | 2987 | # stap stopwatches.stp -c "sleep 1" |
1345075d WC |
2988 | |
2989 | ||
cf29c85e | 2990 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 2991 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
2992 | |
2993 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
2994 | to determine which processes and interrupts are competing with the | |
2995 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
2996 | options to focus on a specific task. The script output the number of | |
2997 | times the task migrates between processors, histograms showing the | |
2998 | length of time on and off processor, lists of processes running while | |
2999 | the task is off the processor, and the interrupts that occurred while | |
3000 | the task was running. | |
3001 | ||
1f115be7 | 3002 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 3003 | |
e01bc08e | 3004 | |
cf29c85e | 3005 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
3006 | keywords: process scheduler time tracepoint |
3007 | ||
3008 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
3009 | amount of time that each process spends in running, sleeping, |
3010 | queuing, and waiting for io. On exit the script prints out the | |
3011 | accumulated time for each state of processes observed. Optionally, | |
3012 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 3013 | specific PID and its children. |
34029cd3 | 3014 | |
1f115be7 | 3015 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 3016 | |
34029cd3 | 3017 | |
210ff7bf FCE |
3018 | = TRACE = |
3019 | ||
cf29c85e | 3020 | general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments |
2515d897 JS |
3021 | keywords: trace callgraph |
3022 | ||
03568589 FCE |
3023 | Print a timed per-thread microsecond-timed callgraph, complete with |
3024 | pretty-printed function parameters and return values. The first | |
3025 | parameter names the function probe points to trace. The optional | |
3026 | second parameter names the probe points for trigger functions, which | |
3027 | acts to enable tracing for only those functions that occur while the | |
3028 | current thread is nested within the trigger. | |
2515d897 | 3029 | |
6a51196c | 3030 | # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 3031 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 3032 | |
2515d897 | 3033 | |
cf29c85e | 3034 | general/para-callgraph.stp - Callgraph Tracing with Arguments |
b9ae0bc4 | 3035 | keywords: _best trace callgraph |
210ff7bf | 3036 | |
03568589 FCE |
3037 | Print a timed per-thread microsecond-timed callgraph, complete with |
3038 | function parameters and return values. The first parameter names the | |
3039 | function probe points to trace. The optional second parameter names | |
3040 | the probe points for trigger functions, which acts to enable tracing | |
3041 | for only those functions that occur while the current thread is | |
3042 | nested within the trigger. | |
210ff7bf | 3043 | |
6a51196c | 3044 | # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \ |
15ecb1fe | 3045 | 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true" |
6a51196c | 3046 | |
210ff7bf | 3047 | |
75ed3b3c FCE |
3048 | general/whythefail.stp - Why did the function fail? |
3049 | keywords: speculation monitoring function trace _best | |
3050 | ||
3051 | The whythefail.stp script prints a statement-execution trace for a | |
3052 | given function, but only for those runs of the function that ended up | |
3053 | with a (configurable) post-return condition. | |
3054 | ||
3055 | # stap whythefail.stp kernel sys_open '$return < 0' -c 'cat \ | |
3056 | /root/no-such-file || true' | |
3057 | ||
3058 | ||
cf29c85e | 3059 | network/tcp_trace.stp - TCP Connection Tracing Utility |
2e251678 DW |
3060 | keywords: network trace |
3061 | ||
cf29c85e | 3062 | This scripts traces a given TCP connection based on the filter |
2e251678 DW |
3063 | parameters given by the user. The indexing is done by the 4 tuples |
3064 | local address, remote address, local port, remote port. | |
3065 | ||
6a51196c FCE |
3066 | # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1 |
3067 | ||
2e251678 | 3068 | |
0e4901b0 WC |
3069 | = TRACEPOINT = |
3070 | ||
cf29c85e FCE |
3071 | network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel |
3072 | keywords: network tracepoint socket | |
0e4901b0 WC |
3073 | |
3074 | Every five seconds the dropwatch.stp script lists the number of | |
3075 | socket buffers freed at locations in the kernel. | |
3076 | ||
1f115be7 | 3077 | # stap dropwatch.stp -c "sleep 1" |
6a51196c | 3078 | |
0e4901b0 | 3079 | |
cf29c85e | 3080 | process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task |
b9ae0bc4 | 3081 | keywords: _best process scheduler time tracepoint interrupt |
e01bc08e WC |
3082 | |
3083 | The cycle_thief.stp script instruments the scheduler and IRQ handler | |
3084 | to determine which processes and interrupts are competing with the | |
3085 | specified task for the cpu cycles. This script uses the '-c' or '-x' | |
3086 | options to focus on a specific task. The script output the number of | |
3087 | times the task migrates between processors, histograms showing the | |
3088 | length of time on and off processor, lists of processes running while | |
3089 | the task is off the processor, and the interrupts that occurred while | |
3090 | the task was running. | |
3091 | ||
1f115be7 | 3092 | # stap cycle_thief.stp -c "sleep 1" |
6a51196c | 3093 | |
e01bc08e | 3094 | |
677b471e JL |
3095 | process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading |
3096 | keywords: process monitoring syscall tracepoint | |
3097 | ||
3098 | The procmod_watcher.stp script monitors calls to fork(), exec(), | |
3099 | exit(), init_module(), and delete_module(). Event-specific details | |
3100 | are also printed out (e.g. for exec(), the file being exec'ed). This | |
3101 | script does not require debuginfo. | |
3102 | ||
1f115be7 | 3103 | # stap procmod_watcher.stp -c "sleep 1" |
677b471e JL |
3104 | |
3105 | ||
cf29c85e | 3106 | process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints |
34029cd3 WC |
3107 | keywords: process scheduler time tracepoint |
3108 | ||
3109 | The schedtimes.stp script instruments the scheduler to track the | |
f3c4da44 MW |
3110 | amount of time that each process spends in running, sleeping, |
3111 | queuing, and waiting for io. On exit the script prints out the | |
3112 | accumulated time for each state of processes observed. Optionally, | |
3113 | this script can be used with the '-c' or '-x' options to focus on a | |
4cba4e80 | 3114 | specific PID and its children. |
34029cd3 | 3115 | |
1f115be7 | 3116 | # stap schedtimes.stp -c "sleep 1" |
6a51196c | 3117 | |
34029cd3 | 3118 | |
210ff7bf FCE |
3119 | = TRAFFIC = |
3120 | ||
53e5699f JS |
3121 | network/netdev.stp - Trace Activity on Network Devices |
3122 | keywords: network device traffic | |
3123 | ||
3124 | The netdev.stp script traces configuration and transmit/receive | |
3125 | activity on network devices. | |
3126 | ||
1f115be7 | 3127 | # stap netdev.stp -c "sleep 1" |
6a51196c | 3128 | |
53e5699f | 3129 | |
57db0e6f | 3130 | network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs |
b9ae0bc4 | 3131 | keywords: _best network traffic |
57db0e6f MW |
3132 | |
3133 | The script watches all IPv4 network traffic on the system. On exit | |
3134 | the script prints a list showing the number of packets sent along | |
3135 | source IP address / destination IP address pair encountered, and the | |
3136 | total number of bytes sent among the pair. The list is ordered from | |
3137 | greatest to least number of packets seen among the source/destination | |
3138 | pairs. | |
3139 | ||
1f115be7 | 3140 | # stap netfilter_summary.stp -c "sleep 1" |
57db0e6f MW |
3141 | |
3142 | ||
a89c2e98 FL |
3143 | network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs |
3144 | keywords: _best network traffic json | |
3145 | ||
3146 | The script watches all IPv4 network traffic on the system. The data | |
3147 | is output in JSON format and includes the number of packets sent | |
3148 | along source IP address / destination IP address pair encountered, | |
3149 | and the total number of bytes sent among the pair. | |
3150 | ||
3151 | # stap netfilter_summary_json.stp -c "sleep 1" | |
3152 | ||
3153 | ||
210ff7bf | 3154 | network/nettop.stp - Periodic Listing of Processes Using Network Interfaces |
74e5c5cf | 3155 | keywords: network traffic |
210ff7bf FCE |
3156 | |
3157 | Every five seconds the nettop.stp script prints out a list of | |
3158 | processed (PID and command) with the number of packets sent/received | |
3159 | and the amount of data sent/received by the process during that | |
3160 | interval. | |
3161 | ||
1f115be7 | 3162 | # stap nettop.stp -c "sleep 1" |
6a51196c | 3163 | |
210ff7bf | 3164 | |
a89c2e98 FL |
3165 | network/stp_dump.stp - Dump of STP packets |
3166 | keywords: network traffic | |
3167 | ||
3168 | The stp_dump.stp prints out the packet contents. Each block contains | |
3169 | the STP protocol ID, version ID, flags, root and bridge MAC | |
3170 | addresses, and various times. | |
3171 | ||
9e6206fd | 3172 | # stap stp_dump.stp -c "sleep 5" |
a89c2e98 FL |
3173 | |
3174 | ||
5cf50309 | 3175 | network/tcpdumplike.stp - Dump of Received UDP/TCP Packets |
7a51212c WC |
3176 | keywords: network traffic |
3177 | ||
5cf50309 FCE |
3178 | The tcpdumplike.stp prints out a line for each TCP & UDP packet |
3179 | received. Each line includes the source and destination IP addresses, | |
3180 | the source and destination ports, and flags. | |
7a51212c | 3181 | |
1f115be7 | 3182 | # stap tcpdumplike.stp -c "sleep 1" |
6a51196c | 3183 | |
7a51212c | 3184 | |
b7f6cfc5 FCE |
3185 | = TTY = |
3186 | ||
cf29c85e | 3187 | io/ttyspy.stp - Monitor TTY Typing |
1ca36a99 | 3188 | keywords: io tty monitoring guru |
b7f6cfc5 FCE |
3189 | |
3190 | The ttyspy.stp script uses tty_audit hooks to monitor recent typing | |
3191 | activity on the system, printing a scrolling record of recent | |
3192 | keystrokes, on a per-tty basis. | |
3193 | ||
6a51196c FCE |
3194 | # stap --skip-badvars -g ttyspy.stp -c "sleep 1" |
3195 | ||
b7f6cfc5 | 3196 | |
cf29c85e | 3197 | = UTILIZATION = |
210ff7bf FCE |
3198 | |
3199 | general/graphs.stp - Graphing Disk and CPU Utilization | |
cf29c85e | 3200 | keywords: disk cpu utilization |
210ff7bf FCE |
3201 | |
3202 | The script tracks the disk and CPU utilization. The resulting output | |
3203 | of the script can be piped into gnuplot to generate a graph of disk | |
3204 | and CPU USE. | |
3205 | ||
1f115be7 | 3206 | # stap graphs.stp -c "sleep 1" |
6a51196c | 3207 | |
210ff7bf | 3208 | |
8c6bb289 WC |
3209 | = VIRTUALIZATION = |
3210 | ||
cf29c85e | 3211 | virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons |
b9ae0bc4 | 3212 | keywords: _best virtualization kvm |
98ec0359 FCE |
3213 | |
3214 | The kvm_service_time.stp script tracks the statistics about the | |
3215 | amount of time that the processor left the guest virtual machine for | |
3216 | each exit reason (for example fixing up a page table or handling an | |
3217 | IO operation). When the script exits it prints out the number of | |
3218 | times each exit reason was encountered, the total duration of time it | |
3219 | left the guest VM, the minimum time, the average time, and the | |
3220 | maximum time in microseconds for that exit reason. On Linux 2.6.38 | |
3221 | and newer kernel the script can automatically determine whether it is | |
3222 | running on Intel or AMD processors. For older kernels with a | |
3223 | kernel.trace("kvm_exit") tracepoint that does not have the $isa | |
3224 | parameter you can explicitly state the kvm type with a "-G kvm=intel" | |
3225 | or "-G kvm=amd" on the command line. | |
3226 | ||
1f115be7 | 3227 | # stap kvm_service_time.stp -c "sleep 1" |
98ec0359 FCE |
3228 | |
3229 | ||
cf29c85e | 3230 | virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events |
8c6bb289 WC |
3231 | keywords: virtualization qemu kvm |
3232 | ||
3233 | The qemu_count.stp script tallies the number of times each of the | |
3234 | user-space qemu probepoints is encountered. When the script exits, it | |
3235 | prints a list of the number of times each user-space qemu probepoint | |
3236 | is encountered. | |
3237 | ||
1f115be7 | 3238 | # stap qemu_count.stp -c "sleep 1" |
8c6bb289 WC |
3239 | |
3240 | ||
cf29c85e | 3241 | virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port |
8c6bb289 WC |
3242 | keywords: virtualization qemu kvm io |
3243 | ||
3244 | The qemu_io.stp script tallies the number of times each of the IO | |
3245 | port on the guest virtual machines is touched by a input or output | |
3246 | operation. When the script exits, it prints a count of the number of | |
3247 | times each IO port read and written. | |
3248 | ||
1f115be7 | 3249 | # stap qemu_io.stp -c "sleep 1" |
8c6bb289 WC |
3250 | |
3251 | ||
2dfeb3bf WC |
3252 | = WATCHDOG = |
3253 | ||
cf29c85e | 3254 | general/watchdog.stp - Watchdog Timer for Arbitrary Events |
2dfeb3bf WC |
3255 | keywords: watchdog backtrace |
3256 | ||
3257 | The watchdog.stp script provides a watchdog timer mechanism for | |
3258 | arbitrary events. The script takes three arguments: the events to | |
3259 | start watchdog timer, the event to stop the watchdog timer, and the | |
3260 | time in millseconds for the watchdog. If the watchdog timer is | |
3261 | exceed, the script will trigger a stack backtrace of the user-process | |
3262 | that timed out using pstack. This script can be used to diagnose what | |
3263 | the userspace application is doing when a slower than expected | |
3264 | operation occurs. | |
3265 | ||
3266 | # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \ | |
1f115be7 | 3267 | -c "sleep 1" |
2dfeb3bf WC |
3268 | |
3269 | ||
54d84647 WC |
3270 | = WATCHPOINT = |
3271 | ||
cf29c85e | 3272 | memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware |
54d84647 WC |
3273 | keywords: memory watchpoint |
3274 | ||
3275 | The script will watch accesses to a single kernel address and prints | |
3276 | a traceback each time the address is accessed. This script needs to | |
3277 | be run as root to allow access to the breakpoint hardware. | |
3278 | ||
23063de1 | 3279 | # stap --all-modules hw_watch_addr.stp 0x`grep "vm_dirty_ratio" \ |
0a9d3bc8 | 3280 | /proc/kallsyms | awk '{print $1}'` -c "sleep 5" |
54d84647 WC |
3281 | |
3282 | ||
cf29c85e | 3283 | memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware |
54d84647 WC |
3284 | keywords: memory watchpoint |
3285 | ||
3286 | The script will watch accesses to the starting address of a single | |
3287 | kernel symbol and prints a traceback each time the symbol is | |
3288 | accessed. This script needs to be run as root to allow access to the | |
3289 | breakpoint hardware. | |
3290 | ||
23063de1 | 3291 | # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -c "sleep 5" |
54d84647 WC |
3292 | |
3293 |