--- /dev/null
+# stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$
+
+cannot touch `/tmp/myXXXbadnameXXXfile.15686': Permission denied
--- /dev/null
+# stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1"
+
+OUT intervals min:1000199us avg:1000199us max:1000199us count:1
+ value |-------------------------------------------------- count
+ 131072 | 0
+ 262144 | 0
+ 524288 |@ 1
+1048576 | 0
+2097152 | 0
--- /dev/null
+% stap sizeof.stp FILE '</usr/include/stdio.h>'
+
+type FILE in </usr/include/stdio.h> byte-size: 148
<li><a href="#virtualization/kvm_service_time.stp">virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons</a></li>
</ul>
-<h2>All 127 Examples</h2>
+<h2>All 137 Examples</h2>
<ul>
<li><a name="general/alias_suffixes.stp"></a><a href="#general/alias_suffixes.stp">¶</a> <a href="general/alias_suffixes.stp">general/alias_suffixes.stp</a> - Count I/O Syscalls using Alias Suffixes<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
<p>The script prints a table showing the available attributes (bold, underline, and inverse) with color combinations for the ans_set_color3() function in the ansi.stp tapset.</p><p><font size="-2"><pre># stap ansi_colors2.stp</pre></font></p>
</li><li><a name="general/badname.stp"></a><a href="#general/badname.stp">¶</a> <a href="general/badname.stp">general/badname.stp</a> - Bad Filename Filter<br>
keywords: <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The badname.stp script shows how one could prevent the creation of files with undesirable names using guru mode.</p><p><font size="-2"><pre># stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \
-denied"</pre></font></p>
+<p>The badname.stp script shows how one could prevent the creation of files with undesirable names using guru mode.</p><p><i><a href="general/badname.txt">sample usage in general/badname.txt</i></font></p>
</li><li><a name="general/eventcount.stp"></a><a href="#general/eventcount.stp">¶</a> <a href="general/eventcount.stp">general/eventcount.stp</a> - Count Specified Events<br>
keywords: <a href="keyword-index.html#_BEST">_BEST</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <a href="keyword-index.html#THREAD">THREAD</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <br>
<p>The script periodically prints a count of specified events and their related tid's over the course of execution. Numerous configuration options exist to control filtering / reporting, see the script source.</p><p><i><a href="general/eventcount.txt">sample usage in general/eventcount.txt</i></font></p>
</li><li><a name="general/func_time_stats.stp"></a><a href="#general/func_time_stats.stp">¶</a> <a href="general/func_time_stats.stp">general/func_time_stats.stp</a> - Function Time Statistics<br>
keywords: <a href="keyword-index.html#FUNCTION">FUNCTION</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The func_time_stats.stp script tracks the wall clock time for each invocation of the function probe listed as the first command line argument. When the script exits it prints out the minimum, average, and maximum times in microseconds followed by a count of times that the function was called and a histogram showing the distributions of times.</p><p><font size="-2"><pre># stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1"</pre></font></p>
+<p>The func_time_stats.stp script tracks the wall clock time for each invocation of the function probe listed as the first command line argument. When the script exits it prints out the minimum, average, and maximum times in microseconds followed by a count of times that the function was called and a histogram showing the distributions of times.</p><p><i><a href="general/func_time_stats.txt">sample usage in general/func_time_stats.txt</i></font></p>
</li><li><a name="general/graphs.stp"></a><a href="#general/graphs.stp">¶</a> <a href="general/graphs.stp">general/graphs.stp</a> - Graphing Disk and CPU Utilization<br>
keywords: <a href="keyword-index.html#DISK">DISK</a> <a href="keyword-index.html#CPU">CPU</a> <a href="keyword-index.html#UTILIZATION">UTILIZATION</a> <br>
<p>The script tracks the disk and CPU utilization. The resulting output of the script can be piped into gnuplot to generate a graph of disk and CPU USE.</p><p><font size="-2"><pre># stap graphs.stp -c "sleep 1"</pre></font></p>
<p>Uses the regex functionality to detect opened files whose names match a pattern given on the command line. If no command line parameter is given, demonstrate by filtering for files that end with an extension showing them to be an archive.</p><p><font size="-2"><pre># stap regex.stp -c "sleep 1"</pre></font></p>
</li><li><a name="general/sizeof.stp"></a><a href="#general/sizeof.stp">¶</a> <a href="general/sizeof.stp">general/sizeof.stp</a> - Print the Size of a C Type<br>
keywords: <a href="keyword-index.html#STATISTICS">STATISTICS</a> <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
-<p>This script prints the size of a type, based on dwarf debuginfo for any kernel or userspace module, or trial-compilation of a given header file name.</p><p><font size="-2"><pre># stap sizeof.stp FILE '</usr/include/stdio.h>'</pre></font></p>
+<p>This script prints the size of a type, based on dwarf debuginfo for any kernel or userspace module, or trial-compilation of a given header file name.</p><p><i><a href="general/sizeof.txt">sample usage in general/sizeof.txt</i></font></p>
</li><li><a name="general/stopwatches.stp"></a><a href="#general/stopwatches.stp">¶</a> <a href="general/stopwatches.stp">general/stopwatches.stp</a> - See the amount of wall clock time a process spends in various states<br>
keywords: <a href="keyword-index.html#TIME">TIME</a> <br>
<p>The stopwatch.stp script illustrates how to use multiple stopwatches record how much wallclock time a process spends in kernel- and user-space. On exit the script prints out the time in seconds, milliseconds, microseconds, and nanoseconds. Note that this output of this script is not directly comparable to the time command because time records the time that the process is actually active in kernel- and user-space.</p><p><font size="-2"><pre># stap stopwatches.stp -c "sleep 1"</pre></font></p>
<p>The ioblktime.stp script tracks the amount of time that each block IO requests spend waiting for completion. The script computes the average waiting time for block IO per device and prints list every 10 seconds. In some cases there can be too many outstanding block IO operations and the script may exceed the default number of MAXMAPENTRIES allowed. In this case the allowed number can be increased with "-DMAXMAPENTRIES=10000" option on the stap command line.</p><p><font size="-2"><pre># stap ioblktime.stp -c "sleep 1"</pre></font></p>
</li><li><a name="io/iodevstats.stp"></a><a href="#io/iodevstats.stp">¶</a> <a href="io/iodevstats.stp">io/iodevstats.stp</a> - List Executables Reading and Writing the Most Data by Device<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p> The iodevstats.stp script measures the amount of data successfully read and written by all the executables for each io device on the system. The output is sorted from greatest sum of bytes read and written to a device by an executable to the least. The output contains device major/minor number, the count of operations (reads and writes), the totals and averages for the number of bytes read and written.</p><p><font size="-2"><pre># stap iodevstats.stp -c "sleep 1"</pre></font></p>
+<p> The iodevstats.stp script measures the amount of data successfully read and written by all the executables for each io device on the system. The output is sorted from greatest sum of bytes read and written to a device by an executable to the least. The output contains device major/minor number, the count of operations (reads and writes), the totals and averages for the number of bytes read and written.</p><p><i><a href="io/iodevstats.txt">sample usage in io/iodevstats.txt</i></font></p>
</li><li><a name="io/iostat-scsi.stp"></a><a href="#io/iostat-scsi.stp">¶</a> <a href="io/iostat-scsi.stp">io/iostat-scsi.stp</a> - IO Statistics for SCSI Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#SCSI">SCSI</a> <br>
<p>The iostat-scsi.stp script provides a breakdown of the number of blks read and written on the machine's various SCSI devices. The script takes one argument which is the number of seconds between reports.</p><p><i><a href="io/iostat-scsi.txt">sample usage in io/iostat-scsi.txt</i></font></p>
</li><li><a name="io/iostats.stp"></a><a href="#io/iostats.stp">¶</a> <a href="io/iostats.stp">io/iostats.stp</a> - List Executables Reading and Writing the Most Data<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p> The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written.</p><p><font size="-2"><pre># stap iostats.stp -c "sleep 1"</pre></font></p>
+<p> The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written.</p><p><i><a href="io/iostats.txt">sample usage in io/iostats.txt</i></font></p>
</li><li><a name="io/iotime.stp"></a><a href="#io/iotime.stp">¶</a> <a href="io/iotime.stp">io/iotime.stp</a> - Trace Time Spent in Read and Write for Files <br>
keywords: <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#SYSCALL">SYSCALL</a> <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILE">FILE</a> <br>
<p>The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spent in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parentheses. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls.</p><p><font size="-2"><pre># stap iotime.stp -c "sleep 1"</pre></font></p>
<p>Every five seconds print out the top ten executables generating I/O traffic during that interval sorted in descending order.</p><p><font size="-2"><pre># stap iotop.stp -c "sleep 1"</pre></font></p>
</li><li><a name="io/mbrwatch.stp"></a><a href="#io/mbrwatch.stp">¶</a> <a href="io/mbrwatch.stp">io/mbrwatch.stp</a> - Monitor Read/Write of the Boot Sector Area of Block Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#DISK">DISK</a> <br>
-<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><font size="-2"><pre># stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
-/proc/partitions | grep . | awk '{print $4}' | head -1`"</pre></font></p>
+<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><i><a href="io/mbrwatch.txt">sample usage in io/mbrwatch.txt</i></font></p>
</li><li><a name="io/nfs_func_users.stp"></a><a href="#io/nfs_func_users.stp">¶</a> <a href="io/nfs_func_users.stp">io/nfs_func_users.stp</a> - Tally the Number of NFS Functions Used by Each Process<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
<p>The nfs_func_users.stp script counts the uses of NFS functions in the kernel on a per process bases. The output is sorted from the process with the greatest number of NFS functions called to the least. The output contains the executable name, the process number, and the total number of NFS functions called by the process.</p><p><font size="-2"><pre># stap nfs_func_users.stp -c "sleep 1"</pre></font></p>
<p>Print out the executable name and process number as reads and writes to the specified device occur.</p><p><font size="-2"><pre># stap traceio2.stp 0x0801 -c "sleep 1"</pre></font></p>
</li><li><a name="io/ttyspy.stp"></a><a href="#io/ttyspy.stp">¶</a> <a href="io/ttyspy.stp">io/ttyspy.stp</a> - Monitor TTY Typing<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#TTY">TTY</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><font size="-2"><pre># stap --skip-badvars -g ttyspy.stp -c "sleep 1"</pre></font></p>
+<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><i><a href="io/ttyspy.txt">sample usage in io/ttyspy.txt</i></font></p>
</li><li><a name="locks/bkl.stp"></a><a href="#locks/bkl.stp">¶</a> <a href="locks/bkl.stp">locks/bkl.stp</a> - Tracing Contention on Big Kernel Lock<br>
keywords: <a href="keyword-index.html#LOCKING">LOCKING</a> <br>
<p>The bkl.stp script can help determine whether the Big Kernel Lock (BKL) is causing serialization on a multiprocessor system due to excessive contention of the BKL. The bkl.stp script takes one argument which is the number of processes waiting for the Big Kernel Lock (BKL). When the number of processes waiting for the BKL is reached or exceeded, the script will print a time stamp, the number of processes waiting for the BKL, the holder of the BKL, and the amount of time the BKL was held.</p><p><font size="-2"><pre># stap bkl.stp -c "sleep 1" 1</pre></font></p>
</li><li><a name="locks/bkl_stats.stp"></a><a href="#locks/bkl_stats.stp">¶</a> <a href="locks/bkl_stats.stp">locks/bkl_stats.stp</a> - Per Process Statistics on Big Kernel Lock Use<br>
keywords: <a href="keyword-index.html#LOCKING">LOCKING</a> <br>
<p>The bkl_stats.stp script can indicate which processes have excessive waits for the Big Kernel Lock (BKL) and which processes are taking the BKL for long periods of time. The bkl_stats.stp script prints lists of all the processes that require the BKL. Every five seconds two tables are printed out. The first table lists the processes that waited for the BKL followed by the number of times that the process waited, the minimum time of the wait, the average and the maximum time waited. The second table lists has similar information for the time spent in holding the lock for each of the processes.</p><p><font size="-2"><pre># stap bkl_stats.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/accept2close-nd.stp"></a><a href="#lwtools/accept2close-nd.stp">¶</a> <a href="lwtools/accept2close-nd.stp">lwtools/accept2close-nd.stp</a> - Show socket lifespan, from accept() to close() (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#NET">NET</a> <a href="keyword-index.html#SOCKET">SOCKET</a> <br>
+<p>This traces socket duration from the accept() syscall to close(), and provides details on the lifespan of these passive connections, showing the distribution as a histogram.</p><p><i><a href="lwtools/accept2close-nd.txt">sample usage in lwtools/accept2close-nd.txt</i></font></p>
+</li><li><a name="lwtools/biolatency-nd.stp"></a><a href="#lwtools/biolatency-nd.stp">¶</a> <a href="lwtools/biolatency-nd.stp">lwtools/biolatency-nd.stp</a> - Measure block I/O latency distribution (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This measures block I/O latency (storage I/O, ie, disk I/O), and shows the distribution as a histogram. This can be useful to identify the characteristics of I/O latency, beyond the averages shown by iostat(1). For example, to study I/O latency outliers, or multi-modal distributions.</p><p><font size="-2"><pre># stap biolatency-nd.stp 1 1</pre></font></p>
+</li><li><a name="lwtools/bitesize-nd.stp"></a><a href="#lwtools/bitesize-nd.stp">¶</a> <a href="lwtools/bitesize-nd.stp">lwtools/bitesize-nd.stp</a> - Measure block I/O size distribution (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This uses the kernel tracepoint block_rq_insert to read the size of I/O. The output includes the name of the process or thread that was on-CPU when the I/O request was inserted on the issue queue.</p><p><font size="-2"><pre># stap bitesize-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/execsnoop-nd.stp"></a><a href="#lwtools/execsnoop-nd.stp">¶</a> <a href="lwtools/execsnoop-nd.stp">lwtools/execsnoop-nd.stp</a> - Trace process exec() with command line argument details (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This can identify if CPU is consumed by short-lived processes, by tracing new process execution. It works by tracing exec() from the fork()->exec() sequence, which means it will not catch new processes that only fork(). It will also show every exec(), including those if a process re-execs.</p><p><font size="-2"><pre># stap execsnoop-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/fslatency-nd.stp"></a><a href="#lwtools/fslatency-nd.stp">¶</a> <a href="lwtools/fslatency-nd.stp">lwtools/fslatency-nd.stp</a> - Measure the distribution of file system synchronous read and write latency (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
+<p>This dynamically traces two common file system functions: do_sync_read() and do_sync_write(), and reports a histogram distribution of latency. Many, but not all, file systems and workloads use these functions. Tracing their time provides one view of suffered file system latency.</p><p><font size="-2"><pre># stap fslatency-nd.stp 1 1</pre></font></p>
+</li><li><a name="lwtools/fsslower-nd.stp"></a><a href="#lwtools/fsslower-nd.stp">¶</a> <a href="lwtools/fsslower-nd.stp">lwtools/fsslower-nd.stp</a> - Trace slow file system synchronous reads and writes (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
+<p>This dynamically traces two common file system functions: do_sync_read() and do_sync_write(), and shows details of each call that is slower than a threshold. Many, but not all, file systems and workloads use these functions. Tracing their time provides one view of suffered file system latency.</p><p><font size="-2"><pre># stap fsslower-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/killsnoop-nd.stp"></a><a href="#lwtools/killsnoop-nd.stp">¶</a> <a href="lwtools/killsnoop-nd.stp">lwtools/killsnoop-nd.stp</a> - Trace kill() signals showing process and signal details (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces signals system-wide, including those sent by the kill(1) command, and shows various details.</p><p><font size="-2"><pre># stap killsnoop-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/opensnoop-nd.stp"></a><a href="#lwtools/opensnoop-nd.stp">¶</a> <a href="lwtools/opensnoop-nd.stp">lwtools/opensnoop-nd.stp</a> - Trace open() syscalls showing filenames (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces the open() syscall system-wide, to show which files are being opened, and by who.</p><p><font size="-2"><pre># stap opensnoop-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/rwtime-nd.stp"></a><a href="#lwtools/rwtime-nd.stp">¶</a> <a href="lwtools/rwtime-nd.stp">lwtools/rwtime-nd.stp</a> - Summarize read() and write() syscall latency (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces read() and write() syscalls, producing a histogram summary of their durations (aka latencies).</p><p><font size="-2"><pre># stap rwtime-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a name="lwtools/syscallbypid-nd.stp"></a><a href="#lwtools/syscallbypid-nd.stp">¶</a> <a href="lwtools/syscallbypid-nd.stp">lwtools/syscallbypid-nd.stp</a> - Count syscalls with process details (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces syscalls system-wide, and produces a summary report showing their counts by process ID, process name, and syscall types.</p><p><font size="-2"><pre># stap syscallbypid-nd.stp -c "sleep 1"</pre></font></p>
</li><li><a name="memory/glibc-malloc.stp"></a><a href="#memory/glibc-malloc.stp">¶</a> <a href="memory/glibc-malloc.stp">memory/glibc-malloc.stp</a> - Overview glibc malloc internal operations<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <br>
<p>This script reports on internal statistics of the glibc malloc implementation, as used by a process restricted by stap -x/-c</p><p><font size="-2"><pre># stap glibc-malloc.stp -c 'stap --dump-functions'</pre></font></p>
<p>The kmalloc-top perl program runs a small systemtap script to collect stack traces for each call to the kmalloc function and counts the time that each stack trace is observed. When kmalloc-top exits it prints out sorted list. The output can be filtered to print only the first N stack traces (-t), stack traces with a minimum counts (-m), or exclude certain stack traces (-e).</p><p><i><a href="memory/kmalloc-top">sample usage in memory/kmalloc-top</i></font></p>
</li><li><a name="memory/last_100_frees.stp"></a><a href="#memory/last_100_frees.stp">¶</a> <a href="memory/last_100_frees.stp">memory/last_100_frees.stp</a> - Log recent free(3) calls.<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
-<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><font size="-2"><pre># stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd</pre></font></p>
+<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><i><a href="memory/last_100_frees.txt">sample usage in memory/last_100_frees.txt</i></font></p>
</li><li><a name="memory/mmanonpage.stp"></a><a href="#memory/mmanonpage.stp">¶</a> <a href="memory/mmanonpage.stp">memory/mmanonpage.stp</a> - Track Virtual Memory System Actions on Anonymous Pages<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
<p>The mmanonpage.stp script uses the virtual memory tracepoints available in some kernels to track the number of faults, user space frees, page ins, copy on writes and unmaps for anonymous pages. When the script is terminated the counts are printed for each process that allocated pages while the script was running. This script displays the anonymous page statistics for each process that ran while the script is active. It's useful in debugging leaks in the anonymous regions of a process.</p><p><font size="-2"><pre># stap mmanonpage.stp -c "sleep 1"</pre></font></p>
<p>The mmwriteback.stp script uses the virtual memory tracepoints available in some kernels to report all of the file writebacks that occur form kupdate, pdflush and kjournald while the script is running. It's useful in determining where writes are coming from on a supposedly idle system that is experiencing unexpected IO.</p><p><font size="-2"><pre># stap mmwriteback.stp -c "sleep 1"</pre></font></p>
</li><li><a name="memory/numa_faults.stp"></a><a href="#memory/numa_faults.stp">¶</a> <a href="memory/numa_faults.stp">memory/numa_faults.stp</a> - Summarize Process Misses across NUMA Nodes<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#NUMA">NUMA</a> <br>
-<p>The numa_faults.stp script tracks the read and write pages faults for each process. When the script exits it prints out the total read and write pages faults for each process. The script also provide a break down of page faults per node for each process. This script is useful for determining whether the program has good locality (page faults limited to a single node) on a NUMA computer.</p><p><font size="-2"><pre># stap numa_faults.stp -c "sleep 1"</pre></font></p>
+<p>The numa_faults.stp script tracks the read and write pages faults for each process. When the script exits it prints out the total read and write pages faults for each process. The script also provide a break down of page faults per node for each process. This script is useful for determining whether the program has good locality (page faults limited to a single node) on a NUMA computer.</p><p><i><a href="memory/numa_faults.txt">sample usage in memory/numa_faults.txt</i></font></p>
</li><li><a name="memory/overcommit.stp"></a><a href="#memory/overcommit.stp">¶</a> <a href="memory/overcommit.stp">memory/overcommit.stp</a> - Log Failed Process Memory Allocation Due to Overcommit Limits<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#LIMITS">LIMITS</a> <br>
<p>The overcommit.stp script prints a line each time the kernel refuses a memory allocation request from a process because of /proc/sys/vm/overcommit* limits.</p><p><font size="-2"><pre># stap overcommit.stp -c "sleep 1"</pre></font></p>
</li><li><a name="memory/pfaults.stp"></a><a href="#memory/pfaults.stp">¶</a> <a href="memory/pfaults.stp">memory/pfaults.stp</a> - Generate Log of Major and Minor Page Faults<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
-<p>The pfaults.stp script generates a simple log for each major and minor page fault that occurs on the system. Each line contains a timestamp (in microseconds) when the page fault servicing was completed, the pid of the process, the address of the page fault, the type of access (read or write), the type of fault (major or minor), and the elapsed time for page fault. This log can be examined to determine where the page faults are occurring.</p><p><font size="-2"><pre># stap pfaults.stp -c "sleep 1"</pre></font></p>
+<p>The pfaults.stp script generates a simple log for each major and minor page fault that occurs on the system. Each line contains a timestamp (in microseconds) when the page fault servicing was completed, the pid of the process, the address of the page fault, the type of access (read or write), the type of fault (major or minor), and the elapsed time for page fault. This log can be examined to determine where the page faults are occurring.</p><p><i><a href="memory/pfaults.txt">sample usage in memory/pfaults.txt</i></font></p>
</li><li><a name="memory/vm.tracepoints.stp"></a><a href="#memory/vm.tracepoints.stp">¶</a> <a href="memory/vm.tracepoints.stp">memory/vm.tracepoints.stp</a> - Collect Slab Allocation Statistics<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#SLAB">SLAB</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><font size="-2"><pre># stap vm.tracepoints.stp -c "sleep 10"</pre></font></p>
+<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><i><a href="memory/vm.tracepoints.txt">sample usage in memory/vm.tracepoints.txt</i></font></p>
</li><li><a name="network/autofs4.stp"></a><a href="#network/autofs4.stp">¶</a> <a href="network/autofs4.stp">network/autofs4.stp</a> - Watch autofs4 Operations<br>
keywords: <a href="keyword-index.html#NETWORK">NETWORK</a> <a href="keyword-index.html#AUTOFS">AUTOFS</a> <a href="keyword-index.html#NFS">NFS</a> <br>
<p>Trace key autofs4 operations such as mounting or unmounting remote filesystems.</p><p><font size="-2"><pre># stap autofs4.stp -c "sleep 1"</pre></font></p>
<p>The pf2.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top ten kernel functions with samples.</p><p><i><a href="profiling/pf2.txt">sample usage in profiling/pf2.txt</i></font></p>
</li><li><a name="profiling/pf3.stp"></a><a href="#profiling/pf3.stp">¶</a> <a href="profiling/pf3.stp">profiling/pf3.stp</a> - Profile Kernel/User Functions<br>
keywords: <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p>The pf3.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty kernel and/or user functions with samples. Use any of --ldd, --all-modules, -d MODULE, -d /PATH/TO/EXEC to add more symbolic info.</p><p><font size="-2"><pre># stap pf3.stp -c "sleep 6" --all-modules --ldd</pre></font></p>
+<p>The pf3.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty kernel and/or user functions with samples. Use any of --ldd, --all-modules, -d MODULE, -d /PATH/TO/EXEC to add more symbolic info.</p><p><i><a href="profiling/pf3.txt">sample usage in profiling/pf3.txt</i></font></p>
</li><li><a name="profiling/pf4.stp"></a><a href="#profiling/pf4.stp">¶</a> <a href="profiling/pf4.stp">profiling/pf4.stp</a> - Profile Kernel/User Backtraces<br>
keywords: <a href="keyword-index.html#_BEST">_BEST</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
<p>The pf4.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty kernel and/or user stack backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d MODULE, -d /PATH/TO/EXEC to add more symbolic info.</p><p><i><a href="profiling/pf4.txt">sample usage in profiling/pf4.txt</i></font></p>
# stap bkl_stats.stp -c "sleep 1"
+lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo)
+keywords: net socket
+
+ This traces socket duration from the accept() syscall to close(), and
+ provides details on the lifespan of these passive connections,
+ showing the distribution as a histogram.
+
+ # stap accept2close-nd.stp -c "sleep 1"
+
+
+lwtools/biolatency-nd.stp - Measure block I/O latency distribution (non-debuginfo)
+keywords: io
+
+ This measures block I/O latency (storage I/O, ie, disk I/O), and
+ shows the distribution as a histogram. This can be useful to identify
+ the characteristics of I/O latency, beyond the averages shown by
+ iostat(1). For example, to study I/O latency outliers, or multi-modal
+ distributions.
+
+ # stap biolatency-nd.stp 1 1
+
+
+lwtools/bitesize-nd.stp - Measure block I/O size distribution (non-debuginfo)
+keywords: io
+
+ This uses the kernel tracepoint block_rq_insert to read the size of
+ I/O. The output includes the name of the process or thread that was
+ on-CPU when the I/O request was inserted on the issue queue.
+
+ # stap bitesize-nd.stp -c "sleep 1"
+
+
+lwtools/execsnoop-nd.stp - Trace process exec() with command line argument details (non-debuginfo)
+keywords: io
+
+ This can identify if CPU is consumed by short-lived processes, by
+ tracing new process execution. It works by tracing exec() from the
+ fork()->exec() sequence, which means it will not catch new processes
+ that only fork(). It will also show every exec(), including those if
+ a process re-execs.
+
+ # stap execsnoop-nd.stp -c "sleep 1"
+
+
+lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo)
+keywords: io filesystem
+
+ This dynamically traces two common file system functions:
+ do_sync_read() and do_sync_write(), and reports a histogram
+ distribution of latency. Many, but not all, file systems and
+ workloads use these functions. Tracing their time provides one view
+ of suffered file system latency.
+
+ # stap fslatency-nd.stp 1 1
+
+
+lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo)
+keywords: io filesystem
+
+ This dynamically traces two common file system functions:
+ do_sync_read() and do_sync_write(), and shows details of each call
+ that is slower than a threshold. Many, but not all, file systems and
+ workloads use these functions. Tracing their time provides one view
+ of suffered file system latency.
+
+ # stap fsslower-nd.stp -c "sleep 1"
+
+
+lwtools/killsnoop-nd.stp - Trace kill() signals showing process and signal details (non-debuginfo)
+keywords: io
+
+ This traces signals system-wide, including those sent by the kill(1)
+ command, and shows various details.
+
+ # stap killsnoop-nd.stp -c "sleep 1"
+
+
+lwtools/opensnoop-nd.stp - Trace open() syscalls showing filenames (non-debuginfo)
+keywords: io
+
+ This traces the open() syscall system-wide, to show which files are
+ being opened, and by who.
+
+ # stap opensnoop-nd.stp -c "sleep 1"
+
+
+lwtools/rwtime-nd.stp - Summarize read() and write() syscall latency (non-debuginfo)
+keywords: io
+
+ This traces read() and write() syscalls, producing a histogram
+ summary of their durations (aka latencies).
+
+ # stap rwtime-nd.stp -c "sleep 1"
+
+
+lwtools/syscallbypid-nd.stp - Count syscalls with process details (non-debuginfo)
+keywords: io
+
+ This traces syscalls system-wide, and produces a summary report
+ showing their counts by process ID, process name, and syscall types.
+
+ # stap syscallbypid-nd.stp -c "sleep 1"
+
+
memory/glibc-malloc.stp - Overview glibc malloc internal operations
keywords: memory process
-#! /usr/bin/env stap
-/*
- * Copyright (C) 2009 IBM Corp.
- * This file is part of systemtap, and is free software. You can
- * redistribute it and/or modify it under the terms of the GNU General
- * Public License (GPL); either version 2, or (at your option) any
- * later version.
- *
- * Version 1.0 prerna@linux.vnet.ibm.com 2009-10-28
- *
- * Name:
- * interrupts-by-dev.stp
- *
- * Description:
- * Script to profile interrupts received by each device per 100 ms.
- *
- *
- */
+This script reports every 100ms, so we limit it to 10 reports.
-global devices
+# stap interrupts-by-dev.stp -c 'sleep 1'
-probe begin {
- printf("\t DEVICE \t NUMBER OF INTERRUPTS \n");
-}
-
-probe irq_handler.entry {
- devices[dev_name]++;
-}
-
-probe timer.ms(100) {
- foreach ( devname in devices )
- printf(" %20s : %5d\n",kernel_string(devname),devices[devname]);
- delete devices
-}
+ DEVICE NUMBER OF INTERRUPTS
+ p2p1 : 4
+ p2p1 : 5
+ p2p1 : 12
+ p2p1 : 8
+ p2p1 : 5
+ p2p1 : 6
+ p2p1 : 4
+ xhci_hcd : 20
+ p2p1 : 5
+ p2p1 : 6
+ p2p1 : 4
+ nvidia : 1
--- /dev/null
+# stap iodevstats.stp -c "sleep 1"
+
+starting probe
+
+ read read write write
+ name device read KB tot B avg write KB tot B avg
+ sleep 253, 0 2 0 368 0 0 0
+ stapio 253, 0 4 0 132 0 0 0
+ stapio 0, 7 7 0 3 0 0 0
+ stapio 0, 8 0 0 0 1 0 15
--- /dev/null
+# stap iostats.stp -c "sleep 1"
+
+starting probe
+
+ read read write write
+ name open read KB tot B avg write KB tot B avg
+ sleep 10 1 0 512 0 0 0
+ stapio 0 7 0 3 1 0 15
--- /dev/null
+# stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/sda"
+
+1+0 records in
+1+0 records out
+512 bytes (512 B) copied, 0.0196359 s, 26.1 kB/s
+device sda sector 0 read, process dd[26847]
+ 0xc05ed2e0 : generic_make_request+0x0/0x5b0 [kernel]
+ 0xc05ed8ee : submit_bio+0x5e/0x120 [kernel]
+ 0xc0562f2d : submit_bh+0x1bd/0x240 [kernel]
+ 0xc0564fcd : block_read_full_page+0x25d/0x2f0 [kernel]
+ 0xc04fc201 : __do_page_cache_readahead+0x1f1/0x210 [kernel]
+ 0xc04fc23e : ra_submit+0x1e/0x30 [kernel]
+ 0xc04fc64b : page_cache_sync_readahead+0x3b/0x60 [kernel]
+ 0xc04ebf6a : generic_file_aio_read+0x59a/0x760 [kernel]
+ 0xc0536cc5 : do_sync_read+0xd5/0x120 [kernel]
+ 0xc053766d : vfs_read+0x9d/0x190 [kernel]
+ 0xc05377a1 : sys_read+0x41/0x70 [kernel]
+ 0xc04099bf : sysenter_do_call+0x12/0x28 [kernel]
probe kernel.function("tty_audit_add_data") {
major=$tty->driver->major;
minor=$tty->driver->minor_start + $tty->index;
- pgrp=$tty->pgrp %( kernel_v >= "2.6.24" %? ->numbers[0]->nr %: %);
+ try { pgrp=$tty->pgrp %( kernel_v >= "2.6.24" %? ->numbers[0]->nr %: %) } catch { }
data=kernel_string_n($data,$size);
uid=uid()
--- /dev/null
+The stream of text is shown on leftward-scrolling ticker-text.
+
+# stap -g ttyspy.txt
+
+(maj,min, pgrp, uid)
+(128, 16, 0, 500) ttyspy.txt\r\n\r\n
+(128, 18, 0, 500) \033[1;1H\033[J(maj,min, pgrp, uid)\r\n\033[1;1H\033
+(136, 11,27836, 500) pwdd\027passwd\n
+(128, 11, 0, 500) word for fche.\r(current) UNIX password:
+(136, 3,21709, 500) pwd\nls -altr\npasswd\027
+(128, 3, 0, 500) l\r\r\033]2;very ls -altr\bp\b\b\b\b\b\b \b\b\b\b\b
+
</ul>
<h2>Examples by Keyword</h2>
-<p><tt><a href="#_BEST">_BEST(15)</a> <a href="#AUDIT">AUDIT(1)</a> <a href="#AUTOFS">AUTOFS(1)</a> <a href="#BACKTRACE">BACKTRACE(8)</a> <a href="#CALLGRAPH">CALLGRAPH(2)</a> <a href="#CPU">CPU(1)</a> <a href="#DEVICE">DEVICE(1)</a> <a href="#DIAGRAM">DIAGRAM(1)</a> <a href="#DISK">DISK(6)</a> <a href="#FILE">FILE(2)</a> <a href="#FILESYSTEM">FILESYSTEM(
2)</a> <a href="#FORMAT">FORMAT(2)</a> <a href="#FUNCTION">FUNCTION(3)</a> <a href="#FUTEX">FUTEX(2)</a> <a href="#GURU">GURU(7)</a> <a href="#INTERRUPT">INTERRUPT(3)</a> <a href="#IO">IO(18)</a> <a href="#KVM">KVM(3)</a> <a href="#LIMITS">LIMITS(2)</a> <a href="#LOCKING">LOCKING(6)</a> <a href="#MEMORY">MEMORY(14)</a> <a href="#MONITORING">MONITORING(5)</a> <a href="#NANOSLEEP">NANOSLEEP(1)</a> <a href="#NETWORK">NETWORK(15)</a> <a href="#NFS">NFS(4)</a> <a href="#NUMA">NUMA(1)</a> <a href="#PACKETS">PACKETS(1)</a> <a href="#PROCESS">PROCESS(22)</a> <a href="#PROFILING">PROFILING(20)</a> <a href="#QEMU">QEMU(2)</a> <a href="#REGEX">REGEX(1)</a> <a href="#SCHEDULER">SCHEDULER(7)</a> <a href="#SCSI">SCSI(1)</a> <a href="#SECURITY">SECURITY(2)</a> <a href="#SIGNALS">SIGNALS(5)</a> <a href="#SIMPLE">SIMPLE(4)</a> <a href="#SLAB">SLAB(1)</a> <a href="#SOCKET">SOCKET(6)</a> <a href="#STAPGAMES">STAPGAMES(5)</a> <a href="#STATISTICS">STATISTICS(8)</a> <a href="#SYSCALL">SYSCALL(12)</a> <a href="#TCP">TCP(3)</a> <a href="#THREAD">THREAD(2)</a> <a href="#TIME">TIME(3)</a> <a href="#TRACE">TRACE(3)</a> <a href="#TRACEPOINT">TRACEPOINT(4)</a> <a href="#TRAFFIC">TRAFFIC(4)</a> <a href="#TTY">TTY(1)</a> <a href="#UTILIZATION">UTILIZATION(1)</a> <a href="#VIRTUALIZATION">VIRTUALIZATION(3)</a> <a href="#WATCHDOG">WATCHDOG(1)</a> <a href="#WATCHPOINT">WATCHPOINT(2)</a> </tt></p>
+<p><tt><a href="#_BEST">_BEST(15)</a> <a href="#AUDIT">AUDIT(1)</a> <a href="#AUTOFS">AUTOFS(1)</a> <a href="#BACKTRACE">BACKTRACE(8)</a> <a href="#CALLGRAPH">CALLGRAPH(2)</a> <a href="#CPU">CPU(1)</a> <a href="#DEVICE">DEVICE(1)</a> <a href="#DIAGRAM">DIAGRAM(1)</a> <a href="#DISK">DISK(6)</a> <a href="#FILE">FILE(2)</a> <a href="#FILESYSTEM">FILESYSTEM(
4)</a> <a href="#FORMAT">FORMAT(2)</a> <a href="#FUNCTION">FUNCTION(3)</a> <a href="#FUTEX">FUTEX(2)</a> <a href="#GURU">GURU(7)</a> <a href="#INTERRUPT">INTERRUPT(3)</a> <a href="#IO">IO(27)</a> <a href="#KVM">KVM(3)</a> <a href="#LIMITS">LIMITS(2)</a> <a href="#LOCKING">LOCKING(6)</a> <a href="#MEMORY">MEMORY(14)</a> <a href="#MONITORING">MONITORING(5)</a> <a href="#NANOSLEEP">NANOSLEEP(1)</a> <a href="#NET">NET(1)</a> <a href="#NETWORK">NETWORK(15)</a> <a href="#NFS">NFS(4)</a> <a href="#NUMA">NUMA(1)</a> <a href="#PACKETS">PACKETS(1)</a> <a href="#PROCESS">PROCESS(22)</a> <a href="#PROFILING">PROFILING(20)</a> <a href="#QEMU">QEMU(2)</a> <a href="#REGEX">REGEX(1)</a> <a href="#SCHEDULER">SCHEDULER(7)</a> <a href="#SCSI">SCSI(1)</a> <a href="#SECURITY">SECURITY(2)</a> <a href="#SIGNALS">SIGNALS(5)</a> <a href="#SIMPLE">SIMPLE(4)</a> <a href="#SLAB">SLAB(1)</a> <a href="#SOCKET">SOCKET(7)</a> <a href="#STAPGAMES">STAPGAMES(5)</a> <a href="#STATISTICS">STATISTICS(8)</a> <a href="#SYSCALL">SYSCALL(12)</a> <a href="#TCP">TCP(3)</a> <a href="#THREAD">THREAD(2)</a> <a href="#TIME">TIME(3)</a> <a href="#TRACE">TRACE(3)</a> <a href="#TRACEPOINT">TRACEPOINT(4)</a> <a href="#TRAFFIC">TRAFFIC(4)</a> <a href="#TTY">TTY(1)</a> <a href="#UTILIZATION">UTILIZATION(1)</a> <a href="#VIRTUALIZATION">VIRTUALIZATION(3)</a> <a href="#WATCHDOG">WATCHDOG(1)</a> <a href="#WATCHPOINT">WATCHPOINT(2)</a> </tt></p>
<h3><a name="_BEST"><a href="#_BEST">¶</a> _BEST</a></h3>
<ul>
<li><a href="general/eventcount.stp">general/eventcount.stp</a> - Count Specified Events<br>
<p>When a reschedule occurs during an AIO io_submit call, accumulate the traceback in a histogram. When the script exits prints out a sorted list from most common to least common backtrace.</p><p><font size="-2"><pre># stap io_submit.stp -c "sleep 1"</pre></font></p>
</li><li><a href="memory/last_100_frees.stp">memory/last_100_frees.stp</a> - Log recent free(3) calls.<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
-<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><font size="-2"><pre># stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd</pre></font></p>
+<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><i><a href="memory/last_100_frees.txt">sample usage in memory/last_100_frees.txt</i></font></p>
</li><li><a href="process/auditbt.stp">process/auditbt.stp</a> - Generate backtraces for kernel audit events<br>
keywords: <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#SECURITY">SECURITY</a> <a href="keyword-index.html#AUDIT">AUDIT</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
<p>Attaches to the kernel audit-log paths (also used by libaudit), and log every record being sent, along with a user-space backtrace of the process that caused it.</p><p><font size="-2"><pre># stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true"</pre></font></p>
<p>This script monitors a filesystem implementations for early internal indications of ENOSPC, and reports these to the system logger and the systemtap console.</p><p><font size="-2"><pre># stap enospc.stp -c "sleep 1"</pre></font></p>
</li><li><a href="io/mbrwatch.stp">io/mbrwatch.stp</a> - Monitor Read/Write of the Boot Sector Area of Block Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#DISK">DISK</a> <br>
-<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><font size="-2"><pre># stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
-/proc/partitions | grep . | awk '{print $4}' | head -1`"</pre></font></p>
+<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><i><a href="io/mbrwatch.txt">sample usage in io/mbrwatch.txt</i></font></p>
</li><li><a href="network/nfsd_unlink.stp">network/nfsd_unlink.stp</a> - Find Which Client Is Removing NFS Files on Server<br>
keywords: <a href="keyword-index.html#NFS">NFS</a> <a href="keyword-index.html#DISK">DISK</a> <br>
<p>The nfsd_unlink.stp script lists the ip address and file name each time time a file is being removed or unlinked by the nfsd. This script is run on the nfs server.</p><p><font size="-2"><pre># stap nfsd_unlink.stp -c "sleep 1"</pre></font></p>
<ul>
<li><a href="general/badname.stp">general/badname.stp</a> - Bad Filename Filter<br>
keywords: <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The badname.stp script shows how one could prevent the creation of files with undesirable names using guru mode.</p><p><font size="-2"><pre># stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \
-denied"</pre></font></p>
+<p>The badname.stp script shows how one could prevent the creation of files with undesirable names using guru mode.</p><p><i><a href="general/badname.txt">sample usage in general/badname.txt</i></font></p>
</li><li><a href="io/enospc.stp">io/enospc.stp</a> - Report an disk out-of-space condition.<br>
keywords: <a href="keyword-index.html#DISK">DISK</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
<p>This script monitors a filesystem implementations for early internal indications of ENOSPC, and reports these to the system logger and the systemtap console.</p><p><font size="-2"><pre># stap enospc.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/fslatency-nd.stp">lwtools/fslatency-nd.stp</a> - Measure the distribution of file system synchronous read and write latency (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
+<p>This dynamically traces two common file system functions: do_sync_read() and do_sync_write(), and reports a histogram distribution of latency. Many, but not all, file systems and workloads use these functions. Tracing their time provides one view of suffered file system latency.</p><p><font size="-2"><pre># stap fslatency-nd.stp 1 1</pre></font></p>
+</li><li><a href="lwtools/fsslower-nd.stp">lwtools/fsslower-nd.stp</a> - Trace slow file system synchronous reads and writes (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
+<p>This dynamically traces two common file system functions: do_sync_read() and do_sync_write(), and shows details of each call that is slower than a threshold. Many, but not all, file systems and workloads use these functions. Tracing their time provides one view of suffered file system latency.</p><p><font size="-2"><pre># stap fsslower-nd.stp -c "sleep 1"</pre></font></p>
</li></ul>
<h3><a name="FORMAT"><a href="#FORMAT">¶</a> FORMAT</a></h3>
<ul>
<ul>
<li><a href="general/func_time_stats.stp">general/func_time_stats.stp</a> - Function Time Statistics<br>
keywords: <a href="keyword-index.html#FUNCTION">FUNCTION</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The func_time_stats.stp script tracks the wall clock time for each invocation of the function probe listed as the first command line argument. When the script exits it prints out the minimum, average, and maximum times in microseconds followed by a count of times that the function was called and a histogram showing the distributions of times.</p><p><font size="-2"><pre># stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1"</pre></font></p>
+<p>The func_time_stats.stp script tracks the wall clock time for each invocation of the function probe listed as the first command line argument. When the script exits it prints out the minimum, average, and maximum times in microseconds followed by a count of times that the function was called and a histogram showing the distributions of times.</p><p><i><a href="general/func_time_stats.txt">sample usage in general/func_time_stats.txt</i></font></p>
</li><li><a href="profiling/functioncallcount.stp">profiling/functioncallcount.stp</a> - Count Times Functions Are Called<br>
keywords: <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#FUNCTION">FUNCTION</a> <br>
<p>The functioncallcount.stp script takes one argument, a list of functions to probe. The script will run and count the number of times that each of the functions on the list is called. On exit the script will print a sorted list from most frequently to least frequently called function.</p><p><font size="-2"><pre># stap -w functioncallcount.stp "*@mm/*.c" -c "sleep 1"</pre></font></p>
<ul>
<li><a href="general/badname.stp">general/badname.stp</a> - Bad Filename Filter<br>
keywords: <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The badname.stp script shows how one could prevent the creation of files with undesirable names using guru mode.</p><p><font size="-2"><pre># stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \
-denied"</pre></font></p>
+<p>The badname.stp script shows how one could prevent the creation of files with undesirable names using guru mode.</p><p><i><a href="general/badname.txt">sample usage in general/badname.txt</i></font></p>
</li><li><a href="io/eatmydata.stp">io/eatmydata.stp</a> - disable fsync<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#GURU">GURU</a> <a href="keyword-index.html#SIMPLE">SIMPLE</a> <br>
<p>Suppresses fsync() syscalls from processes identified by stap -c/-x by turning them into presumed-faster fsync() on some dummy or other file descriptor</p><p><i><a href="io/eatmydata.txt">sample usage in io/eatmydata.txt</i></font></p>
</li><li><a href="io/ttyspy.stp">io/ttyspy.stp</a> - Monitor TTY Typing<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#TTY">TTY</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><font size="-2"><pre># stap --skip-badvars -g ttyspy.stp -c "sleep 1"</pre></font></p>
+<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><i><a href="io/ttyspy.txt">sample usage in io/ttyspy.txt</i></font></p>
</li><li><a href="network/netfilter_drop.stp">network/netfilter_drop.stp</a> - System-Wide Network Packet Dropping Tool<br>
keywords: <a href="keyword-index.html#NETWORK">NETWORK</a> <a href="keyword-index.html#PACKETS">PACKETS</a> <a href="keyword-index.html#GURU">GURU</a> <br>
<p>The script drops the specified number of packets of the specified protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified, all incoming packets are dropped. The number of packets to drop can be specified with a positive integer. A value of 0 indicates that packets should be dropped until the user manually exits.</p><p><font size="-2"><pre># stap -g netfilter_drop.stp TCP 1 -c "sleep 2"</pre></font></p>
<p>The ioblktime.stp script tracks the amount of time that each block IO requests spend waiting for completion. The script computes the average waiting time for block IO per device and prints list every 10 seconds. In some cases there can be too many outstanding block IO operations and the script may exceed the default number of MAXMAPENTRIES allowed. In this case the allowed number can be increased with "-DMAXMAPENTRIES=10000" option on the stap command line.</p><p><font size="-2"><pre># stap ioblktime.stp -c "sleep 1"</pre></font></p>
</li><li><a href="io/iodevstats.stp">io/iodevstats.stp</a> - List Executables Reading and Writing the Most Data by Device<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p> The iodevstats.stp script measures the amount of data successfully read and written by all the executables for each io device on the system. The output is sorted from greatest sum of bytes read and written to a device by an executable to the least. The output contains device major/minor number, the count of operations (reads and writes), the totals and averages for the number of bytes read and written.</p><p><font size="-2"><pre># stap iodevstats.stp -c "sleep 1"</pre></font></p>
+<p> The iodevstats.stp script measures the amount of data successfully read and written by all the executables for each io device on the system. The output is sorted from greatest sum of bytes read and written to a device by an executable to the least. The output contains device major/minor number, the count of operations (reads and writes), the totals and averages for the number of bytes read and written.</p><p><i><a href="io/iodevstats.txt">sample usage in io/iodevstats.txt</i></font></p>
</li><li><a href="io/iostat-scsi.stp">io/iostat-scsi.stp</a> - IO Statistics for SCSI Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#SCSI">SCSI</a> <br>
<p>The iostat-scsi.stp script provides a breakdown of the number of blks read and written on the machine's various SCSI devices. The script takes one argument which is the number of seconds between reports.</p><p><i><a href="io/iostat-scsi.txt">sample usage in io/iostat-scsi.txt</i></font></p>
</li><li><a href="io/iostats.stp">io/iostats.stp</a> - List Executables Reading and Writing the Most Data<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p> The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written.</p><p><font size="-2"><pre># stap iostats.stp -c "sleep 1"</pre></font></p>
+<p> The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written.</p><p><i><a href="io/iostats.txt">sample usage in io/iostats.txt</i></font></p>
</li><li><a href="io/iotime.stp">io/iotime.stp</a> - Trace Time Spent in Read and Write for Files <br>
keywords: <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#SYSCALL">SYSCALL</a> <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILE">FILE</a> <br>
<p>The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spent in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parentheses. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls.</p><p><font size="-2"><pre># stap iotime.stp -c "sleep 1"</pre></font></p>
<p>Every five seconds print out the top ten executables generating I/O traffic during that interval sorted in descending order.</p><p><font size="-2"><pre># stap iotop.stp -c "sleep 1"</pre></font></p>
</li><li><a href="io/mbrwatch.stp">io/mbrwatch.stp</a> - Monitor Read/Write of the Boot Sector Area of Block Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#DISK">DISK</a> <br>
-<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><font size="-2"><pre># stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
-/proc/partitions | grep . | awk '{print $4}' | head -1`"</pre></font></p>
+<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><i><a href="io/mbrwatch.txt">sample usage in io/mbrwatch.txt</i></font></p>
</li><li><a href="io/nfs_func_users.stp">io/nfs_func_users.stp</a> - Tally the Number of NFS Functions Used by Each Process<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
<p>The nfs_func_users.stp script counts the uses of NFS functions in the kernel on a per process bases. The output is sorted from the process with the greatest number of NFS functions called to the least. The output contains the executable name, the process number, and the total number of NFS functions called by the process.</p><p><font size="-2"><pre># stap nfs_func_users.stp -c "sleep 1"</pre></font></p>
<p>Print out the executable name and process number as reads and writes to the specified device occur.</p><p><font size="-2"><pre># stap traceio2.stp 0x0801 -c "sleep 1"</pre></font></p>
</li><li><a href="io/ttyspy.stp">io/ttyspy.stp</a> - Monitor TTY Typing<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#TTY">TTY</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><font size="-2"><pre># stap --skip-badvars -g ttyspy.stp -c "sleep 1"</pre></font></p>
+<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><i><a href="io/ttyspy.txt">sample usage in io/ttyspy.txt</i></font></p>
+</li><li><a href="lwtools/biolatency-nd.stp">lwtools/biolatency-nd.stp</a> - Measure block I/O latency distribution (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This measures block I/O latency (storage I/O, ie, disk I/O), and shows the distribution as a histogram. This can be useful to identify the characteristics of I/O latency, beyond the averages shown by iostat(1). For example, to study I/O latency outliers, or multi-modal distributions.</p><p><font size="-2"><pre># stap biolatency-nd.stp 1 1</pre></font></p>
+</li><li><a href="lwtools/bitesize-nd.stp">lwtools/bitesize-nd.stp</a> - Measure block I/O size distribution (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This uses the kernel tracepoint block_rq_insert to read the size of I/O. The output includes the name of the process or thread that was on-CPU when the I/O request was inserted on the issue queue.</p><p><font size="-2"><pre># stap bitesize-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/execsnoop-nd.stp">lwtools/execsnoop-nd.stp</a> - Trace process exec() with command line argument details (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This can identify if CPU is consumed by short-lived processes, by tracing new process execution. It works by tracing exec() from the fork()->exec() sequence, which means it will not catch new processes that only fork(). It will also show every exec(), including those if a process re-execs.</p><p><font size="-2"><pre># stap execsnoop-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/fslatency-nd.stp">lwtools/fslatency-nd.stp</a> - Measure the distribution of file system synchronous read and write latency (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
+<p>This dynamically traces two common file system functions: do_sync_read() and do_sync_write(), and reports a histogram distribution of latency. Many, but not all, file systems and workloads use these functions. Tracing their time provides one view of suffered file system latency.</p><p><font size="-2"><pre># stap fslatency-nd.stp 1 1</pre></font></p>
+</li><li><a href="lwtools/fsslower-nd.stp">lwtools/fsslower-nd.stp</a> - Trace slow file system synchronous reads and writes (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILESYSTEM">FILESYSTEM</a> <br>
+<p>This dynamically traces two common file system functions: do_sync_read() and do_sync_write(), and shows details of each call that is slower than a threshold. Many, but not all, file systems and workloads use these functions. Tracing their time provides one view of suffered file system latency.</p><p><font size="-2"><pre># stap fsslower-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/killsnoop-nd.stp">lwtools/killsnoop-nd.stp</a> - Trace kill() signals showing process and signal details (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces signals system-wide, including those sent by the kill(1) command, and shows various details.</p><p><font size="-2"><pre># stap killsnoop-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/opensnoop-nd.stp">lwtools/opensnoop-nd.stp</a> - Trace open() syscalls showing filenames (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces the open() syscall system-wide, to show which files are being opened, and by who.</p><p><font size="-2"><pre># stap opensnoop-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/rwtime-nd.stp">lwtools/rwtime-nd.stp</a> - Summarize read() and write() syscall latency (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces read() and write() syscalls, producing a histogram summary of their durations (aka latencies).</p><p><font size="-2"><pre># stap rwtime-nd.stp -c "sleep 1"</pre></font></p>
+</li><li><a href="lwtools/syscallbypid-nd.stp">lwtools/syscallbypid-nd.stp</a> - Count syscalls with process details (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#IO">IO</a> <br>
+<p>This traces syscalls system-wide, and produces a summary report showing their counts by process ID, process name, and syscall types.</p><p><font size="-2"><pre># stap syscallbypid-nd.stp -c "sleep 1"</pre></font></p>
</li><li><a href="process/sleepingBeauties.stp">process/sleepingBeauties.stp</a> - Generate Backtraces of Threads Waiting for IO Operations<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#SCHEDULER">SCHEDULER</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
<p>The script monitors the time that threads spend in waiting for IO operations (in "D" state) in the wait_for_completion function. If a thread spends over 10ms, its name and backtrace is printed, and later so is the total delay.</p><p><font size="-2"><pre># stap sleepingBeauties.stp -c "sleep 1"</pre></font></p>
<ul>
<li><a href="general/sizeof.stp">general/sizeof.stp</a> - Print the Size of a C Type<br>
keywords: <a href="keyword-index.html#STATISTICS">STATISTICS</a> <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
-<p>This script prints the size of a type, based on dwarf debuginfo for any kernel or userspace module, or trial-compilation of a given header file name.</p><p><font size="-2"><pre># stap sizeof.stp FILE '</usr/include/stdio.h>'</pre></font></p>
+<p>This script prints the size of a type, based on dwarf debuginfo for any kernel or userspace module, or trial-compilation of a given header file name.</p><p><i><a href="general/sizeof.txt">sample usage in general/sizeof.txt</i></font></p>
</li><li><a href="memory/glibc-malloc.stp">memory/glibc-malloc.stp</a> - Overview glibc malloc internal operations<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <br>
<p>This script reports on internal statistics of the glibc malloc implementation, as used by a process restricted by stap -x/-c</p><p><font size="-2"><pre># stap glibc-malloc.stp -c 'stap --dump-functions'</pre></font></p>
<p>The kmalloc-top perl program runs a small systemtap script to collect stack traces for each call to the kmalloc function and counts the time that each stack trace is observed. When kmalloc-top exits it prints out sorted list. The output can be filtered to print only the first N stack traces (-t), stack traces with a minimum counts (-m), or exclude certain stack traces (-e).</p><p><i><a href="memory/kmalloc-top">sample usage in memory/kmalloc-top</i></font></p>
</li><li><a href="memory/last_100_frees.stp">memory/last_100_frees.stp</a> - Log recent free(3) calls.<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
-<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><font size="-2"><pre># stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd</pre></font></p>
+<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><i><a href="memory/last_100_frees.txt">sample usage in memory/last_100_frees.txt</i></font></p>
</li><li><a href="memory/mmanonpage.stp">memory/mmanonpage.stp</a> - Track Virtual Memory System Actions on Anonymous Pages<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
<p>The mmanonpage.stp script uses the virtual memory tracepoints available in some kernels to track the number of faults, user space frees, page ins, copy on writes and unmaps for anonymous pages. When the script is terminated the counts are printed for each process that allocated pages while the script was running. This script displays the anonymous page statistics for each process that ran while the script is active. It's useful in debugging leaks in the anonymous regions of a process.</p><p><font size="-2"><pre># stap mmanonpage.stp -c "sleep 1"</pre></font></p>
<p>The mmwriteback.stp script uses the virtual memory tracepoints available in some kernels to report all of the file writebacks that occur form kupdate, pdflush and kjournald while the script is running. It's useful in determining where writes are coming from on a supposedly idle system that is experiencing unexpected IO.</p><p><font size="-2"><pre># stap mmwriteback.stp -c "sleep 1"</pre></font></p>
</li><li><a href="memory/numa_faults.stp">memory/numa_faults.stp</a> - Summarize Process Misses across NUMA Nodes<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#NUMA">NUMA</a> <br>
-<p>The numa_faults.stp script tracks the read and write pages faults for each process. When the script exits it prints out the total read and write pages faults for each process. The script also provide a break down of page faults per node for each process. This script is useful for determining whether the program has good locality (page faults limited to a single node) on a NUMA computer.</p><p><font size="-2"><pre># stap numa_faults.stp -c "sleep 1"</pre></font></p>
+<p>The numa_faults.stp script tracks the read and write pages faults for each process. When the script exits it prints out the total read and write pages faults for each process. The script also provide a break down of page faults per node for each process. This script is useful for determining whether the program has good locality (page faults limited to a single node) on a NUMA computer.</p><p><i><a href="memory/numa_faults.txt">sample usage in memory/numa_faults.txt</i></font></p>
</li><li><a href="memory/overcommit.stp">memory/overcommit.stp</a> - Log Failed Process Memory Allocation Due to Overcommit Limits<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#LIMITS">LIMITS</a> <br>
<p>The overcommit.stp script prints a line each time the kernel refuses a memory allocation request from a process because of /proc/sys/vm/overcommit* limits.</p><p><font size="-2"><pre># stap overcommit.stp -c "sleep 1"</pre></font></p>
</li><li><a href="memory/pfaults.stp">memory/pfaults.stp</a> - Generate Log of Major and Minor Page Faults<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
-<p>The pfaults.stp script generates a simple log for each major and minor page fault that occurs on the system. Each line contains a timestamp (in microseconds) when the page fault servicing was completed, the pid of the process, the address of the page fault, the type of access (read or write), the type of fault (major or minor), and the elapsed time for page fault. This log can be examined to determine where the page faults are occurring.</p><p><font size="-2"><pre># stap pfaults.stp -c "sleep 1"</pre></font></p>
+<p>The pfaults.stp script generates a simple log for each major and minor page fault that occurs on the system. Each line contains a timestamp (in microseconds) when the page fault servicing was completed, the pid of the process, the address of the page fault, the type of access (read or write), the type of fault (major or minor), and the elapsed time for page fault. This log can be examined to determine where the page faults are occurring.</p><p><i><a href="memory/pfaults.txt">sample usage in memory/pfaults.txt</i></font></p>
</li><li><a href="memory/vm.tracepoints.stp">memory/vm.tracepoints.stp</a> - Collect Slab Allocation Statistics<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#SLAB">SLAB</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><font size="-2"><pre># stap vm.tracepoints.stp -c "sleep 10"</pre></font></p>
+<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><i><a href="memory/vm.tracepoints.txt">sample usage in memory/vm.tracepoints.txt</i></font></p>
</li></ul>
<h3><a name="MONITORING"><a href="#MONITORING">¶</a> MONITORING</a></h3>
<ul>
<p>This script places a set of probes (specified by $1), each of which monitors the state of some context $variable expression (specified by $2). Whenever the value changes, with respect to the active thread, the event is traced.</p><p><i><a href="general/varwatch.txt">sample usage in general/varwatch.txt</i></font></p>
</li><li><a href="io/mbrwatch.stp">io/mbrwatch.stp</a> - Monitor Read/Write of the Boot Sector Area of Block Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#DISK">DISK</a> <br>
-<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><font size="-2"><pre># stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
-/proc/partitions | grep . | awk '{print $4}' | head -1`"</pre></font></p>
+<p> The mbrwatch.stp script reports any attempted reads/writes of the first few sectors of a raw block device.</p><p><i><a href="io/mbrwatch.txt">sample usage in io/mbrwatch.txt</i></font></p>
</li><li><a href="io/ttyspy.stp">io/ttyspy.stp</a> - Monitor TTY Typing<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#TTY">TTY</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><font size="-2"><pre># stap --skip-badvars -g ttyspy.stp -c "sleep 1"</pre></font></p>
+<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><i><a href="io/ttyspy.txt">sample usage in io/ttyspy.txt</i></font></p>
</li><li><a href="process/auditbt.stp">process/auditbt.stp</a> - Generate backtraces for kernel audit events<br>
keywords: <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#SECURITY">SECURITY</a> <a href="keyword-index.html#AUDIT">AUDIT</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
<p>Attaches to the kernel audit-log paths (also used by libaudit), and log every record being sent, along with a user-space backtrace of the process that caused it.</p><p><font size="-2"><pre># stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true"</pre></font></p>
keywords: <a href="keyword-index.html#SYSCALL">SYSCALL</a> <a href="keyword-index.html#NANOSLEEP">NANOSLEEP</a> <br>
<p>The script watches each nanosleep syscall on the system. At the end of each nanosleep syscall the script prints out a line with a timestamp in microseconds, the pid, the executable name in parentheses, the "nanosleep:" key, and the duration of the sleep in microseconds.</p><p><font size="-2"><pre># stap sleeptime.stp -c "sleep 1"</pre></font></p>
</li></ul>
+<h3><a name="NET"><a href="#NET">¶</a> NET</a></h3>
+<ul>
+<li><a href="lwtools/accept2close-nd.stp">lwtools/accept2close-nd.stp</a> - Show socket lifespan, from accept() to close() (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#NET">NET</a> <a href="keyword-index.html#SOCKET">SOCKET</a> <br>
+<p>This traces socket duration from the accept() syscall to close(), and provides details on the lifespan of these passive connections, showing the distribution as a histogram.</p><p><i><a href="lwtools/accept2close-nd.txt">sample usage in lwtools/accept2close-nd.txt</i></font></p>
+</li></ul>
<h3><a name="NETWORK"><a href="#NETWORK">¶</a> NETWORK</a></h3>
<ul>
<li><a href="network/autofs4.stp">network/autofs4.stp</a> - Watch autofs4 Operations<br>
<ul>
<li><a href="memory/numa_faults.stp">memory/numa_faults.stp</a> - Summarize Process Misses across NUMA Nodes<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#NUMA">NUMA</a> <br>
-<p>The numa_faults.stp script tracks the read and write pages faults for each process. When the script exits it prints out the total read and write pages faults for each process. The script also provide a break down of page faults per node for each process. This script is useful for determining whether the program has good locality (page faults limited to a single node) on a NUMA computer.</p><p><font size="-2"><pre># stap numa_faults.stp -c "sleep 1"</pre></font></p>
+<p>The numa_faults.stp script tracks the read and write pages faults for each process. When the script exits it prints out the total read and write pages faults for each process. The script also provide a break down of page faults per node for each process. This script is useful for determining whether the program has good locality (page faults limited to a single node) on a NUMA computer.</p><p><i><a href="memory/numa_faults.txt">sample usage in memory/numa_faults.txt</i></font></p>
</li></ul>
<h3><a name="PACKETS"><a href="#PACKETS">¶</a> PACKETS</a></h3>
<ul>
<p>This script reports on internal statistics of the glibc malloc implementation, as used by a process restricted by stap -x/-c</p><p><font size="-2"><pre># stap glibc-malloc.stp -c 'stap --dump-functions'</pre></font></p>
</li><li><a href="memory/last_100_frees.stp">memory/last_100_frees.stp</a> - Log recent free(3) calls.<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
-<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><font size="-2"><pre># stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd</pre></font></p>
+<p>This script reports on the last few free(3) libc calls done by processes (possibly restricted by stap -x/-c), along with a userspace backtrace at those moments.</p><p><i><a href="memory/last_100_frees.txt">sample usage in memory/last_100_frees.txt</i></font></p>
</li><li><a href="network/connect_stat.stp">network/connect_stat.stp</a> - Show Process Ancestry for IP Connections<br>
keywords: <a href="keyword-index.html#NETWORK">NETWORK</a> <a href="keyword-index.html#SOCKET">SOCKET</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <br>
<p>The connect_stat.stp script prints a task's entire ancestry (parent process name/uid/gid) whenever it attempts an outgoing socket connection to a given IP address.</p><p><font size="-2"><pre># stap connect_stat.stp 127.0.0.1 -c "sleep 1"</pre></font></p>
<ul>
<li><a href="io/iodevstats.stp">io/iodevstats.stp</a> - List Executables Reading and Writing the Most Data by Device<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p> The iodevstats.stp script measures the amount of data successfully read and written by all the executables for each io device on the system. The output is sorted from greatest sum of bytes read and written to a device by an executable to the least. The output contains device major/minor number, the count of operations (reads and writes), the totals and averages for the number of bytes read and written.</p><p><font size="-2"><pre># stap iodevstats.stp -c "sleep 1"</pre></font></p>
+<p> The iodevstats.stp script measures the amount of data successfully read and written by all the executables for each io device on the system. The output is sorted from greatest sum of bytes read and written to a device by an executable to the least. The output contains device major/minor number, the count of operations (reads and writes), the totals and averages for the number of bytes read and written.</p><p><i><a href="io/iodevstats.txt">sample usage in io/iodevstats.txt</i></font></p>
</li><li><a href="io/iostat-scsi.stp">io/iostat-scsi.stp</a> - IO Statistics for SCSI Devices<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#SCSI">SCSI</a> <br>
<p>The iostat-scsi.stp script provides a breakdown of the number of blks read and written on the machine's various SCSI devices. The script takes one argument which is the number of seconds between reports.</p><p><i><a href="io/iostat-scsi.txt">sample usage in io/iostat-scsi.txt</i></font></p>
</li><li><a href="io/iostats.stp">io/iostats.stp</a> - List Executables Reading and Writing the Most Data<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p> The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written.</p><p><font size="-2"><pre># stap iostats.stp -c "sleep 1"</pre></font></p>
+<p> The iostat.stp script measures the amount of data successfully read and written by all the executables on the system. The output is sorted from most greatest sum of bytes read and written by an executable to the least. The output contains the count of operations (opens, reads, and writes), the totals and averages for the number of bytes read and written.</p><p><i><a href="io/iostats.txt">sample usage in io/iostats.txt</i></font></p>
</li><li><a href="io/iotime.stp">io/iotime.stp</a> - Trace Time Spent in Read and Write for Files <br>
keywords: <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#SYSCALL">SYSCALL</a> <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#FILE">FILE</a> <br>
<p>The script watches each open, close, read, and write syscalls on the system. For each file the scripts observes opened it accumulates the amount of wall clock time spent in read and write operations and the number of bytes read and written. When a file is closed the script prints out a pair of lines for the file. Both lines begin with a timestamp in microseconds, the PID number, and the executable name in parentheses. The first line with the "access" keyword lists the file name, the attempted number of bytes for the read and write operations. The second line with the "iotime" keyword list the file name and the number of microseconds accumulated in the read and write syscalls.</p><p><font size="-2"><pre># stap iotime.stp -c "sleep 1"</pre></font></p>
<p>The pf2.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top ten kernel functions with samples.</p><p><i><a href="profiling/pf2.txt">sample usage in profiling/pf2.txt</i></font></p>
</li><li><a href="profiling/pf3.stp">profiling/pf3.stp</a> - Profile Kernel/User Functions<br>
keywords: <a href="keyword-index.html#PROFILING">PROFILING</a> <br>
-<p>The pf3.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty kernel and/or user functions with samples. Use any of --ldd, --all-modules, -d MODULE, -d /PATH/TO/EXEC to add more symbolic info.</p><p><font size="-2"><pre># stap pf3.stp -c "sleep 6" --all-modules --ldd</pre></font></p>
+<p>The pf3.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty kernel and/or user functions with samples. Use any of --ldd, --all-modules, -d MODULE, -d /PATH/TO/EXEC to add more symbolic info.</p><p><i><a href="profiling/pf3.txt">sample usage in profiling/pf3.txt</i></font></p>
</li><li><a href="profiling/pf4.stp">profiling/pf4.stp</a> - Profile Kernel/User Backtraces<br>
keywords: <a href="keyword-index.html#_BEST">_BEST</a> <a href="keyword-index.html#PROFILING">PROFILING</a> <a href="keyword-index.html#BACKTRACE">BACKTRACE</a> <br>
<p>The pf4.stp script sets up time-based sampling. Every five seconds it prints out a sorted list with the top twenty kernel and/or user stack backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d MODULE, -d /PATH/TO/EXEC to add more symbolic info.</p><p><i><a href="profiling/pf4.txt">sample usage in profiling/pf4.txt</i></font></p>
<ul>
<li><a href="memory/vm.tracepoints.stp">memory/vm.tracepoints.stp</a> - Collect Slab Allocation Statistics<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#SLAB">SLAB</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><font size="-2"><pre># stap vm.tracepoints.stp -c "sleep 10"</pre></font></p>
+<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><i><a href="memory/vm.tracepoints.txt">sample usage in memory/vm.tracepoints.txt</i></font></p>
</li></ul>
<h3><a name="SOCKET"><a href="#SOCKET">¶</a> SOCKET</a></h3>
<ul>
-<li><a href="network/connect_stat.stp">network/connect_stat.stp</a> - Show Process Ancestry for IP Connections<br>
+<li><a href="lwtools/accept2close-nd.stp">lwtools/accept2close-nd.stp</a> - Show socket lifespan, from accept() to close() (non-debuginfo)<br>
+keywords: <a href="keyword-index.html#NET">NET</a> <a href="keyword-index.html#SOCKET">SOCKET</a> <br>
+<p>This traces socket duration from the accept() syscall to close(), and provides details on the lifespan of these passive connections, showing the distribution as a histogram.</p><p><i><a href="lwtools/accept2close-nd.txt">sample usage in lwtools/accept2close-nd.txt</i></font></p>
+</li><li><a href="network/connect_stat.stp">network/connect_stat.stp</a> - Show Process Ancestry for IP Connections<br>
keywords: <a href="keyword-index.html#NETWORK">NETWORK</a> <a href="keyword-index.html#SOCKET">SOCKET</a> <a href="keyword-index.html#PROCESS">PROCESS</a> <br>
<p>The connect_stat.stp script prints a task's entire ancestry (parent process name/uid/gid) whenever it attempts an outgoing socket connection to a given IP address.</p><p><font size="-2"><pre># stap connect_stat.stp 127.0.0.1 -c "sleep 1"</pre></font></p>
</li><li><a href="network/dropwatch.stp">network/dropwatch.stp</a> - Watch Where Socket Buffers Are Freed in the Kernel<br>
<p>The script periodically prints a count of specified events and their related tid's over the course of execution. Numerous configuration options exist to control filtering / reporting, see the script source.</p><p><i><a href="general/eventcount.txt">sample usage in general/eventcount.txt</i></font></p>
</li><li><a href="general/func_time_stats.stp">general/func_time_stats.stp</a> - Function Time Statistics<br>
keywords: <a href="keyword-index.html#FUNCTION">FUNCTION</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The func_time_stats.stp script tracks the wall clock time for each invocation of the function probe listed as the first command line argument. When the script exits it prints out the minimum, average, and maximum times in microseconds followed by a count of times that the function was called and a histogram showing the distributions of times.</p><p><font size="-2"><pre># stap func_time_stats.stp 'syscall.nanosleep' -c "sleep 1"</pre></font></p>
+<p>The func_time_stats.stp script tracks the wall clock time for each invocation of the function probe listed as the first command line argument. When the script exits it prints out the minimum, average, and maximum times in microseconds followed by a count of times that the function was called and a histogram showing the distributions of times.</p><p><i><a href="general/func_time_stats.txt">sample usage in general/func_time_stats.txt</i></font></p>
</li><li><a href="general/sizeof.stp">general/sizeof.stp</a> - Print the Size of a C Type<br>
keywords: <a href="keyword-index.html#STATISTICS">STATISTICS</a> <a href="keyword-index.html#MEMORY">MEMORY</a> <br>
-<p>This script prints the size of a type, based on dwarf debuginfo for any kernel or userspace module, or trial-compilation of a given header file name.</p><p><font size="-2"><pre># stap sizeof.stp FILE '</usr/include/stdio.h>'</pre></font></p>
+<p>This script prints the size of a type, based on dwarf debuginfo for any kernel or userspace module, or trial-compilation of a given header file name.</p><p><i><a href="general/sizeof.txt">sample usage in general/sizeof.txt</i></font></p>
</li><li><a href="memory/vm.tracepoints.stp">memory/vm.tracepoints.stp</a> - Collect Slab Allocation Statistics<br>
keywords: <a href="keyword-index.html#MEMORY">MEMORY</a> <a href="keyword-index.html#SLAB">SLAB</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
-<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><font size="-2"><pre># stap vm.tracepoints.stp -c "sleep 10"</pre></font></p>
+<p>The script will probe all memory slab/slub allocations and collects information about the size of the object (bytes requested) and user-space process in execution. When run over a period of time, it helps to correlate kernel-space memory consumption owing to user-space processes.</p><p><i><a href="memory/vm.tracepoints.txt">sample usage in memory/vm.tracepoints.txt</i></font></p>
</li><li><a href="network/nfsd-recent.stp">network/nfsd-recent.stp</a> - Keep track of NFS server statistics<br>
keywords: <a href="keyword-index.html#NFS">NFS</a> <a href="keyword-index.html#STATISTICS">STATISTICS</a> <br>
<p>This script tracks all nfsd server operations by client_ip address, and periodically lists those clients that have made recent requests. It's a way of finding out which nfs clients might be considered still connected.</p><p><font size="-2"><pre># stap nfsd-recent.stp -c "sleep 1"</pre></font></p>
<ul>
<li><a href="io/ttyspy.stp">io/ttyspy.stp</a> - Monitor TTY Typing<br>
keywords: <a href="keyword-index.html#IO">IO</a> <a href="keyword-index.html#TTY">TTY</a> <a href="keyword-index.html#MONITORING">MONITORING</a> <a href="keyword-index.html#GURU">GURU</a> <br>
-<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><font size="-2"><pre># stap --skip-badvars -g ttyspy.stp -c "sleep 1"</pre></font></p>
+<p>The ttyspy.stp script uses tty_audit hooks to monitor recent typing activity on the system, printing a scrolling record of recent keystrokes, on a per-tty basis.</p><p><i><a href="io/ttyspy.txt">sample usage in io/ttyspy.txt</i></font></p>
</li></ul>
<h3><a name="UTILIZATION"><a href="#UTILIZATION">¶</a> UTILIZATION</a></h3>
<ul>
# stap enospc.stp -c "sleep 1"
+lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo)
+keywords: io filesystem
+
+ This dynamically traces two common file system functions:
+ do_sync_read() and do_sync_write(), and reports a histogram
+ distribution of latency. Many, but not all, file systems and
+ workloads use these functions. Tracing their time provides one view
+ of suffered file system latency.
+
+ # stap fslatency-nd.stp 1 1
+
+
+lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo)
+keywords: io filesystem
+
+ This dynamically traces two common file system functions:
+ do_sync_read() and do_sync_write(), and shows details of each call
+ that is slower than a threshold. Many, but not all, file systems and
+ workloads use these functions. Tracing their time provides one view
+ of suffered file system latency.
+
+ # stap fsslower-nd.stp -c "sleep 1"
+
+
= FORMAT =
general/ansi_colors.stp - Color Table for ansi_set_color2() and ansi_set_color3()
# stap --skip-badvars -g ttyspy.stp -c "sleep 1"
+lwtools/biolatency-nd.stp - Measure block I/O latency distribution (non-debuginfo)
+keywords: io
+
+ This measures block I/O latency (storage I/O, ie, disk I/O), and
+ shows the distribution as a histogram. This can be useful to identify
+ the characteristics of I/O latency, beyond the averages shown by
+ iostat(1). For example, to study I/O latency outliers, or multi-modal
+ distributions.
+
+ # stap biolatency-nd.stp 1 1
+
+
+lwtools/bitesize-nd.stp - Measure block I/O size distribution (non-debuginfo)
+keywords: io
+
+ This uses the kernel tracepoint block_rq_insert to read the size of
+ I/O. The output includes the name of the process or thread that was
+ on-CPU when the I/O request was inserted on the issue queue.
+
+ # stap bitesize-nd.stp -c "sleep 1"
+
+
+lwtools/execsnoop-nd.stp - Trace process exec() with command line argument details (non-debuginfo)
+keywords: io
+
+ This can identify if CPU is consumed by short-lived processes, by
+ tracing new process execution. It works by tracing exec() from the
+ fork()->exec() sequence, which means it will not catch new processes
+ that only fork(). It will also show every exec(), including those if
+ a process re-execs.
+
+ # stap execsnoop-nd.stp -c "sleep 1"
+
+
+lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo)
+keywords: io filesystem
+
+ This dynamically traces two common file system functions:
+ do_sync_read() and do_sync_write(), and reports a histogram
+ distribution of latency. Many, but not all, file systems and
+ workloads use these functions. Tracing their time provides one view
+ of suffered file system latency.
+
+ # stap fslatency-nd.stp 1 1
+
+
+lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo)
+keywords: io filesystem
+
+ This dynamically traces two common file system functions:
+ do_sync_read() and do_sync_write(), and shows details of each call
+ that is slower than a threshold. Many, but not all, file systems and
+ workloads use these functions. Tracing their time provides one view
+ of suffered file system latency.
+
+ # stap fsslower-nd.stp -c "sleep 1"
+
+
+lwtools/killsnoop-nd.stp - Trace kill() signals showing process and signal details (non-debuginfo)
+keywords: io
+
+ This traces signals system-wide, including those sent by the kill(1)
+ command, and shows various details.
+
+ # stap killsnoop-nd.stp -c "sleep 1"
+
+
+lwtools/opensnoop-nd.stp - Trace open() syscalls showing filenames (non-debuginfo)
+keywords: io
+
+ This traces the open() syscall system-wide, to show which files are
+ being opened, and by who.
+
+ # stap opensnoop-nd.stp -c "sleep 1"
+
+
+lwtools/rwtime-nd.stp - Summarize read() and write() syscall latency (non-debuginfo)
+keywords: io
+
+ This traces read() and write() syscalls, producing a histogram
+ summary of their durations (aka latencies).
+
+ # stap rwtime-nd.stp -c "sleep 1"
+
+
+lwtools/syscallbypid-nd.stp - Count syscalls with process details (non-debuginfo)
+keywords: io
+
+ This traces syscalls system-wide, and produces a summary report
+ showing their counts by process ID, process name, and syscall types.
+
+ # stap syscallbypid-nd.stp -c "sleep 1"
+
+
process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations
keywords: io scheduler backtrace
# stap sleeptime.stp -c "sleep 1"
+= NET =
+
+lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo)
+keywords: net socket
+
+ This traces socket duration from the accept() syscall to close(), and
+ provides details on the lifespan of these passive connections,
+ showing the distribution as a histogram.
+
+ # stap accept2close-nd.stp -c "sleep 1"
+
+
= NETWORK =
network/autofs4.stp - Watch autofs4 Operations
= SOCKET =
+lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo)
+keywords: net socket
+
+ This traces socket duration from the accept() syscall to close(), and
+ provides details on the lifespan of these passive connections,
+ showing the distribution as a histogram.
+
+ # stap accept2close-nd.stp -c "sleep 1"
+
+
network/connect_stat.stp - Show Process Ancestry for IP Connections
keywords: network socket process
--- /dev/null
+% stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd
+
+free #187843928538773: stap[1377], pointer 0x8e07fa0:
+free+0x1d [libc-2.12.so]
+_ZdlPv+0x22 [libstdc++.so.6.0.13]
+_ZN17systemtap_sessionD1Ev+0xe5 [stap]
+main+0xde3 [stap]
+
+free #187843928596932: stap[1377], pointer 0x8e0a6c0:
+free+0x1d [libc-2.12.so]
+_ZdlPv+0x22 [libstdc++.so.6.0.13]
+_ZNSs4_Rep10_M_destroyERKSaIcE+0x1e [libstdc++.so.6.0.13]
+_ZNSsD2Ev+0x4e [libstdc++.so.6.0.13]
+_ZN17systemtap_sessionD1Ev+0x368 [stap]
+main+0xde3 [stap]
+
+free #187843928647723: stap[1377], pointer 0x8e08138:
+free+0x1d [libc-2.12.so]
+_ZdlPv+0x22 [libstdc++.so.6.0.13]
+_ZNSs4_Rep10_M_destroyERKSaIcE+0x1e [libstdc++.so.6.0.13]
+_ZNSsD2Ev+0x4e [libstdc++.so.6.0.13]
+_ZN17systemtap_sessionD1Ev+0x398 [stap]
+main+0xde3 [stap]
+
+[...]
--- /dev/null
+# stap numa_faults.stp
+
+Starting pagefault counters
+^C
+Printing counters:
+
+Execname PID RD Faults WR Faults Node:Faults
+======================= ========== ========== =============
+systemd-journal 587 0 107
+firefox 25587 1 676
+memcheck-amd64- 16210 0 7762
+pmcd 29170 0 238
+systemd-udevd 29519 22 29
+systemd-udevd 1199 0 23
+zsh 24943 0 76
+zsh 29525 32 71
+hostname 29526 62 53
+cut 29527 70 55
+emacs 29528 914 5252
+xfce4-terminal 24458 0 5
+pmlogger 29202 10 0
+systemd 1 0 162
+rm 29532 132 750
+sh 29533 95 754
+pmlogcheck 29534 122 165
+sh 29535 95 752
+pmlogcheck 29536 126 165
+pmlogger 29537 208 934
+modprobe 29539 54 45
+xfce4-panel 24165 0 1869
+epiphany 29540 1613 15552
+dbus-daemon 24298 0 3
+dbus-daemon 24018 0 24
+epiphany 29549 114 44
+WebKitNetworkPr 29550 1355 25654
+dbus-daemon 835 0 9
+epiphany 29558 114 50
+WebKitWebProces 29559 1741 10835
+Xorg.bin 24002 1600 0
+gvfsd-http 7374 3 7
+
--- /dev/null
+# stap pfaults.stp
+
+461:3407:2830640d:r:minor:55
+494:3407:2858244d:r:minor:16
+522:3407:2897376d:r:minor:16
+553:3407:3481840d:r:minor:15
+582:3407:4219988d:r:minor:16
+609:3407:3344800d:r:minor:15
+636:3407:3323072d:r:minor:15
+663:3407:3328112d:r:minor:15
+1083:3407:3221225457d:w:minor:44
+1880:3407:2807864d:w:minor:51
+1946:3407:3218071435d:w:minor:43
+2017:3407:2680912d:r:minor:19
+2042:3407:2778107d:r:minor:15
+2086:3407:2805420d:w:minor:32
+2127:3407:2682880d:r:minor:31
+2155:3407:2717264d:r:minor:15
+2181:3407:2767040d:r:minor:15
+2207:3407:2781652d:r:minor:15
--- /dev/null
+# stap vm.tracepoints.stp
+
+Process:swapper
+Slab_size:240 Count:2
+Slab_size:192 Count:433
+
+Process:pmcd
+Slab_size:384 Count:4
+Slab_size:192 Count:8
+
+Process:swapper
+Slab_size:128 Count:2
+
+Process:stapio
+Slab_size:144 Count:1
+
+Process:crond
+Slab_size:128 Count:8
+
+Process:pmcd
+Slab_size:56 Count:1
+
+Process:crond
+Slab_size:24 Count:8
+
+Process:pmcd
+Slab_size:448 Count:2
+
+Process:swapper
+Slab_size:1280 Count:2
+
+Process:pmcd
+Slab_size:128 Count:3
+Slab_size:24 Count:1
+Slab_size:644 Count:1
+
+Process:stapio
+Slab_size:4096 Count:3
--- /dev/null
+# stap pf3.stp -c "sleep 6" --all-modules --ldd
+
+--- 831 samples recorded:
+k:kernel:native_safe_halt 818
+k:kernel:_spin_unlock_irqrestore 3
+k:kernel:drain_array 2
+k:kernel:flush_old_exec 1
+k:kernel:native_flush_tlb_single 1
+k:kernel:handle_pte_fault 1
+u:/lib/libc-2.12.so:0x3d144e 1
+k:kernel:finish_task_switch 1
+k:kernel:copy_from_user 1
+k:kernel:run_timer_softirq 1
+u:<unknown>:0x47d3f4 1
git://sourceware.org / systemtap.git / commitdiff