2.3. Running SystemTap Scripts

⁠2.3. Running SystemTap Scripts

SystemTap is distributed with a number of command line tools that allow you to monitor the activities of the system. The stap command reads probing instructions from a SystemTap script, translates these instructions into C code, builds a kernel module, and loads it into the running Linux kernel. The staprun command runs SystemTap instrumentation, that is, a kernel module built from SystemTap scripts during a cross-instrumentation.

Running stap and staprun requires elevated privileges to the system. Because not all users can be granted root access just to run SystemTap, you can allow a non-privileged user to run SystemTap instrumentation on their machine by adding them to one of the following user groups:

stapdev: Members of this group can use the stap command to run SystemTap scripts, or staprun to run SystemTap instrumentation modules.
Running the stap command involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This operation requires elevated privileges to the system, which are granted to stapdev members. Unfortunately, such privileges also grant effective root access to stapdev members. As a consequence, only grant stapdev group membership to users whom you can trust with root access.
stapusr: Members of this group can only use the staprun command to run SystemTap instrumentation modules. In addition, they can only run modules from the /lib/modules/kernel_version/systemtap/ directory. Note that this directory must be owned only by the root user, and must only be writable by the root user.

The stap command reads a SystemTap script either from a file, or from standard input. To tell stap to read a SystemTap script from a file, specify the file name on the command line:

stap file_name

To instruct stap to read a SystemTap script from standard input, use the - switch instead of the file name. Note that any command-line options you wish to use must be inserted before the - switch. For example, to make the output of the stap command more verbose, type:

echo "probe timer.s(1) {exit()}" | stap -v -

Below is a list of commonly used stap options:

-v

Makes the output of the SystemTap session more verbose. You can repeat this option multiple times to provide more details on the script's execution, for example:

stap -vvv script.stp

This option is particularly useful if you encounter any errors in running the script. For more information about common SystemTap script errors, refer to Chapter 6, Understanding SystemTap Errors.

-o file_name

Sends the standard output to a file named file_name.

-S size,count

Limits the maximum size of output files to size megabytes and the maximum number of stored files to count. This option implements logrotate operations for SystemTap and the resulting file names have a sequence number suffix.

-x process_id

Sets the SystemTap handler function target() to the specified process ID. For more information about target(), refer to SystemTap Functions.

-c 'command'

Sets the SystemTap handler function target() to the specified command and runs the SystemTap instrumentation for the duration of this command. For more information about target(), refer to SystemTap Functions.

-e 'script'

Uses script rather than a file as input for the SystemTap translator.

-F

Uses SystemTap's flight recorder mode and makes the script a background process. For more information about flight recorder mode, refer to Section 2.3.1, “SystemTap Flight Recorder Mode”.

For more information about the stap command, refer to the stap(1) man page. For more information about the staprun command and cross-instrumentation, refer to Section 2.2, “Generating Instrumentation for Other Computers” or the staprun(8) man page.

⁠2.3.1. SystemTap Flight Recorder Mode

SystemTap's flight recorder mode allows you to run a SystemTap script for long periods of time and just focus on recent output. The flight recorder mode limits the amount of output generated.

There are two variations of the flight recorder mode: in-memory and file mode. In both cases, the SystemTap script runs as a background process.

⁠2.3.1.1. In-memory Flight Recorder

When flight recorder mode is used without a file name, SystemTap uses a buffer in kernel memory to store the output of the script. Once the SystemTap instrumentation module is loaded and the probes start running, the instrumentation detaches and is put in the background. When the interesting event occurs, you can reattach to the instrumentation to see the recent output in the memory buffer and any continuing output.

To run a SystemTap script by using the flight recorder in-memory mode, run the stap command with the -F command line option:

stap -F iotime.stp

Once the script starts, stap prints a message similar to the following to provide you with the command to reconnect to the running script:

Disconnecting from systemtap module.
To reconnect, type "staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556"

When the interesting event occurs, run the following command to connect to the currently running script, output the recent data in the memory buffer, and get continuing output:

staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556

By default, the kernel buffer is 1MB in size. You can increase this value by using the -s option with the size in megabytes (rounded up to the next power over 2) for the buffer. For example, -s2 on the SystemTap command line would specify 2MB for the buffer.

⁠2.3.1.2. File Flight Recorder

The flight recorder mode can also store data to files. You can control the number and size of the files kept by using the -S option followed by two numerical arguments separated by a comma: the first argument is the maximum size in megabytes for the each output file, the second argument is the number of recent files to keep. To specify the file name, use the -o option followed by the name. SystemTap automatically adds a number suffix to the file name to indicate the order of the files.

The following command starts SystemTap in file flight recorder mode with the output going to files named /tmp/pfaults.log.[0-9]+, each file 1MB or smaller, and keeping latest two files:

stap -F -o /tmp/pfaults.log -S 1,2  pfaults.stp

The command prints the process ID to standard output. Sending a SIGTERM to the process terminates the SystemTap script and stops the data collection. For example, if the previous command listed 7590 as the process ID, the following command would stop the SystemTap script:

kill -s SIGTERM 7590

In this example, only the most recent two files generated by the script are kept: SystemTap automatically removes older files. As a result, the ls -sh /tmp/pfaults.log.* command lists two files:

1020K /tmp/pfaults.log.5    44K /tmp/pfaults.log.6

To examine the latest data, read the file with the highest number, in this case /tmp/pfaults.log.6.