Preface

Abstract

- This guide provides basic instructions on how to use SystemTap to monitor different subsystems of Fedora in finer detail. The SystemTap Beginners Guide is recommended for users who have taken RHCT or have a similar level of expertise in Fedora. -

Index

Symbols

$count

sample usage

local variables, Tracking I/O Time For Each File Read or Write

$return

sample usage

local variables, Summarizing Disk Read/Write Traffic, Track Cumulative IO

@avg (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@count (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@max (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@min (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@sum (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

A

adding values to statistical aggregates

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

advantages of cross-instrumentation, Generating Instrumentation for Other Computers

aggregate element not found

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

aggregates (statistical)

array operations, Computing for Statistical Aggregates

aggregation overflow

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

algebraic formulas using arrays

reading values from arrays

array operations, Reading Values From Arrays

architecture notation, determining, Generating Instrumentation for Other Computers

architecture of SystemTap, Architecture

array locals not supported

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

array operations

assigning associated values, Assigning an Associated Value

associating timestamps to process names, Assigning an Associated Value

associative arrays, Array Operations in SystemTap

clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements

delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements

computing for statistical aggregates, Computing for Statistical Aggregates

@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates

conditional statements, using arrays in, Using Arrays in Conditional Statements

testing for array membership, Using Arrays in Conditional Statements

deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements

incrementing associated values, Incrementing Associated Values

tallying virtual file system reads (VFS reads), Incrementing Associated Values

multiple elements in an array, Processing Multiple Elements in an Array

processing multiple elements in an array, Processing Multiple Elements in an Array

cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array

reading values from arrays, Reading Values From Arrays

computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays

arrays, Associative Arrays

(see also associative arrays)

assigning associated values

array operations, Assigning an Associated Value

associating timestamps to process names, Assigning an Associated Value

associating timestamps to process names

array operations, Assigning an Associated Value

associated values

introduction

arrays, Associative Arrays

associating timestamps to process names

assigning associated values

array operations, Assigning an Associated Value

associative arrays

introduction, Associative Arrays

associated values, Associative Arrays
example, Associative Arrays
index expression, Associative Arrays
key pairs, Associative Arrays
syntax, Associative Arrays
unique keys, Associative Arrays

asynchronous events

Events, Event

B

begin

Events, Event

building instrumentation modules from SystemTap scripts, Generating Instrumentation for Other Computers

building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers

C

call graph tracing

examples of SystemTap scripts, Call Graph Tracing

capabilities of SystemTap

Introduction, SystemTap Capabilities

changes to file attributes, monitoring

examples of SystemTap scripts, Monitoring Changes to File Attributes

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements

command-line arguments

SystemTap handler constructs

handlers, Command-Line Arguments

compiling instrumentation/kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers

components

SystemTap scripts

introduction, SystemTap Scripts

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates

computing for timestamp deltas

reading values from arrays

array operations, Reading Values From Arrays

conditional operators

conditional statements

handlers, Conditional Statements

conditional statements, using arrays in

array operations, Using Arrays in Conditional Statements

testing for array membership, Using Arrays in Conditional Statements

CONFIG_HZ, computing for, Variables

contended user-space locks (futex contentions), identifying

examples of SystemTap scripts, Identifying Contended User-Space Locks

copy fault

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

count operator

computing for statistical aggregates

array (operator), Computing for Statistical Aggregates

counting function calls

examples of SystemTap scripts, Counting Function Calls Made

CPU ticks

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

cpu()

functions, Systemtap Handler/Body

cross-compiling, Generating Instrumentation for Other Computers

cross-instrumentation

advantages of, Generating Instrumentation for Other Computers

building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers

configuration

host system and target system, Generating Instrumentation for Other Computers

generating instrumentation from SystemTap scripts, Generating Instrumentation for Other Computers

host system, Generating Instrumentation for Other Computers

instrumentation module, Generating Instrumentation for Other Computers

target kernel, Generating Instrumentation for Other Computers

target system, Generating Instrumentation for Other Computers

ctime()

functions, Systemtap Handler/Body

ctime(), example of usage

script examples, Summarizing Disk Read/Write Traffic

cumulative I/O, tracking

examples of SystemTap scripts, Track Cumulative IO

cumulative virtual file system reads, tallying

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

D

delete operator

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

determining architecture notation, Generating Instrumentation for Other Computers

determining the kernel version, Installing Required Kernel Information RPMs

determining time spent in kernel and user space

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

device I/O, monitoring

examples of SystemTap scripts, I/O Monitoring (By Device)

device number of a file (integer format)

examples of SystemTap scripts, Monitoring Reads and Writes to a File

disk I/O traffic, summarizing

script examples, Summarizing Disk Read/Write Traffic

division by 0

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

documentation goals

Introduction, Documentation Goals

E

embedded code in unprivileged script

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

empty unique keys

reading values from arrays

array operations, Reading Values From Arrays

end

Events, Event

errors

parse/semantics error, Parse and Semantic Errors

embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors

runtime errors/warnings, Run Time Errors and Warnings

aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings

event types

Understanding How SystemTap Works, Understanding How SystemTap Works

Events

asynchronous events, Event
begin, Event
end, Event
examples of synchronous and asynchronous events, Event
introduction, Event
kernel.function("function"), Event
kernel.trace("tracepoint"), Event
module("module"), Event
synchronous events, Event
syscall.system_call, Event
timer events, Event
user-space, User-Space Events
vfs.file_operation, Event
wildcards, Event

events and handlers, Understanding How SystemTap Works

events wildcards, Event

example

introduction

arrays, Associative Arrays

example of multiple command-line arguments

examples of SystemTap scripts, Call Graph Tracing

examples of synchronous and asynchronous events

Events, Event

examples of SystemTap scripts, Useful SystemTap Scripts

call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex (lock) contentions, Identifying Contended User-Space Locks
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)

exceeded MAXACTION

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

exceeded MAXNESTING

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

exit()

functions, Systemtap Handler/Body

expected symbol/array index expression

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

extracting data collected by statistical aggregates

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

F

fedoradebugurl.sh, Installing Required Kernel Information RPMs

feedback

contact information for this manual, We Need Feedback!

file attributes, monitoring changes to

examples of SystemTap scripts, Monitoring Changes to File Attributes

file device number (integer format)

examples of SystemTap scripts, Monitoring Reads and Writes to a File

file reads/writes, monitoring

examples of SystemTap scripts, Monitoring Reads and Writes to a File

flight recorder mode, SystemTap Flight Recorder Mode

file mode, File Flight Recorder
in-memory mode, In-memory Flight Recorder

for loops

conditional statements

handlers, Conditional Statements

foreach

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

format

introduction

arrays, Associative Arrays

format and syntax

printf(), Systemtap Handler/Body

SystemTap handler constructs

handlers, Variables

SystemTap scripts

introduction, SystemTap Scripts

format specifiers

printf(), Systemtap Handler/Body

format strings

printf(), Systemtap Handler/Body

function call (unresolved)

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

function calls (incoming/outgoing), tracing

examples of SystemTap scripts, Call Graph Tracing

function calls, tallying

examples of SystemTap scripts, Counting Function Calls Made

functions, Systemtap Handler/Body

cpu(), Systemtap Handler/Body

ctime(), Systemtap Handler/Body

gettimeofday_s(), Systemtap Handler/Body

pp(), Systemtap Handler/Body

SystemTap scripts

introduction, SystemTap Scripts

target(), Systemtap Handler/Body

thread_indent(), Systemtap Handler/Body

tid(), Systemtap Handler/Body

uid(), Systemtap Handler/Body

functions (used in handlers)

exit(), Systemtap Handler/Body

functions called in network socket code, tracing

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

futex (lock) contentions

examples of SystemTap scripts, Identifying Contended User-Space Locks

futex contention, definition

examples of SystemTap scripts, Identifying Contended User-Space Locks

futex contentions, identifying

examples of SystemTap scripts, Identifying Contended User-Space Locks

futex system call

examples of SystemTap scripts, Identifying Contended User-Space Locks

G

gettimeofday_s()

functions, Systemtap Handler/Body

global

SystemTap handler constructs

handlers, Variables

goals, documentation

Introduction, Documentation Goals

grammatical/typographical script error

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

guru mode

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

H

handler functions, Systemtap Handler/Body

handlers

conditional statements, Conditional Statements

conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements

introduction, Systemtap Handler/Body

SystemTap handler constructs, Basic SystemTap Handler Constructs

command-line arguments, Command-Line Arguments
global, Variables
syntax and format, Basic SystemTap Handler Constructs
variable notations, Command-Line Arguments
variables, Variables

target variables, Target Variables

handlers and events, Understanding How SystemTap Works

SystemTap scripts

introduction, SystemTap Scripts

heaviest disk reads/writes, identifying

script examples, Summarizing Disk Read/Write Traffic

host system

cross-instrumentation, Generating Instrumentation for Other Computers

host system and target system

cross-instrumentation

configuration, Generating Instrumentation for Other Computers

I

I/O block time, monitoring

examples of SystemTap scripts, Periodically Print I/O Block Time

I/O monitoring (by device)

examples of SystemTap scripts, I/O Monitoring (By Device)

I/O time, monitoring

examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write

I/O traffic, summarizing

script examples, Summarizing Disk Read/Write Traffic

identifier type mismatch

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

identifying contended user-space locks (i.e. futex contentions)

examples of SystemTap scripts, Identifying Contended User-Space Locks

identifying heaviest disk reads/writes

script examples, Summarizing Disk Read/Write Traffic

if/else

conditional statements

handlers, Conditional Statements

if/else conditionals, alternative syntax

examples of SystemTap scripts, Network Profiling

if/else statements, using arrays in

array operations, Using Arrays in Conditional Statements

incoming TCP connections, monitoring

examples of SystemTap scripts, Monitoring Incoming TCP Connections

incoming/outgoing function calls, tracing

examples of SystemTap scripts, Call Graph Tracing

incrementing associated values

array operations, Incrementing Associated Values

tallying virtual file system reads (VFS reads), Incrementing Associated Values

index expression

introduction

arrays, Associative Arrays

initial testing, Initial Testing

inode number

examples of SystemTap scripts, Monitoring Reads and Writes to a File

Installation

fedoradebugurl.sh, Installing Required Kernel Information RPMs
initial testing, Initial Testing
kernel information packages, Installing Required Kernel Information RPMs
kernel version, determining the, Installing Required Kernel Information RPMs
required packages, Installing Required Kernel Information RPMs
Setup and Installation, Installation and Setup
systemtap package, Installing SystemTap
systemtap-runtime package, Installing SystemTap

instrumentation module

cross-instrumentation, Generating Instrumentation for Other Computers

instrumentation modules from SystemTap scripts, building, Generating Instrumentation for Other Computers

integer extractors

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

Introduction

capabilities of SystemTap, SystemTap Capabilities
documentation goals, Documentation Goals
goals, documentation, Documentation Goals
limitations of SystemTap, SystemTap Capabilities
performance monitoring, Introduction

invalid division

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

invalid values to variables/arrays

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

iterations, processing elements in an array as

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

K

kernel and user space, determining time spent in

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

kernel information packages, Installing Required Kernel Information RPMs

kernel modules from SystemTap scripts, building, Generating Instrumentation for Other Computers

kernel version, determining the, Installing Required Kernel Information RPMs

kernel.function("function")

Events, Event

kernel.trace("tracepoint")

Events, Event

key pairs

introduction

arrays, Associative Arrays

L

libdwfl failure

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

limitations of SystemTap

Introduction, SystemTap Capabilities

limiting the output of foreach

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

local variables

name, Systemtap Handler/Body

sample usage

$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO

M

MAXACTION exceeded

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

MAXNESTING exceeded

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

membership (in array), testing for

conditional statements, using arrays in

array operations, Using Arrays in Conditional Statements

module("module")

Events, Event

monitoring changes to file attributes

examples of SystemTap scripts, Monitoring Changes to File Attributes

monitoring cumulative I/O

examples of SystemTap scripts, Track Cumulative IO

monitoring device I/O

examples of SystemTap scripts, I/O Monitoring (By Device)

monitoring I/O block time

examples of SystemTap scripts, Periodically Print I/O Block Time

monitoring I/O time

examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write

monitoring incoming TCP connections

examples of SystemTap scripts, Monitoring Incoming TCP Connections

monitoring polling applications

examples of SystemTap scripts, Monitoring Polling Applications

monitoring reads and writes to a file

examples of SystemTap scripts, Monitoring Reads and Writes to a File

monitoring system calls

examples of SystemTap scripts, Tracking Most Frequently Used System Calls

monitoring system calls (volume per process)

examples of SystemTap scripts, Tracking System Call Volume Per Process

monitoring TCP packets

examples of SystemTap scripts, Monitoring TCP Packets

multiple array operations within the same probe

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

multiple command-line arguments, example of

examples of SystemTap scripts, Call Graph Tracing

multiple elements in an array

array operations, Processing Multiple Elements in an Array

N

name

local variables, Systemtap Handler/Body

net/socket.c, tracing functions from

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

network profiling

examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel

network socket code, tracing functions called in

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

network traffic, monitoring

examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel

no match for probe point

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

non-global arrays

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

number of errors: N, skipped probes: M

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

O

operations

assigning associated values

associating timestamps to process names, Assigning an Associated Value

associative arrays, Array Operations in SystemTap

clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements

delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements

computing for statistical aggregates, Computing for Statistical Aggregates

@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates

conditional statements, using arrays in, Using Arrays in Conditional Statements

testing for array membership, Using Arrays in Conditional Statements

deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements

incrementing associated values, Incrementing Associated Values

tallying virtual file system reads (VFS reads), Incrementing Associated Values

multiple elements in an array, Processing Multiple Elements in an Array

processing multiple elements in an array, Processing Multiple Elements in an Array

cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array

reading values from arrays, Reading Values From Arrays

computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays

options, stap

Usage, Running SystemTap Scripts

ordering the output of foreach

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

overflow of aggregation

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

P

packages required to run SystemTap, Installing Required Kernel Information RPMs

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors

performance monitoring

Introduction, Introduction

pointer dereference fault

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

polling applications, monitoring

examples of SystemTap scripts, Monitoring Polling Applications

pp()

functions, Systemtap Handler/Body

printf()

format specifiers, Systemtap Handler/Body
format strings, Systemtap Handler/Body
syntax and format, Systemtap Handler/Body

printing I/O activity (cumulative)

examples of SystemTap scripts, Track Cumulative IO

printing I/O block time (periodically)

examples of SystemTap scripts, Periodically Print I/O Block Time

probe mismatch

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

probe point (no match for)

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

probes

SystemTap scripts

introduction, SystemTap Scripts

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

cumulative virtual file system reads, tallying

array operations, Processing Multiple Elements in an Array

foreach

array operations, Processing Multiple Elements in an Array

limiting the output of foreach

array operations, Processing Multiple Elements in an Array

ordering the output of foreach

array operations, Processing Multiple Elements in an Array

profiling the network

examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel

R

reading values from arrays

array operations, Reading Values From Arrays

empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays

computing for timestamp deltas

array operations, Reading Values From Arrays

reads/writes to a file, monitoring

examples of SystemTap scripts, Monitoring Reads and Writes to a File

required packages, Installing Required Kernel Information RPMs

RPMs required to run SystemTap, Installing Required Kernel Information RPMs

running scripts from standard input, Running SystemTap Scripts

running SystemTap scripts

Usage, Running SystemTap Scripts

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings

S

script examples

call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex (lock) contentions, Identifying Contended User-Space Locks
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)

scripts

introduction, SystemTap Scripts

components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts

sessions, SystemTap, Architecture

Setup and Installation, Installation and Setup

Stack backtrace

user-space, User-Space Stack Backtraces

standard input, running scripts from

Usage, Running SystemTap Scripts

stap

Usage, Running SystemTap Scripts

stap options, Running SystemTap Scripts

stapdev

Usage, Running SystemTap Scripts

staprun

Usage, Running SystemTap Scripts

stapusr

Usage, Running SystemTap Scripts

stat -c, determining file device number (integer format)

examples of SystemTap scripts, Monitoring Reads and Writes to a File

stat -c, determining whole device number

examples of SystemTap scripts, I/O Monitoring (By Device)

statement blocks

SystemTap scripts

introduction, SystemTap Scripts

statistical aggregates

array operations, Computing for Statistical Aggregates

summarizing disk I/O traffic

script examples, Summarizing Disk Read/Write Traffic

synchronous events

Events, Event

syntax

introduction

arrays, Associative Arrays

syntax and format

printf(), Systemtap Handler/Body

SystemTap handler constructs

handlers, Basic SystemTap Handler Constructs

SystemTap scripts

introduction, SystemTap Scripts

syscall.system_call

Events, Event

system calls volume (per process), monitoring

examples of SystemTap scripts, Tracking System Call Volume Per Process

system calls, monitoring

examples of SystemTap scripts, Tracking Most Frequently Used System Calls

SystemTap architecture, Architecture

SystemTap handlers

SystemTap handler constructs, Basic SystemTap Handler Constructs

syntax and format, Basic SystemTap Handler Constructs

systemtap package, Installing SystemTap

SystemTap script functions, Systemtap Handler/Body

SystemTap scripts

introduction, SystemTap Scripts

components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts

useful examples, Useful SystemTap Scripts

SystemTap scripts, how to run, Running SystemTap Scripts

SystemTap sessions, Architecture

SystemTap statements

conditional statements, Conditional Statements

conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements

SystemTap handler constructs

command-line arguments, Command-Line Arguments
global, Variables
variable notations, Command-Line Arguments
variables, Variables

systemtap-runtime package, Installing SystemTap

systemtap-testsuite package

sample scripts, Useful SystemTap Scripts

T

tallying function calls

examples of SystemTap scripts, Counting Function Calls Made

tallying virtual file system reads (VFS reads)

incrementing associated values

array operations, Incrementing Associated Values

Tapsets

definition of, Tapsets

target kernel

cross-instrumentation, Generating Instrumentation for Other Computers

target system

cross-instrumentation, Generating Instrumentation for Other Computers

target system and host system

configuration, Generating Instrumentation for Other Computers

target variables, Target Variables

pretty printing, Pretty Printing Target Variables
typecasting, Typecasting
user-space, Accessing User-Space Target Variables
variable availability, Checking Target Variable Availablility

target()

functions, Systemtap Handler/Body

target-symbol expression, unresolved

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

TCP connections (incoming), monitoring

examples of SystemTap scripts, Monitoring Incoming TCP Connections

TCP packets, monitoring

examples of SystemTap scripts, Monitoring TCP Packets

testing for array membership

conditional statements, using arrays in

array operations, Using Arrays in Conditional Statements

testing, initial, Initial Testing

thread_indent()

functions, Systemtap Handler/Body

thread_indent(), sample usage

examples of SystemTap scripts, Call Graph Tracing

tid()

functions, Systemtap Handler/Body

time of I/O

examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write

time spent in kernel/user space, determining

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

timer events

Events, Event

timer.ms(), sample usage

examples of SystemTap scripts, Counting Function Calls Made

timer.s(), sample usage

examples of SystemTap scripts, Monitoring Polling Applications, Tracking Most Frequently Used System Calls

timestamp deltas, computing for

reading values from arrays

array operations, Reading Values From Arrays

timestamps, association thereof to process names

assigning associated values

array operations, Assigning an Associated Value

tracepoint, Event, Monitoring Network Packets Drops in Kernel

tracing call graph

examples of SystemTap scripts, Call Graph Tracing

tracing functions called in network socket code

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

tracing incoming/outgoing function calls

examples of SystemTap scripts, Call Graph Tracing

tracking cumulative I/O

examples of SystemTap scripts, Track Cumulative IO

trigger function

examples of SystemTap scripts, Call Graph Tracing

type mismatch for identifier

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

typographical script error

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

U

uid()

functions, Systemtap Handler/Body

uname -m, Generating Instrumentation for Other Computers

uname -r, Installing Required Kernel Information RPMs

understainding SystemTap errors

runtime errors/warnings, Run Time Errors and Warnings

aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings

Understanding How SystemTap Works, Understanding How SystemTap Works

architecture, Architecture
event types, Understanding How SystemTap Works
events and handlers, Understanding How SystemTap Works
SystemTap sessions, Architecture

understanding SystemTap errors

parse/semantics error, Parse and Semantic Errors

embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors

unique keys

introduction

arrays, Associative Arrays

unprivileged script, embedded code in

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unresolved function call

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unresolved target-symbol expression

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unresolved type for identifier

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unsafe embedded code in unprivileged script

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

Usage

options, stap, Running SystemTap Scripts
running SystemTap scripts, Running SystemTap Scripts
standard input, running scripts from, Running SystemTap Scripts
stap, Running SystemTap Scripts
stapdev, Running SystemTap Scripts
staprun, Running SystemTap Scripts
stapusr, Running SystemTap Scripts

useful examples of SystemTap scripts, Useful SystemTap Scripts

user and kernel space, determining time spent in

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

using arrays in simple computations

reading values from arrays

array operations, Reading Values From Arrays

Using SystemTap, Using SystemTap

usrdev2kerndev()

examples of SystemTap scripts, I/O Monitoring (By Device)

V

values, assignment of

array operations, Assigning an Associated Value

variable modified during 'foreach'

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

variable notations

SystemTap handler constructs

handlers, Command-Line Arguments

variables

SystemTap handler constructs

handlers, Variables

variables (local)

name, Systemtap Handler/Body

sample usage

$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO

VFS reads, tallying of

incrementing associated values

array operations, Incrementing Associated Values

vfs.file_operation

Events, Event

virtual file system reads (cumulative), tallying

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

virtual file system reads (non-cumulative), tallying

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

W

while loops

conditional statements

handlers, Conditional Statements

whole device number (usage as a command-line argument)

examples of SystemTap scripts, I/O Monitoring (By Device)

wildcards in events, Event

writes/reads to a file, monitoring

examples of SystemTap scripts, Monitoring Reads and Writes to a File

3.2.Â SystemTap Scripts

- For the most part, SystemTap scripts are the foundation of each SystemTap session. SystemTap scripts instruct SystemTap on what type of information to collect, and what to do once that information is collected. -

- As stated in ChapterÂ 3, Understanding How SystemTap Works, SystemTap scripts are made up of two components: events and handlers. Once a SystemTap session is underway, SystemTap monitors the operating system for the specified events and executes the handlers as they occur. -

Note

- An event and its corresponding handler is collectively called a probe. A SystemTap script can have multiple probes. -

- A probe's handler is commonly referred to as a probe body. -

- In terms of application development, using events and handlers is similar to instrumenting the code by inserting diagnostic print statements in a program's sequence of commands. These diagnostic print statements allow you to view a history of commands executed once the program is run. -

- SystemTap scripts allow insertion of the instrumentation code without recompilation of the code and allows more flexibility with regard to handlers. Events serve as the triggers for handlers to run; handlers can be specified to record specified data and print it in a certain manner. -

Format

- SystemTap scripts use the file extension .stp, and contains probes written in the following format: -

probe	event {statements}

- SystemTap supports multiple events per probe; multiple events are delimited by a comma (,). If multiple events are specified in a single probe, SystemTap will execute the handler when any of the specified events occur. -

- Each probe has a corresponding statement block. This statement block is enclosed in braces ({ }) and contains the statements to be executed per event. SystemTap executes these statements in sequence; special separators or terminators are generally not necessary between multiple statements. -

Note

- Statement blocks in SystemTap scripts follow the same syntax and semantics as the C programming language. A statement block can be nested within another statement block. -

- Systemtap allows you to write functions to factor out code to be used by a number of probes. Thus, rather than repeatedly writing the same series of statements in multiple probes, you can just place the instructions in a function, as in: -

function function_name(arguments) {statements}
-probe event {function_name(arguments)}

- The statements in function_name are executed when the probe for event executes. The arguments are optional values passed into the function. -

Important

- SectionÂ 3.2, âSystemTap Scriptsâ is designed to introduce readers to the basics of SystemTap scripts. To understand SystemTap scripts better, it is advisable that you refer to ChapterÂ 5, Useful SystemTap Scripts; each section therein provides a detailed explanation of the script, its events, handlers, and expected output. -

3.2.1.Â Event

- SystemTap events can be broadly classified into two types: synchronous and asynchronous. -

Synchronous Events

- A synchronous event occurs when any process executes an instruction at a particular location in kernel code. This gives other events a reference point from which more contextual data may be available. -

- Examples of synchronous events include: -

syscall.system_call

- The entry to the system call system_call. If the exit from a syscall is desired, appending a .return to the event monitor the exit of the system call instead. For example, to specify the entry and exit of the system call close, use syscall.close and syscall.close.return respectively. -

vfs.file_operation

- The entry to the file_operation event for Virtual File System (VFS). Similar to syscall event, appending a .return to the event monitors the exit of the file_operation operation. -

kernel.function("function")

- The entry to the kernel function function. For example, kernel.function("sys_open") refers to the "event" that occurs when the kernel function sys_open is called by any thread in the system. To specify the return of the kernel function sys_open, append the return string to the event statement; i.e. kernel.function("sys_open").return. -

- When defining probe events, you can use asterisk (*) for wildcards. You can also trace the entry or exit of a function in a kernel source file. Consider the following example: -

ExampleÂ 3.1.Â wildcards.stp

probe kernel.function("*@net/socket.c") { }
-probe kernel.function("*@net/socket.c").return { }

- In the previous example, the first probe's event specifies the entry of ALL functions in the kernel source file net/socket.c. The second probe specifies the exit of all those functions. Note that in this example, there are no statements in the handler; as such, no information will be collected or displayed. -

kernel.trace("tracepoint")

- The static probe for tracepoint. Recent kernels (2.6.30 and newer) include instrumentation for specific events in the kernel. These events are statically marked with tracepoints. One example of a tracepoint available in systemtap is kernel.trace("kfree_skb") which indicates each time a network buffer is freed in the kernel. -

module("module").function("function")

- Allows you to probe functions within modules. For example: -

ExampleÂ 3.2.Â moduleprobe.stp

probe module("ext3").function("*") { }
-probe module("ext3").function("*").return { }

- The first probe in ExampleÂ 3.2, âmoduleprobe.stpâ points to the entry of all functions for the ext3 module. The second probe points to the exits of all functions for that same module; the use of the .return suffix is similar to kernel.function(). Note that the probes in ExampleÂ 3.2, âmoduleprobe.stpâ do not contain statements in the probe handlers, and as such will not print any useful data (as in ExampleÂ 3.1, âwildcards.stpâ). -

- A system's kernel modules are typically located in /lib/modules/kernel_version, where kernel_version refers to the currently loaded kernel version. Modules use the file name extension .ko. -

Asynchronous Events

- Asynchronous events are not tied to a particular instruction or location in code. This family of probe points consists mainly of counters, timers, and similar constructs. -

- Examples of asynchronous events include: -

begin

- The startup of a SystemTap session; i.e. as soon as the SystemTap script is run. -

end

- The end of a SystemTap session. -

timer events

- An event that specifies a handler to be executed periodically. For example: -

ExampleÂ 3.3.Â timer-s.stp

probe timer.s(4)
-{
-  printf("hello world\n")
-}

- ExampleÂ 3.3, âtimer-s.stpâ is an example of a probe that prints hello world every 4 seconds. Note that you can also use the following timer events: -

- timer.ms(milliseconds) -
- timer.us(microseconds) -
- timer.ns(nanoseconds) -
- timer.hz(hertz) -
- timer.jiffies(jiffies) -

- When used in conjunction with other probes that collect information, timer events allows you to print out get periodic updates and see how that information changes over time. -

Important

- SystemTap supports the use of a large collection of probe events. For more information about supported events, refer to man stapprobes. The SEE ALSO section of man stapprobes also contains links to other man pages that discuss supported events for specific subsystems and components. -

3.2.2.Â Systemtap Handler/Body

- Consider the following sample script: -

ExampleÂ 3.4.Â helloworld.stp

probe begin
-{
-  printf ("hello world\n")
-  exit ()
-}

- In ExampleÂ 3.4, âhelloworld.stpâ, the event begin (i.e. the start of the session) triggers the handler enclosed in { }, which simply prints hello world followed by a new-line, then exits. -

Note

- SystemTap scripts continue to run until the exit() function executes. If the users wants to stop the execution of the script, it can interrupted manually with Ctrl+C. -

printf ( ) Statements

- The printf () statement is one of the simplest functions for printing data. printf () can also be used to display data using a wide variety of SystemTap functions in the following format: -

		printf ("format string\n", arguments)
-

- The format string specifies how arguments should be printed. The format string of ExampleÂ 3.4, âhelloworld.stpâ simply instructs SystemTap to print hello world, and contains no format specifiers. -

- You can use the format specifiers %s (for strings) and %d (for numbers) in format strings, depending on your list of arguments. Format strings can have multiple format specifiers, each matching a corresponding argument; multiple arguments are delimited by a comma (,). -

Note

- Semantically, the SystemTap printf function is very similar to its C language counterpart. The aforementioned syntax and format for SystemTap's printf function is identical to that of the C-style printf. -

- To illustrate this, consider the following probe example: -

ExampleÂ 3.5.Â variables-in-printf-statements.stp

probe syscall.open
-{
-  printf ("%s(%d) open\n", execname(), pid())
-}

- ExampleÂ 3.5, âvariables-in-printf-statements.stpâ instructs SystemTap to probe all entries to the system call open; for each event, it prints the current execname() (a string with the executable name) and pid() (the current process ID number), followed by the word open. A snippet of this probe's output would look like: -

vmware-guestd(2206) open
-hald(2360) open
-hald(2360) open
-hald(2360) open
-df(3433) open
-df(3433) open
-df(3433) open
-hald(2360) open

SystemTap Functions

- SystemTap supports a wide variety of functions that can be used as printf () arguments. ExampleÂ 3.5, âvariables-in-printf-statements.stpâ uses the SystemTap functions execname() (name of the process that called a kernel function/performed a system call) and pid() (current process ID). -

- The following is a list of commonly-used SystemTap functions: -

tid()

- The ID of the current thread. -

uid()

- The ID of the current user. -

cpu()

- The current CPU number. -

gettimeofday_s()

- The number of seconds since UNIX epoch (January 1, 1970). -

ctime()

- Convert number of seconds since UNIX epoch to date. -

pp()

- A string describing the probe point currently being handled. -

thread_indent()

- This particular function is quite useful, providing you with a way to better organize your print results. The function takes one argument, an indentation delta, which indicates how many spaces to add or remove from a thread's "indentation counter". It then returns a string with some generic trace data along with an appropriate number of indentation spaces. -

- The generic data included in the returned string includes a timestamp (number of microseconds since the first call to thread_indent() by the thread), a process name, and the thread ID. This allows you to identify what functions were called, who called them, and the duration of each function call. -

- If call entries and exits immediately precede each other, it is easy to match them. However, in most cases, after a first function call entry is made several other call entries and exits may be made before the first call exits. The indentation counter helps you match an entry with its corresponding exit by indenting the next function call if it is not the exit of the previous one. -

- Consider the following example on the use of thread_indent(): -

ExampleÂ 3.6.Â thread_indent.stp

probe kernel.function("*@net/socket.c").call
-{
-  printf ("%s -> %s\n", thread_indent(1), probefunc())
-}
-probe kernel.function("*@net/socket.c").return
-{
-  printf ("%s <- %s\n", thread_indent(-1), probefunc())
-}

- ExampleÂ 3.6, âthread_indent.stpâ prints out the thread_indent() and probe functions at each event in the following format: -

0 ftp(7223): -> sys_socketcall
-1159 ftp(7223):  -> sys_socket
-2173 ftp(7223):   -> __sock_create
-2286 ftp(7223):    -> sock_alloc_inode
-2737 ftp(7223):    <- sock_alloc_inode
-3349 ftp(7223):    -> sock_alloc
-3389 ftp(7223):    <- sock_alloc
-3417 ftp(7223):   <- __sock_create
-4117 ftp(7223):   -> sock_create
-4160 ftp(7223):   <- sock_create
-4301 ftp(7223):   -> sock_map_fd
-4644 ftp(7223):    -> sock_map_file
-4699 ftp(7223):    <- sock_map_file
-4715 ftp(7223):   <- sock_map_fd
-4732 ftp(7223):  <- sys_socket
-4775 ftp(7223): <- sys_socketcall

- This sample output contains the following information: -

- The time (in microseconds) since the initial thread_indent() call for the thread (included in the string from thread_indent()). -
- The process name (and its corresponding ID) that made the function call (included in the string from thread_indent()). -
- An arrow signifying whether the call was an entry (<-) or an exit (->); the indentations help you match specific function call entries with their corresponding exits. -
- The name of the function called by the process. -

name

- Identifies the name of a specific system call. This variable can only be used in probes that use the event syscall.system_call. -

target()

- Used in conjunction with stap script -x process ID or stap script -c command. If you want to specify a script to take an argument of a process ID or command, use target() as the variable in the script to refer to it. For example: -

ExampleÂ 3.7.Â targetexample.stp

probe syscall.* {
-  if (pid() == target())
-    printf("%s/n", name)
-}

- When ExampleÂ 3.7, âtargetexample.stpâ is run with the argument -x process ID, it watches all system calls (as specified by the event syscall.*) and prints out the name of all system calls made by the specified process. -

- This has the same effect as specifying if (pid() == process ID) each time you wish to target a specific process. However, using target() makes it easier for you to re-use the script, giving you the ability to simply pass a process ID as an argument each time you wish to run the script (e.g. stap targetexample.stp -x process ID). -

- For more information about supported SystemTap functions, refer to man stapfuncs. -

ChapterÂ 5.Â Useful SystemTap Scripts

- This chapter enumerates several SystemTap scripts you can use to monitor and investigate different subsystems. All of these scripts are available at /usr/share/systemtap/testsuite/systemtap.examples/ once you install the systemtap-testsuite RPM. -

5.1.Â Network

- The following sections showcase scripts that trace network-related functions and build a profile of network activity. -

5.1.1.Â Network Profiling

- This section describes how to profile network activity. nettop.stp provides a glimpse into how much network traffic each process is generating on a machine. -

nettop.stp

- -

#! /usr/bin/env stap
-
-global ifxmit, ifrecv
-global ifmerged
-
-probe netdev.transmit
-{
-  ifxmit[pid(), dev_name, execname(), uid()] <<< length
-}
-
-probe netdev.receive
-{
-  ifrecv[pid(), dev_name, execname(), uid()] <<< length
-}
-
-function print_activity()
-{
-  printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n",
-         "PID", "UID", "DEV", "XMIT_PK", "RECV_PK",
-         "XMIT_KB", "RECV_KB", "COMMAND")
-
-  foreach ([pid, dev, exec, uid] in ifrecv) {
-	  ifmerged[pid, dev, exec, uid] += @count(ifrecv[pid,dev,exec,uid]);
-  }
-  foreach ([pid, dev, exec, uid] in ifxmit) {
-	  ifmerged[pid, dev, exec, uid] += @count(ifxmit[pid,dev,exec,uid]);
-  }
-  foreach ([pid, dev, exec, uid] in ifmerged-) {
-    n_xmit = @count(ifxmit[pid, dev, exec, uid])
-    n_recv = @count(ifrecv[pid, dev, exec, uid])
-    printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n",
-           pid, uid, dev, n_xmit, n_recv,
-           n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0,
-           n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0,
-           exec)
-  }
-
-  print("\n")
-
-  delete ifxmit
-  delete ifrecv
-  delete ifmerged
-}
-
-probe timer.ms(5000), end, error
-{
-  print_activity()
-}
-

- Note that function print_activity() uses the following expressions: -

n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0
-n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0

- These expressions are if/else conditionals. The first statement is simply a more concise way of writing the following psuedo code: -

if n_recv != 0 then
-  @sum(ifrecv[pid, dev, exec, uid])/1024
-else
-  0

- nettop.stp tracks which processes are generating network traffic on the system, and provides the following information about each process: -

- PID â the ID of the listed process. -
- UID â user ID. A user ID of 0 refers to the root user. -
- DEV â which ethernet device the process used to send / receive data (e.g. eth0, eth1) -
- XMIT_PK â number of packets transmitted by the process -
- RECV_PK â number of packets received by the process -
- XMIT_KB â amount of data sent by the process, in kilobytes -
- RECV_KB â amount of data received by the service, in kilobytes -

- nettop.stp provides network profile sampling every 5 seconds. You can change this setting by editing probe timer.ms(5000) accordingly. ExampleÂ 5.1, ânettop.stp Sample Outputâ contains an excerpt of the output from nettop.stp over a 20-second period: -

ExampleÂ 5.1.Â nettop.stp Sample Output

[...]
-  PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
-    0     0 eth0          0       5       0       0 swapper
-11178     0 eth0          2       0       0       0 synergyc
-
-  PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
- 2886     4 eth0         79       0       5       0 cups-polld
-11362     0 eth0          0      61       0       5 firefox
-    0     0 eth0          3      32       0       3 swapper
- 2886     4 lo            4       4       0       0 cups-polld
-11178     0 eth0          3       0       0       0 synergyc
-
-  PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
-    0     0 eth0          0       6       0       0 swapper
- 2886     4 lo            2       2       0       0 cups-polld
-11178     0 eth0          3       0       0       0 synergyc
- 3611     0 eth0          0       1       0       0 Xorg
-
-  PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
-    0     0 eth0          3      42       0       2 swapper
-11178     0 eth0         43       1       3       0 synergyc
-11362     0 eth0          0       7       0       0 firefox
- 3897     0 eth0          0       1       0       0 multiload-apple
-[...]

5.1.2.Â Tracing Functions Called in Network Socket Code

- This section describes how to trace functions called from the kernel's net/socket.c file. This task helps you identify, in finer detail, how each process interacts with the network at the kernel level. -

socket-trace.stp

- -

#! /usr/bin/env stap
-
-probe kernel.function("*@net/socket.c").call {
-  printf ("%s -> %s\n", thread_indent(1), probefunc())
-}
-probe kernel.function("*@net/socket.c").return {
-  printf ("%s <- %s\n", thread_indent(-1), probefunc())
-}
-

- socket-trace.stp is identical to ExampleÂ 3.6, âthread_indent.stpâ, which was earlier used in SystemTap Functions to illustrate how thread_indent() works. -

ExampleÂ 5.2.Â socket-trace.stp Sample Output

[...]
-0 Xorg(3611): -> sock_poll
-3 Xorg(3611): <- sock_poll
-0 Xorg(3611): -> sock_poll
-3 Xorg(3611): <- sock_poll
-0 gnome-terminal(11106): -> sock_poll
-5 gnome-terminal(11106): <- sock_poll
-0 scim-bridge(3883): -> sock_poll
-3 scim-bridge(3883): <- sock_poll
-0 scim-bridge(3883): -> sys_socketcall
-4 scim-bridge(3883):  -> sys_recv
-8 scim-bridge(3883):   -> sys_recvfrom
-12 scim-bridge(3883):-> sock_from_file
-16 scim-bridge(3883):<- sock_from_file
-20 scim-bridge(3883):-> sock_recvmsg
-24 scim-bridge(3883):<- sock_recvmsg
-28 scim-bridge(3883):   <- sys_recvfrom
-31 scim-bridge(3883):  <- sys_recv
-35 scim-bridge(3883): <- sys_socketcall
-[...]

- ExampleÂ 5.2, âsocket-trace.stp Sample Outputâ contains a 3-second excerpt of the output for socket-trace.stp. For more information about the output of this script as provided by thread_indent(), refer to SystemTap Functions ExampleÂ 3.6, âthread_indent.stpâ. -

5.1.3.Â Monitoring Incoming TCP Connections

- This section illustrates how to monitor incoming TCP connections. This task is useful in identifying any unauthorized, suspicious, or otherwise unwanted network access requests in real time. -

tcp_connections.stp

- -

#! /usr/bin/env stap
-
-probe begin {
-  printf("%6s %16s %6s %6s %16s\n",
-         "UID", "CMD", "PID", "PORT", "IP_SOURCE")
-}
-
-probe kernel.function("tcp_accept").return?,
-      kernel.function("inet_csk_accept").return? {
-  sock = $return
-  if (sock != 0)
-    printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(),
-           inet_get_local_port(sock), inet_get_ip_source(sock))
-}
-

- While tcp_connections.stp is running, it will print out the following information about any incoming TCP connections accepted by the system in real time: -

- Current UID -
- CMD - the command accepting the connection -
- PID of the command -
- Port used by the connection -
- IP address from which the TCP connection originated -

ExampleÂ 5.3.Â tcp_connections.stp Sample Output

UID            CMD    PID   PORT        IP_SOURCE
-0             sshd   3165     22      10.64.0.227
-0             sshd   3165     22      10.64.0.227

5.1.4.Â Monitoring TCP Packets

- This section illustrates how to monitor TCP packets received by the system. This is useful in analyzing network traffic generated by applications running on the system. -

tcpdumplike.stp

- -

	#! /usr/bin/env stap
-
-// A TCP dump like example
-
-probe begin, timer.s(1) {
-  printf("-----------------------------------------------------------------\n")
-  printf("       Source IP         Dest IP  SPort  DPort  U  A  P  R  S  F \n")
-  printf("-----------------------------------------------------------------\n")
-}
-
-probe tcp.receive {
-  printf(" %15s %15s  %5d  %5d  %d  %d  %d  %d  %d  %d\n",
-         saddr, daddr, sport, dport, urg, ack, psh, rst, syn, fin)
-}
-

- While tcpdumplike.stp is running, it will print out the following information about any received TCP packets in real time: -

- Source and destination IP address (saddr, daddr, respectively) -
- Source and destination ports (sport, dport, respectively) -
- Packet flags -

- To determine the flags used by the packet, tcpdumplike.stp uses the following functions: -

- urg - urgent -
- ack - acknowledgement -
- psh - push -
- rst - reset -
- syn - synchronize -
- fin - finished -

- The aforementioned functions return 1 or 0 to specify whether the packet uses the corresponding flag. -

ExampleÂ 5.4.Â tcpdumplike.stp Sample Output

-----------------------------------------------------------------
-       Source IP         Dest IP  SPort  DPort  U  A  P  R  S  F
------------------------------------------------------------------
-  209.85.229.147       10.0.2.15     80  20373  0  1  1  0  0  0
-  92.122.126.240       10.0.2.15     80  53214  0  1  0  0  1  0
-  92.122.126.240       10.0.2.15     80  53214  0  1  0  0  0  0
-  209.85.229.118       10.0.2.15     80  63433  0  1  0  0  1  0
-  209.85.229.118       10.0.2.15     80  63433  0  1  0  0  0  0
-  209.85.229.147       10.0.2.15     80  21141  0  1  1  0  0  0
-  209.85.229.147       10.0.2.15     80  21141  0  1  1  0  0  0
-  209.85.229.147       10.0.2.15     80  21141  0  1  1  0  0  0
-  209.85.229.147       10.0.2.15     80  21141  0  1  1  0  0  0
-  209.85.229.147       10.0.2.15     80  21141  0  1  1  0  0  0
-  209.85.229.118       10.0.2.15     80  63433  0  1  1  0  0  0
-[...]

5.1.5.Â Monitoring Network Packets Drops in Kernel

- - The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb"), which easily tracks where packets are discarded. dropwatch.stp uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes which locations discard packets every five-second interval. -

dropwatch.stp

- -

#! /usr/bin/env stap
-
-############################################################
-# Dropwatch.stp
-# Author: Neil Horman <nhorman@redhat.com>
-# An example script to mimic the behavior of the dropwatch utility
-# http://fedorahosted.org/dropwatch
-############################################################
-
-# Array to hold the list of drop points we find
-global locations
-
-# Note when we turn the monitor on and off
-probe begin { printf("Monitoring for dropped packets\n") }
-probe end { printf("Stopping dropped packet monitor\n") }
-
-# increment a drop counter for every location we drop at
-probe kernel.trace("kfree_skb") { locations[$location] <<< 1 }
-
-# Every 5 seconds report our drop locations
-probe timer.sec(5)
-{
-  printf("\n")
-  foreach (l in locations-) {
-    printf("%d packets dropped at %s\n",
-           @count(locations[l]), symname(l))
-  }
-  delete locations
-}
-

- The kernel.trace("kfree_skb") traces which places in the kernel drop network packets. The kernel.trace("kfree_skb") has two arguments: a pointer to the buffer being freed ($skb) and the location in kernel code the buffer is being freed ($location). The dropwatch.stp script provides the function containing $location where possible. The information to map $location back to the function is not in the instrumentation by default. On Systemtap 1.4 the --all-modules option will include the required mapping information and the following command can be used to run the script: -

stap --all-modules dropwatch.stp

- On older versions of Systemtap you can use the following command to emulate the --all-modules option: -

stap -dkernel \
-`cat /proc/modules | awk 'BEGIN { ORS = " " } {print "-d"$1}'` \
-dropwatch.stp

- Running the dropwatch.stp script 15 seconds would result in output similar in ExampleÂ 5.5, âdropwatch.stp Sample Outputâ. The output lists the number of misses for each tracepoint location with either the function name or the address. -

ExampleÂ 5.5.Â dropwatch.stp Sample Output

Monitoring for dropped packets
-
-1762 packets dropped at unix_stream_recvmsg
-4 packets dropped at tun_do_read
-2 packets dropped at nf_hook_slow
-
-467 packets dropped at unix_stream_recvmsg
-20 packets dropped at nf_hook_slow
-6 packets dropped at tun_do_read
-
-446 packets dropped at unix_stream_recvmsg
-4 packets dropped at tun_do_read
-4 packets dropped at nf_hook_slow
-Stopping dropped packet monitor
-

- When the script is being compiled on one machine and run on another the --all-modules and /proc/modules directory are not available; the symname function will just print out the raw address. To make the raw address of packet drops more meaningful, refer to the /boot/System.map-`uname -r` file. This file lists the starting addresses for each function, allowing you to map the addresses in the output of ExampleÂ 5.5, âdropwatch.stp Sample Outputâ to a specific function name. Given the following snippet of the /boot/System.map-`uname -r` file, the address 0xffffffff8149a8ed maps to the function unix_stream_recvmsg: -

[...]
-ffffffff8149a420 t unix_dgram_poll
-ffffffff8149a5e0 t unix_stream_recvmsg
-ffffffff8149ad00 t unix_find_other
-[...]

ChapterÂ 2.Â Using SystemTap

2.1. Installation and Setup

2.1.1. Installing SystemTap
2.1.2. Installing Required Kernel Information RPMs
2.1.3. Initial Testing

2.2. Generating Instrumentation for Other Computers

2.3. Running SystemTap Scripts

2.3.1. SystemTap Flight Recorder Mode

- This chapter instructs users how to install SystemTap, and provides an introduction on how to run SystemTap scripts. -

2.1.Â Installation and Setup

- To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of -devel, -debuginfo and -debuginfo-common packages for your kernel. If your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel kernel, you will need to install the -devel and -debuginfo packages for each of those kernel versions. -

- These procedures will be discussed in detail in the following sections. -

Important

- Many users confuse -debuginfo with -debug. Remember that the deployment of SystemTap requires the installation of the -debuginfo package of the kernel, not the -debug version of the kernel. -

2.1.1.Â Installing SystemTap

- To deploy Systemtap, you will need to to install the following RPMs: -

- systemtap -
- systemtap-runtime -

- Assuming that yum is installed in the system, these two rpms can be installed with yum install systemtap systemtap-runtime. Note that before you can use SystemTap, you will still need to install the required kernel information RPMs. -

2.1.2.Â Installing Required Kernel Information RPMs

- SystemTap needs information about the kernel in order to place instrumentation in it (i.e. probe it). This information also allows SystemTap to generate the code for the instrumentation. This information is contained in the matching -devel, -debuginfo, and -debuginfo-common packages for your kernel. The necessary -devel and -debuginfo packages for the ordinary "vanilla" kernel are as follows: -

- kernel-debuginfo -
- kernel-debuginfo-common -
- kernel-devel -

- Likewise, the necessary packages for the PAE kernel would be kernel-PAE-debuginfo, kernel-PAE-debuginfo-common, and kernel-PAE-devel. -

- To determine what kernel your system is currently using, use: -

uname -r

- For example, if you wish to use SystemTap on kernel version 2.6.18-53.el5 on an i686 machine, then you would need to download and install the following RPMs: -

- kernel-debuginfo-2.6.18-53.1.13.el5.i686.rpm -
- kernel-debuginfo-common-2.6.18-53.1.13.el5.i686.rpm -
- kernel-devel-2.6.18-53.1.13.el5.i686.rpm -

Important

- The version, variant, and architecture of the -devel, -debuginfo and -debuginfo-common packages must match the kernel you wish to probe with SystemTap exactly. -

- The easiest way to install the required kernel information packages is through yum install and debuginfo-install commands. debuginfo-install is included with later versions of the yum-utils package (for example, version 1.1.10), and also requires an appropriate yum repository from which to download and install -debuginfo/-debuginfo-common packages. You can install the required -devel, -debuginfo, and -debuginfo-common packages for your kernel. -

- With the appropriate software repositories are enabled, install the corresponding packages for a specific kernel using yum with the following commands: -

- yum install kernelname-devel-version -
- debuginfo-install kernelname-version -

- Replace kernelname with the appropriate kernel variant name (for example, kernel-PAE), and version with the target kernel's version. For example, to install the required kernel information packages for the kernel-PAE-2.6.18-53.1.13.el5 kernel, run: -

- yum install kernel-PAE-devel-2.6.18-53.1.13.el5 -
- debuginfo-install kernel-PAE-2.6.18-53.1.13.el5 -

- If you do not have yum and yum-utils installed (and you are unable to install them), you will have to manually download and install the required kernel information packages. To generate the URL from which to download the required packages, use the following script: -

fedoradebugurl.sh

- -

#! /bin/bash
-echo -n "Enter nvr of kernel-debuginfo (e.g. 2.6.25-14.fc9.x86_64) " ; \
-read NVR; \
-BASE=`uname -m` ; \
-NVR=`echo $NVR | sed s/.$BASE//` ; \
-VERSION=`echo $NVR | awk -F- '{print $1}'` ; \
-RELEASE=`echo $NVR | awk -F- '{print $2}'` ; \
-echo "http://kojipkgs.fedoraproject.org/\
-packages/kernel/$VERSION/$RELEASE/$BASE/"
-

- Once you have manually downloaded the required packages to the machine, install the RPMs by running rpm --force -ivh package_names. -

2.1.3.Â Initial Testing

- If you are currently using the kernel you wish to probe with SystemTap, you can immediately test whether the deployment was successful. If not, you will need to reboot and load the appropriate kernel. -

- To start the test, run the command stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'. This command simply instructs SystemTap to print read performed then exit properly once a virtual file system read is detected. If the SystemTap deployment was successful, you should get output similar to the following: -

Pass 1: parsed user script and 45 library script(s) in 340usr/0sys/358real ms.
-Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 290usr/260sys/568real ms.
-Pass 3: translated to C into "/tmp/stapiArgLX/stap_e5886fa50499994e6a87aacdc43cd392_399.c" in 490usr/430sys/938real ms.
-Pass 4: compiled C into "stap_e5886fa50499994e6a87aacdc43cd392_399.ko" in 3310usr/430sys/3714real ms.
-Pass 5: starting run.
-read performed
-Pass 5: run completed in 10usr/40sys/73real ms.

- The last three lines of the output (i.e. beginning with Pass 5) indicate that SystemTap was able to successfully create the instrumentation to probe the kernel, run the instrumentation, detect the event being probed (in this case, a virtual file system read), and execute a valid handler (print text then close it with no errors). -

2.3.Â Running SystemTap Scripts

- SystemTap scripts are run through the command stap. stap can run SystemTap scripts from standard input or from file. -

- Running stap and staprun requires elevated privileges to the system. However, not all users can be granted root access just to run SystemTap. In some cases, for instance, you may want to allow a non-privileged user to run SystemTap instrumentation on his machine. -

- To allow ordinary users to run SystemTap without root access, add them to one of these user groups: -

stapdev: - Members of this group can use stap to run SystemTap scripts, or staprun to run SystemTap instrumentation modules. -
- Running stap involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This requires elevated privileges to the system, which are granted to stapdev members. Unfortunately, such privileges also grant effective root access to stapdev members. As such, you should only grant stapdev group membership to users whom you can trust root access. -
stapusr: - Members of this group can only run staprun to run SystemTap instrumentation modules. In addition, they can only run those modules from /lib/modules/kernel_version/systemtap/. Note that this directory must be owned only by the root user, and must only be writable by the root user. -

- Below is a list of commonly used stap options: -

-v: - Makes the output of the SystemTap session more verbose. You can repeat this option (for example, stap -vvv script.stp) to provide more details on the script's execution. This option is particularly useful if you encounter any errors in running the script. -
- For more information about common SystemTap script errors, refer to ChapterÂ 6, Understanding SystemTap Errors. -
-o filename: - Sends the standard output to file (filename). -
-S size,count: - Limit files to size megabytes and limit the the number of files kept around to count. The file names will have a sequence number suffix. This option implements logrotate operations for SystemTap. -
-x process ID: - Sets the SystemTap handler function target() to the specified process ID. For more information about target(), refer to SystemTap Functions. -
-c 'command': - Sets the SystemTap handler function target() to the specified command and runs the SystemTap instrumentation for the duration of the specified command. For more information about target(), refer to SystemTap Functions. -
-e 'script': - Use script string rather than a file as input for systemtap translator. -
-F: - Use SystemTap's Flight recorder mode and make the script a background process. For more information about flight recorder mode, refer to SectionÂ 2.3.1, âSystemTap Flight Recorder Modeâ. -

- You can also instruct stap to run scripts from standard input using the switch -. To illustrate: -

ExampleÂ 2.1.Â Running Scripts From Standard Input

echo "probe timer.s(1) {exit()}" | stap -

- ExampleÂ 2.1, âRunning Scripts From Standard Inputâ instructs stap to run the script passed by echo to standard input. Any stap options you wish to use should be inserted before the - switch; for instance, to make the example in ExampleÂ 2.1, âRunning Scripts From Standard Inputâ more verbose, the command would be: -

- echo "probe timer.s(1) {exit()}" | stap -v - -

- For more information about stap, refer to man stap. -

- To run SystemTap instrumentation (i.e. the kernel module built from SystemTap scripts during a cross-instrumentation), use staprun instead. For more information about staprun and cross-instrumentation, refer to SectionÂ 2.2, âGenerating Instrumentation for Other Computersâ. -

Note

- The stap options -v and -o also work for staprun. For more information about staprun, refer to man staprun. -

2.3.1.Â SystemTap Flight Recorder Mode

- SystemTap's flight recorder mode allows you to run a SystemTap script run for long periods and just focus on recent output. The flight recorder mode (the -F option) limits the amount of output generated. There are two variations of the flight recorder mode: in-memory and file mode. In both cases the SystemTap script runs as a background process. -

2.3.1.1.Â In-memory Flight Recorder

- When flight recorder mode (the -F option) is used without a file name SystemTap uses a buffer in kernel memory to store the output of the script. The SystemTap instrumentation module will load and the probes start running, the instrumentation will then detach and be put in the background. When the interesting event occurs, you can reattach to the instrumentation and see the recent output in the memory buffer and any continuing output. The following command starts a script using the flight recorder in-memory mode: -

stap -F iotime.stp

- Once the script starts, you will see a message like the following that provides the command to reconnect to the running script: -

Disconnecting from systemtap module.
-To reconnect, type "staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556"

- When the interesting event occurs, you reattach to the currently running script and output the recent data in the memory buffer and get continuing output with the following command: -

staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556

- By default the kernel buffer is 1MB in size and it can be increased with the -s option specifying the size in megabytes (rounded up to the next power over 2) for the buffer. For example -s2 on the SystemTap command line would specify 2MB for the buffer. -

2.3.1.2.Â File Flight Recorder

- The flight recorder mode can also store data to files. The number and size of the files kept is controlled by the -S option followed by two numerical arguments separated by a comma. The first argument is the maximum size in megabytes for the each output file. The second argument is the number of recent files to keep. The file name is specified by the -o option followed by the name. SystemTap will add a number suffix to the file name to indicate the order of the files. The following will start SystemTap in file flight recorder mode with the output going to files named /tmp/iotime.log.[0-9]+ and each file 1MB or smaller and keeping latest two files: -

stap -F -o /tmp/pfaults.log -S 1,2  pfaults.stp

- The number printed by the command is the process ID. Sending a SIGTERM to the process will shutdown the SystemTap script and stop the data collection. For example if the previous command listed the 7590 as the process ID, the following command whould shutdown the systemtap script: -

kill -s SIGTERM 7590

- Only the most recent two file generated by the script are kept and the older files are been removed. Thus, ls -sh /tmp/pfaults.log.* shows the only two files: -

1020K /tmp/pfaults.log.5    44K /tmp/pfaults.log.6

- One can look at the highest number file for the latest data, in this case /tmp/pfaults.log.6. -

Preface

1.Â Document Conventions

1.1.Â Typographic Conventions

1.2.Â Pull-quote Conventions

1.3.Â Notes and Warnings

Note

Important

Warning

Revision History

3.5.Â Array Operations in SystemTap

3.5.1.Â Assigning an Associated Value

ExampleÂ 3.14.Â Associating Timestamps to Process Names

3.5.7.Â Computing for Statistical Aggregates

ExampleÂ 3.21.Â stat-aggregates.stp

Note

ExampleÂ 3.22.Â Multiple Array Indexes

3.5.6.Â Using Arrays in Conditional Statements

ExampleÂ 3.19.Â vfsreads-print-if-1kb.stp

Testing for Membership

ExampleÂ 3.20.Â vfsreads-stop-on-stapio2.stp

3.5.5.Â Clearing/Deleting Arrays and Array Elements

ExampleÂ 3.18.Â noncumulative-vfsreads.stp

Note

3.5.4.Â Processing Multiple Elements in an Array

ExampleÂ 3.17.Â cumulative-vfsreads.stp

3.5.3.Â Incrementing Associated Values

ExampleÂ 3.16.Â vfsreads.stp

3.5.2.Â Reading Values From Arrays

ExampleÂ 3.15.Â Using Array Values in Simple Computations

Note

3.4.Â Associative Arrays

ExampleÂ 3.13.Â Basic Array Statements

Important

3.3.4.Â Command-Line Arguments

ExampleÂ 3.12.Â commandlineargs.stp

2.2.Â Generating Instrumentation for Other Computers

Note

ProcedureÂ 2.1.Â Configuring a Host System and Target Systems

Note

Important

ChapterÂ 6.Â Understanding SystemTap Errors

6.1.Â Parse and Semantic Errors

parse error: expected foo, saw bar

parse error: embedded code in unprivileged script

semantic error: type mismatch for identifier 'foo' ... string vs. long

ExampleÂ 6.1.Â error-variable.stp

semantic error: unresolved type for identifier 'foo'

semantic error: Expecting symbol or array index expression

while searching for arity N function, semantic error: unresolved function call

semantic error: array locals not supported, missing global declaration?

semantic error: variable âfooâ modiï¬ed during âforeachâ iteration

semantic error: probe point mismatch at position N, while resolving probe point foo

semantic error: no match for probe point, while resolving probe point foo

semantic error: unresolved target-symbol expression

semantic error: libdwï¬ failure

semantic error: cannot find foo debuginfo

5.4.Â Identifying Contended User-Space Locks

futexes.stp

ExampleÂ 5.19.Â futexes.stp Sample Output

3.3.3.Â Conditional Statements

ExampleÂ 3.11.Â ifelse.stp

Conditional Operators

SystemTap Beginners Guide

Introduction to SystemTap (for Fedora)

- Logo - -

- Red Hat, Inc. -

Don Domingo

William Cohen

Legal Notice

Abstract

5.2.6.Â Monitoring Changes to File Attributes

inodewatch2.stp

ExampleÂ 5.11.Â inodewatch2.stp Sample Output

5.2.5.Â Monitoring Reads and Writes to a File

inodewatch.stp

ExampleÂ 5.10.Â inodewatch.stp Sample Output

1.2.Â SystemTap Capabilities

Limitations

ChapterÂ 1.Â Introduction

1.1.Â Documentation Goals

parse error: expected `foo`, saw `bar`

semantic error: type mismatch for identifier '`foo`' ... string vs. long

semantic error: unresolved type for identifier '`foo`'

while searching for arity `N` function, semantic error: unresolved function call

semantic error: variable â`foo`â modiï¬ed during âforeachâ iteration

semantic error: probe point mismatch at position `N`, while resolving probe point `foo`

semantic error: no match for probe point, while resolving probe point `foo`

semantic error: libdwï¬ failure

semantic error: cannot find `foo` debuginfo

- - -

WARNING: Number of errors: `N`, skipped probes: `M`

kernel/user string copy fault at `ADDR`