Preface

Abstract

+ This guide provides basic instructions on how to use SystemTap to monitor different subsystems of Fedora in finer detail. The SystemTap Beginners Guide is recommended for users who have taken RHCT or have a similar level of expertise in Fedora. +

Index

Symbols

$count

sample usage

local variables, Tracking I/O Time For Each File Read or Write

$return

sample usage

local variables, Summarizing Disk Read/Write Traffic, Track Cumulative IO

@avg (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@count (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@max (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@min (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@sum (integer extractor)

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

A

adding values to statistical aggregates

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

advantages of cross-instrumentation, Generating Instrumentation for Other Computers

aggregate element not found

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

aggregates (statistical)

array operations, Computing for Statistical Aggregates

aggregation overflow

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

algebraic formulas using arrays

reading values from arrays

array operations, Reading Values From Arrays

architecture notation, determining, Generating Instrumentation for Other Computers

architecture of SystemTap, Architecture

array locals not supported

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

array operations

assigning associated values, Assigning an Associated Value

associating timestamps to process names, Assigning an Associated Value

associative arrays, Array Operations in SystemTap

clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements

delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements

computing for statistical aggregates, Computing for Statistical Aggregates

@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates

conditional statements, using arrays in, Using Arrays in Conditional Statements

testing for array membership, Using Arrays in Conditional Statements

deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements

incrementing associated values, Incrementing Associated Values

tallying virtual file system reads (VFS reads), Incrementing Associated Values

multiple elements in an array, Processing Multiple Elements in an Array

processing multiple elements in an array, Processing Multiple Elements in an Array

cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array

reading values from arrays, Reading Values From Arrays

computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays

arrays, Associative Arrays

(see also associative arrays)

assigning associated values

array operations, Assigning an Associated Value

associating timestamps to process names, Assigning an Associated Value

associating timestamps to process names

array operations, Assigning an Associated Value

associated values

introduction

arrays, Associative Arrays

associating timestamps to process names

assigning associated values

array operations, Assigning an Associated Value

associative arrays

introduction, Associative Arrays

associated values, Associative Arrays
example, Associative Arrays
index expression, Associative Arrays
key pairs, Associative Arrays
syntax, Associative Arrays
unique keys, Associative Arrays

asynchronous events

Events, Event

B

begin

Events, Event

building instrumentation modules from SystemTap scripts, Generating Instrumentation for Other Computers

building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers

C

call graph tracing

examples of SystemTap scripts, Call Graph Tracing

capabilities of SystemTap

Introduction, SystemTap Capabilities

changes to file attributes, monitoring

examples of SystemTap scripts, Monitoring Changes to File Attributes

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements

command-line arguments

SystemTap handler constructs

handlers, Command-Line Arguments

compiling instrumentation/kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers

components

SystemTap scripts

introduction, SystemTap Scripts

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates

computing for timestamp deltas

reading values from arrays

array operations, Reading Values From Arrays

conditional operators

conditional statements

handlers, Conditional Statements

conditional statements, using arrays in

array operations, Using Arrays in Conditional Statements

testing for array membership, Using Arrays in Conditional Statements

CONFIG_HZ, computing for, Variables

contended user-space locks (futex contentions), identifying

examples of SystemTap scripts, Identifying Contended User-Space Locks

copy fault

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

count operator

computing for statistical aggregates

array (operator), Computing for Statistical Aggregates

counting function calls

examples of SystemTap scripts, Counting Function Calls Made

CPU ticks

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

cpu()

functions, Systemtap Handler/Body

cross-compiling, Generating Instrumentation for Other Computers

cross-instrumentation

advantages of, Generating Instrumentation for Other Computers

building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers

configuration

host system and target system, Generating Instrumentation for Other Computers

generating instrumentation from SystemTap scripts, Generating Instrumentation for Other Computers

host system, Generating Instrumentation for Other Computers

instrumentation module, Generating Instrumentation for Other Computers

target kernel, Generating Instrumentation for Other Computers

target system, Generating Instrumentation for Other Computers

ctime()

functions, Systemtap Handler/Body

ctime(), example of usage

script examples, Summarizing Disk Read/Write Traffic

cumulative I/O, tracking

examples of SystemTap scripts, Track Cumulative IO

cumulative virtual file system reads, tallying

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

D

delete operator

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

determining architecture notation, Generating Instrumentation for Other Computers

determining the kernel version, Installing Required Kernel Information RPMs

determining time spent in kernel and user space

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

device I/O, monitoring

examples of SystemTap scripts, I/O Monitoring (By Device)

device number of a file (integer format)

examples of SystemTap scripts, Monitoring Reads and Writes to a File

disk I/O traffic, summarizing

script examples, Summarizing Disk Read/Write Traffic

division by 0

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

documentation goals

Introduction, Documentation Goals

E

embedded code in unprivileged script

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

empty unique keys

reading values from arrays

array operations, Reading Values From Arrays

end

Events, Event

errors

parse/semantics error, Parse and Semantic Errors

embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors

runtime errors/warnings, Run Time Errors and Warnings

aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings

event types

Understanding How SystemTap Works, Understanding How SystemTap Works

Events

asynchronous events, Event
begin, Event
end, Event
examples of synchronous and asynchronous events, Event
introduction, Event
kernel.function("function"), Event
kernel.trace("tracepoint"), Event
module("module"), Event
synchronous events, Event
syscall.system_call, Event
timer events, Event
user-space, User-Space Events
vfs.file_operation, Event
wildcards, Event

events and handlers, Understanding How SystemTap Works

events wildcards, Event

example

introduction

arrays, Associative Arrays

example of multiple command-line arguments

examples of SystemTap scripts, Call Graph Tracing

examples of synchronous and asynchronous events

Events, Event

examples of SystemTap scripts, Useful SystemTap Scripts

call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex (lock) contentions, Identifying Contended User-Space Locks
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)

exceeded MAXACTION

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

exceeded MAXNESTING

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

exit()

functions, Systemtap Handler/Body

expected symbol/array index expression

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

extracting data collected by statistical aggregates

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

F

fedoradebugurl.sh, Installing Required Kernel Information RPMs

feedback

contact information for this manual, We Need Feedback!

file attributes, monitoring changes to

examples of SystemTap scripts, Monitoring Changes to File Attributes

file device number (integer format)

examples of SystemTap scripts, Monitoring Reads and Writes to a File

file reads/writes, monitoring

examples of SystemTap scripts, Monitoring Reads and Writes to a File

flight recorder mode, SystemTap Flight Recorder Mode

file mode, File Flight Recorder
in-memory mode, In-memory Flight Recorder

for loops

conditional statements

handlers, Conditional Statements

foreach

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

format

introduction

arrays, Associative Arrays

format and syntax

printf(), Systemtap Handler/Body

SystemTap handler constructs

handlers, Variables

SystemTap scripts

introduction, SystemTap Scripts

format specifiers

printf(), Systemtap Handler/Body

format strings

printf(), Systemtap Handler/Body

function call (unresolved)

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

function calls (incoming/outgoing), tracing

examples of SystemTap scripts, Call Graph Tracing

function calls, tallying

examples of SystemTap scripts, Counting Function Calls Made

functions, Systemtap Handler/Body

cpu(), Systemtap Handler/Body

ctime(), Systemtap Handler/Body

gettimeofday_s(), Systemtap Handler/Body

pp(), Systemtap Handler/Body

SystemTap scripts

introduction, SystemTap Scripts

target(), Systemtap Handler/Body

thread_indent(), Systemtap Handler/Body

tid(), Systemtap Handler/Body

uid(), Systemtap Handler/Body

functions (used in handlers)

exit(), Systemtap Handler/Body

functions called in network socket code, tracing

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

futex (lock) contentions

examples of SystemTap scripts, Identifying Contended User-Space Locks

futex contention, definition

examples of SystemTap scripts, Identifying Contended User-Space Locks

futex contentions, identifying

examples of SystemTap scripts, Identifying Contended User-Space Locks

futex system call

examples of SystemTap scripts, Identifying Contended User-Space Locks

G

gettimeofday_s()

functions, Systemtap Handler/Body

global

SystemTap handler constructs

handlers, Variables

goals, documentation

Introduction, Documentation Goals

grammatical/typographical script error

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

guru mode

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

H

handler functions, Systemtap Handler/Body

handlers

conditional statements, Conditional Statements

conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements

introduction, Systemtap Handler/Body

SystemTap handler constructs, Basic SystemTap Handler Constructs

command-line arguments, Command-Line Arguments
global, Variables
syntax and format, Basic SystemTap Handler Constructs
variable notations, Command-Line Arguments
variables, Variables

target variables, Target Variables

handlers and events, Understanding How SystemTap Works

SystemTap scripts

introduction, SystemTap Scripts

heaviest disk reads/writes, identifying

script examples, Summarizing Disk Read/Write Traffic

host system

cross-instrumentation, Generating Instrumentation for Other Computers

host system and target system

cross-instrumentation

configuration, Generating Instrumentation for Other Computers

I

I/O block time, monitoring

examples of SystemTap scripts, Periodically Print I/O Block Time

I/O monitoring (by device)

examples of SystemTap scripts, I/O Monitoring (By Device)

I/O time, monitoring

examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write

I/O traffic, summarizing

script examples, Summarizing Disk Read/Write Traffic

identifier type mismatch

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

identifying contended user-space locks (i.e. futex contentions)

examples of SystemTap scripts, Identifying Contended User-Space Locks

identifying heaviest disk reads/writes

script examples, Summarizing Disk Read/Write Traffic

if/else

conditional statements

handlers, Conditional Statements

if/else conditionals, alternative syntax

examples of SystemTap scripts, Network Profiling

if/else statements, using arrays in

array operations, Using Arrays in Conditional Statements

incoming TCP connections, monitoring

examples of SystemTap scripts, Monitoring Incoming TCP Connections

incoming/outgoing function calls, tracing

examples of SystemTap scripts, Call Graph Tracing

incrementing associated values

array operations, Incrementing Associated Values

tallying virtual file system reads (VFS reads), Incrementing Associated Values

index expression

introduction

arrays, Associative Arrays

initial testing, Initial Testing

inode number

examples of SystemTap scripts, Monitoring Reads and Writes to a File

Installation

fedoradebugurl.sh, Installing Required Kernel Information RPMs
initial testing, Initial Testing
kernel information packages, Installing Required Kernel Information RPMs
kernel version, determining the, Installing Required Kernel Information RPMs
required packages, Installing Required Kernel Information RPMs
Setup and Installation, Installation and Setup
systemtap package, Installing SystemTap
systemtap-runtime package, Installing SystemTap

instrumentation module

cross-instrumentation, Generating Instrumentation for Other Computers

instrumentation modules from SystemTap scripts, building, Generating Instrumentation for Other Computers

integer extractors

computing for statistical aggregates

array operations, Computing for Statistical Aggregates

Introduction

capabilities of SystemTap, SystemTap Capabilities
documentation goals, Documentation Goals
goals, documentation, Documentation Goals
limitations of SystemTap, SystemTap Capabilities
performance monitoring, Introduction

invalid division

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

invalid values to variables/arrays

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

iterations, processing elements in an array as

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

K

kernel and user space, determining time spent in

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

kernel information packages, Installing Required Kernel Information RPMs

kernel modules from SystemTap scripts, building, Generating Instrumentation for Other Computers

kernel version, determining the, Installing Required Kernel Information RPMs

kernel.function("function")

Events, Event

kernel.trace("tracepoint")

Events, Event

key pairs

introduction

arrays, Associative Arrays

L

libdwfl failure

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

limitations of SystemTap

Introduction, SystemTap Capabilities

limiting the output of foreach

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

local variables

name, Systemtap Handler/Body

sample usage

$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO

M

MAXACTION exceeded

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

MAXNESTING exceeded

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

membership (in array), testing for

conditional statements, using arrays in

array operations, Using Arrays in Conditional Statements

module("module")

Events, Event

monitoring changes to file attributes

examples of SystemTap scripts, Monitoring Changes to File Attributes

monitoring cumulative I/O

examples of SystemTap scripts, Track Cumulative IO

monitoring device I/O

examples of SystemTap scripts, I/O Monitoring (By Device)

monitoring I/O block time

examples of SystemTap scripts, Periodically Print I/O Block Time

monitoring I/O time

examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write

monitoring incoming TCP connections

examples of SystemTap scripts, Monitoring Incoming TCP Connections

monitoring polling applications

examples of SystemTap scripts, Monitoring Polling Applications

monitoring reads and writes to a file

examples of SystemTap scripts, Monitoring Reads and Writes to a File

monitoring system calls

examples of SystemTap scripts, Tracking Most Frequently Used System Calls

monitoring system calls (volume per process)

examples of SystemTap scripts, Tracking System Call Volume Per Process

monitoring TCP packets

examples of SystemTap scripts, Monitoring TCP Packets

multiple array operations within the same probe

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

multiple command-line arguments, example of

examples of SystemTap scripts, Call Graph Tracing

multiple elements in an array

array operations, Processing Multiple Elements in an Array

N

name

local variables, Systemtap Handler/Body

net/socket.c, tracing functions from

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

network profiling

examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel

network socket code, tracing functions called in

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

network traffic, monitoring

examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel

no match for probe point

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

non-global arrays

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

number of errors: N, skipped probes: M

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

O

operations

assigning associated values

associating timestamps to process names, Assigning an Associated Value

associative arrays, Array Operations in SystemTap

clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements

delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements

computing for statistical aggregates, Computing for Statistical Aggregates

@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates

conditional statements, using arrays in, Using Arrays in Conditional Statements

testing for array membership, Using Arrays in Conditional Statements

deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements

incrementing associated values, Incrementing Associated Values

tallying virtual file system reads (VFS reads), Incrementing Associated Values

multiple elements in an array, Processing Multiple Elements in an Array

processing multiple elements in an array, Processing Multiple Elements in an Array

cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array

reading values from arrays, Reading Values From Arrays

computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays

options, stap

Usage, Running SystemTap Scripts

ordering the output of foreach

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

overflow of aggregation

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

P

packages required to run SystemTap, Installing Required Kernel Information RPMs

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors

performance monitoring

Introduction, Introduction

pointer dereference fault

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

polling applications, monitoring

examples of SystemTap scripts, Monitoring Polling Applications

pp()

functions, Systemtap Handler/Body

printf()

format specifiers, Systemtap Handler/Body
format strings, Systemtap Handler/Body
syntax and format, Systemtap Handler/Body

printing I/O activity (cumulative)

examples of SystemTap scripts, Track Cumulative IO

printing I/O block time (periodically)

examples of SystemTap scripts, Periodically Print I/O Block Time

probe mismatch

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

probe point (no match for)

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

probes

SystemTap scripts

introduction, SystemTap Scripts

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

cumulative virtual file system reads, tallying

array operations, Processing Multiple Elements in an Array

foreach

array operations, Processing Multiple Elements in an Array

limiting the output of foreach

array operations, Processing Multiple Elements in an Array

ordering the output of foreach

array operations, Processing Multiple Elements in an Array

profiling the network

examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel

R

reading values from arrays

array operations, Reading Values From Arrays

empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays

computing for timestamp deltas

array operations, Reading Values From Arrays

reads/writes to a file, monitoring

examples of SystemTap scripts, Monitoring Reads and Writes to a File

required packages, Installing Required Kernel Information RPMs

RPMs required to run SystemTap, Installing Required Kernel Information RPMs

running scripts from standard input, Running SystemTap Scripts

running SystemTap scripts

Usage, Running SystemTap Scripts

runtime errors/warnings

understainding SystemTap errors, Run Time Errors and Warnings

aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings

S

script examples

call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex (lock) contentions, Identifying Contended User-Space Locks
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)

scripts

introduction, SystemTap Scripts

components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts

sessions, SystemTap, Architecture

Setup and Installation, Installation and Setup

Stack backtrace

user-space, User-Space Stack Backtraces

standard input, running scripts from

Usage, Running SystemTap Scripts

stap

Usage, Running SystemTap Scripts

stap options, Running SystemTap Scripts

stapdev

Usage, Running SystemTap Scripts

staprun

Usage, Running SystemTap Scripts

stapusr

Usage, Running SystemTap Scripts

stat -c, determining file device number (integer format)

examples of SystemTap scripts, Monitoring Reads and Writes to a File

stat -c, determining whole device number

examples of SystemTap scripts, I/O Monitoring (By Device)

statement blocks

SystemTap scripts

introduction, SystemTap Scripts

statistical aggregates

array operations, Computing for Statistical Aggregates

summarizing disk I/O traffic

script examples, Summarizing Disk Read/Write Traffic

synchronous events

Events, Event

syntax

introduction

arrays, Associative Arrays

syntax and format

printf(), Systemtap Handler/Body

SystemTap handler constructs

handlers, Basic SystemTap Handler Constructs

SystemTap scripts

introduction, SystemTap Scripts

syscall.system_call

Events, Event

system calls volume (per process), monitoring

examples of SystemTap scripts, Tracking System Call Volume Per Process

system calls, monitoring

examples of SystemTap scripts, Tracking Most Frequently Used System Calls

SystemTap architecture, Architecture

SystemTap handlers

SystemTap handler constructs, Basic SystemTap Handler Constructs

syntax and format, Basic SystemTap Handler Constructs

systemtap package, Installing SystemTap

SystemTap script functions, Systemtap Handler/Body

SystemTap scripts

introduction, SystemTap Scripts

components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts

useful examples, Useful SystemTap Scripts

SystemTap scripts, how to run, Running SystemTap Scripts

SystemTap sessions, Architecture

SystemTap statements

conditional statements, Conditional Statements

conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements

SystemTap handler constructs

command-line arguments, Command-Line Arguments
global, Variables
variable notations, Command-Line Arguments
variables, Variables

systemtap-runtime package, Installing SystemTap

systemtap-testsuite package

sample scripts, Useful SystemTap Scripts

T

tallying function calls

examples of SystemTap scripts, Counting Function Calls Made

tallying virtual file system reads (VFS reads)

incrementing associated values

array operations, Incrementing Associated Values

Tapsets

definition of, Tapsets

target kernel

cross-instrumentation, Generating Instrumentation for Other Computers

target system

cross-instrumentation, Generating Instrumentation for Other Computers

target system and host system

configuration, Generating Instrumentation for Other Computers

target variables, Target Variables

pretty printing, Pretty Printing Target Variables
typecasting, Typecasting
user-space, Accessing User-Space Target Variables
variable availability, Checking Target Variable Availablility

target()

functions, Systemtap Handler/Body

target-symbol expression, unresolved

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

TCP connections (incoming), monitoring

examples of SystemTap scripts, Monitoring Incoming TCP Connections

TCP packets, monitoring

examples of SystemTap scripts, Monitoring TCP Packets

testing for array membership

conditional statements, using arrays in

array operations, Using Arrays in Conditional Statements

testing, initial, Initial Testing

thread_indent()

functions, Systemtap Handler/Body

thread_indent(), sample usage

examples of SystemTap scripts, Call Graph Tracing

tid()

functions, Systemtap Handler/Body

time of I/O

examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write

time spent in kernel/user space, determining

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

timer events

Events, Event

timer.ms(), sample usage

examples of SystemTap scripts, Counting Function Calls Made

timer.s(), sample usage

examples of SystemTap scripts, Monitoring Polling Applications, Tracking Most Frequently Used System Calls

timestamp deltas, computing for

reading values from arrays

array operations, Reading Values From Arrays

timestamps, association thereof to process names

assigning associated values

array operations, Assigning an Associated Value

tracepoint, Event, Monitoring Network Packets Drops in Kernel

tracing call graph

examples of SystemTap scripts, Call Graph Tracing

tracing functions called in network socket code

examples of SystemTap scripts, Tracing Functions Called in Network Socket Code

tracing incoming/outgoing function calls

examples of SystemTap scripts, Call Graph Tracing

tracking cumulative I/O

examples of SystemTap scripts, Track Cumulative IO

trigger function

examples of SystemTap scripts, Call Graph Tracing

type mismatch for identifier

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

typographical script error

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

U

uid()

functions, Systemtap Handler/Body

uname -m, Generating Instrumentation for Other Computers

uname -r, Installing Required Kernel Information RPMs

understainding SystemTap errors

runtime errors/warnings, Run Time Errors and Warnings

aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings

Understanding How SystemTap Works, Understanding How SystemTap Works

architecture, Architecture
event types, Understanding How SystemTap Works
events and handlers, Understanding How SystemTap Works
SystemTap sessions, Architecture

understanding SystemTap errors

parse/semantics error, Parse and Semantic Errors

embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors

unique keys

introduction

arrays, Associative Arrays

unprivileged script, embedded code in

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unresolved function call

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unresolved target-symbol expression

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unresolved type for identifier

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

unsafe embedded code in unprivileged script

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

Usage

options, stap, Running SystemTap Scripts
running SystemTap scripts, Running SystemTap Scripts
standard input, running scripts from, Running SystemTap Scripts
stap, Running SystemTap Scripts
stapdev, Running SystemTap Scripts
staprun, Running SystemTap Scripts
stapusr, Running SystemTap Scripts

useful examples of SystemTap scripts, Useful SystemTap Scripts

user and kernel space, determining time spent in

examples of SystemTap scripts, Determining Time Spent in Kernel and User Space

using arrays in simple computations

reading values from arrays

array operations, Reading Values From Arrays

Using SystemTap, Using SystemTap

usrdev2kerndev()

examples of SystemTap scripts, I/O Monitoring (By Device)

V

values, assignment of

array operations, Assigning an Associated Value

variable modified during 'foreach'

parse/semantics error

understanding SystemTap errors, Parse and Semantic Errors

variable notations

SystemTap handler constructs

handlers, Command-Line Arguments

variables

SystemTap handler constructs

handlers, Variables

variables (local)

name, Systemtap Handler/Body

sample usage

$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO

VFS reads, tallying of

incrementing associated values

array operations, Incrementing Associated Values

vfs.file_operation

Events, Event

virtual file system reads (cumulative), tallying

processing multiple elements in an array

array operations, Processing Multiple Elements in an Array

virtual file system reads (non-cumulative), tallying

clearing arrays/array elements

array operations, Clearing/Deleting Arrays and Array Elements

W

while loops

conditional statements

handlers, Conditional Statements

whole device number (usage as a command-line argument)

examples of SystemTap scripts, I/O Monitoring (By Device)

wildcards in events, Event

writes/reads to a file, monitoring

examples of SystemTap scripts, Monitoring Reads and Writes to a File

Preface

1.Â Document Conventions

1.1.Â Typographic Conventions

1.2.Â Pull-quote Conventions

1.3.Â Notes and Warnings

Note

Important

Warning

Revision History

3.5.Â Array Operations in SystemTap

3.5.1.Â Assigning an Associated Value

ExampleÂ 3.14.Â Associating Timestamps to Process Names

3.5.7.Â Computing for Statistical Aggregates

ExampleÂ 3.21.Â stat-aggregates.stp

Note

ExampleÂ 3.22.Â Multiple Array Indexes

3.5.6.Â Using Arrays in Conditional Statements

ExampleÂ 3.19.Â vfsreads-print-if-1kb.stp

Testing for Membership

ExampleÂ 3.20.Â vfsreads-stop-on-stapio2.stp

3.5.5.Â Clearing/Deleting Arrays and Array Elements

ExampleÂ 3.18.Â noncumulative-vfsreads.stp

Note

3.5.4.Â Processing Multiple Elements in an Array

ExampleÂ 3.17.Â cumulative-vfsreads.stp

3.5.3.Â Incrementing Associated Values

ExampleÂ 3.16.Â vfsreads.stp

3.5.2.Â Reading Values From Arrays

ExampleÂ 3.15.Â Using Array Values in Simple Computations

Note

3.4.Â Associative Arrays

ExampleÂ 3.13.Â Basic Array Statements

Important

3.3.4.Â Command-Line Arguments

ExampleÂ 3.12.Â commandlineargs.stp

2.2.Â Generating Instrumentation for Other Computers

Note

ProcedureÂ 2.1.Â Configuring a Host System and Target Systems

Note

Important

ChapterÂ 6.Â Understanding SystemTap Errors

6.1.Â Parse and Semantic Errors

parse error: expected foo, saw bar

parse error: embedded code in unprivileged script

semantic error: type mismatch for identifier 'foo' ... string vs. long

ExampleÂ 6.1.Â error-variable.stp

semantic error: unresolved type for identifier 'foo'

semantic error: Expecting symbol or array index expression

while searching for arity N function, semantic error: unresolved function call

semantic error: array locals not supported, missing global declaration?

semantic error: variable âfooâ modiï¬ed during âforeachâ iteration

semantic error: probe point mismatch at position N, while resolving probe point foo

semantic error: no match for probe point, while resolving probe point foo

semantic error: unresolved target-symbol expression

semantic error: libdwï¬ failure

semantic error: cannot find foo debuginfo

5.4.Â Identifying Contended User-Space Locks

futexes.stp

ExampleÂ 5.19.Â futexes.stp Sample Output

3.3.3.Â Conditional Statements

ExampleÂ 3.11.Â ifelse.stp

Conditional Operators

SystemTap Beginners Guide

Introduction to SystemTap (for Fedora)

+ Logo + +

+ Red Hat, Inc. +

Don Domingo

William Cohen

Legal Notice

Abstract

5.2.6.Â Monitoring Changes to File Attributes

inodewatch2.stp

ExampleÂ 5.11.Â inodewatch2.stp Sample Output

5.2.5.Â Monitoring Reads and Writes to a File

inodewatch.stp

ExampleÂ 5.10.Â inodewatch.stp Sample Output

1.2.Â SystemTap Capabilities

Limitations

ChapterÂ 1.Â Introduction

1.1.Â Documentation Goals

parse error: expected `foo`, saw `bar`

semantic error: type mismatch for identifier '`foo`' ... string vs. long

semantic error: unresolved type for identifier '`foo`'

while searching for arity `N` function, semantic error: unresolved function call

semantic error: variable â`foo`â modiï¬ed during âforeachâ iteration

semantic error: probe point mismatch at position `N`, while resolving probe point `foo`

semantic error: no match for probe point, while resolving probe point `foo`

semantic error: libdwï¬ failure

semantic error: cannot find `foo` debuginfo

+ + +

WARNING: Number of errors: `N`, skipped probes: `M`

kernel/user string copy fault at `ADDR`