Preface

1.Â Document Conventions

+Preface
Prev
Next
Preface
+ +
1.Â Document Conventions
This manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information.
In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts set. The Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not, alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later includes the Liberation Fonts set by default. -
1.1.Â Typographic Conventions
+
1.1.Â Typographic Conventions
Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold @@ -20,7 +21,7 @@
Press Enter to execute the command.
- Press Ctrl+Alt+F1 to switch to the first virtual terminal. Press Ctrl+Alt+F7 to return to your X-Windows session. + Press Ctrl+Alt+F2 to switch to the first virtual terminal. Press Ctrl+Alt+F1 to return to your X-Windows session.
The first paragraph highlights the particular keycap to press. The second highlights two key combinations (each a set of three keycaps with each set pressed simultaneously).
@@ -53,13 +54,12 @@ Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system. -
1.2.Â Pull-quote Conventions
+
1.2.Â Pull-quote Conventions
Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus:
books Desktop documentation drafts mss photos stuff svn -books_tests Desktop1 downloads images notes scripts svgs -
+books_tests Desktop1 downloads images notes scripts svgs
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows:
package org.jboss.book.jca.ex1; @@ -79,13 +79,12 @@ books_tests Desktop1 downloads images notes scripts svgs System.out.println("Echo.echo('Hello') = " + echo.echo("Hello")); } -} -
1.3.Â Notes and Warnings
+}
1.3.Â Notes and Warnings
Finally, we use three visual styles to draw attention to information that might otherwise be overlooked. -
Note
+
Note
Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier. -
Important
- Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled 'Important' won't cause data loss but may cause irritation and frustration. -
Warning
+
Important
+ Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled 'Important' will not cause data loss but may cause irritation and frustration. +
Warning
Warnings should not be ignored. Ignoring warnings will most likely cause data loss. -
PrevSystemTap Beginners Guide
Up
Home
Next2.Â We Need Feedback!
+

diff --git a/SystemTap_Beginners_Guide/appe-Publican-Revision_History.html b/SystemTap_Beginners_Guide/appe-Publican-Revision_History.html index 13f81096..d7a0093d 100644 --- a/SystemTap_Beginners_Guide/appe-Publican-Revision_History.html +++ b/SystemTap_Beginners_Guide/appe-Publican-Revision_History.html @@ -1,9 +1,12 @@ -AppendixÂ A.Â Revision History
Prev
Next
Revision History
+AppendixÂ A.Â Revision History
Prev
Next
Revision History

Revision History
Revision 2.0 Mon Jul 20 2009 Don Domingo

includes 5.4 minor updates and additional script "dropwatch.stp"
+
Revision 1.0 Wed Jun 17 2009 Don Domingo

Building+pushing to RHEL
+
-
PrevChapterÂ 6.Â References
Up
Home
NextIndex
+ +
PrevChapterÂ 7.Â References
Up
Home
NextIndex
diff --git a/SystemTap_Beginners_Guide/arrayoperators.html b/SystemTap_Beginners_Guide/arrayoperators.html index cab52234..fbed3044 100644 --- a/SystemTap_Beginners_Guide/arrayoperators.html +++ b/SystemTap_Beginners_Guide/arrayoperators.html @@ -1,17 +1,13 @@ -3.5.Â Array Operations in SystemTap
Prev
Next
3.5.Â Array Operations in SystemTap
+3.5.Â Array Operations in SystemTap
Prev
Next
3.5.Â Array Operations in SystemTap
This section enumerates some of the most commonly used array operations in SystemTap. -
3.5.1.Â Assigning an Associated Value
+
3.5.1.Â Assigning an Associated Value
Use = to set an associated value to indexed unique pairs, as in: -
-array_name[index_expression] = value -
- ExampleÂ 3.11, âBasic Array Statementsâ shows a very basic example of how to set an explicit associated value to a unique key. You can also use a handler function as both your index_expression and value. For example, you can use arrays to set a timestamp as the associated value to a process name (which you wish to use as your unique key), as in: -
-foo[tid()] = gettimeofday_s() -
ExampleÂ 3.12.Â Associating Timestamps to Process Names

- Whenever an event invokes the statement in ExampleÂ 3.12, âAssociating Timestamps to Process Namesâ, SystemTap returns the appropriate tid() value (i.e. the ID of a thread, which is then used as the unique key). At the same time, SystemTap also uses the function gettimeofday_s() to set the corresponding timestamp as the associated value to the unique key defined by the function tid(). This creates an array composed of key pairs containing thread IDs and timestamps. +
array_name[index_expression] = value
+ ExampleÂ 3.13, âBasic Array Statementsâ shows a very basic example of how to set an explicit associated value to a unique key. You can also use a handler function as both your index_expression and value. For example, you can use arrays to set a timestamp as the associated value to a process name (which you wish to use as your unique key), as in: +
ExampleÂ 3.14.Â Associating Timestamps to Process Names
foo[tid()] = gettimeofday_s()

+ Whenever an event invokes the statement in ExampleÂ 3.14, âAssociating Timestamps to Process Namesâ, SystemTap returns the appropriate tid() value (i.e. the ID of a thread, which is then used as the unique key). At the same time, SystemTap also uses the function gettimeofday_s() to set the corresponding timestamp as the associated value to the unique key defined by the function tid(). This creates an array composed of key pairs containing thread IDs and timestamps.
In this same example, if tid() returns a value that is already defined in the array foo, the operator will discard the original associated value to it, and replace it with the current timestamp from gettimeofday_s().
Prev3.4.Â Associative Arrays
Up
Home
Next3.5.2.Â Reading Values From Arrays
diff --git a/SystemTap_Beginners_Guide/arrayops-aggregates.html b/SystemTap_Beginners_Guide/arrayops-aggregates.html index 7aa867b3..b7430d1d 100644 --- a/SystemTap_Beginners_Guide/arrayops-aggregates.html +++ b/SystemTap_Beginners_Guide/arrayops-aggregates.html @@ -1,35 +1,32 @@ -3.5.7.Â Computing for Statistical Aggregates
Prev
Next
3.5.7.Â Computing for Statistical Aggregates
+3.5.7.Â Computing for Statistical Aggregates
Prev
Next
3.5.7.Â Computing for Statistical Aggregates
Statistical aggregates are used to collect statistics on numerical values where it is important to accumulate new data quickly and in large volume (i.e. storing only aggregated stream statistics). Statistical aggregates can be used in global variables or as elements in an array. -
+
To add value to a statistical aggregate, use the operator <<< value. -
-global reads +
ExampleÂ 3.21.Â stat-aggregates.stp
global reads probe vfs.read { reads[execname()] <<< count -} -
ExampleÂ 3.19.Â stat-aggregates.stp

- In ExampleÂ 3.19, âstat-aggregates.stpâ, the operator <<< count stores the amount returned by count to to the associated value of the corresponding execname() in the reads array. Remember, these values are stored; they are not added to the associated values of each unique key, nor are they used to replace the current associated values. In a manner of speaking, think of it as having each unique key (execname()) having multiple associated values, accumulating with each probe handler run. -
Note
- In the context of ExampleÂ 3.19, âstat-aggregates.stpâ, count returns the amount of data written by the returned execname() to the virtual file system. -
+}

+ In ExampleÂ 3.21, âstat-aggregates.stpâ, the operator <<< count stores the amount returned by count to to the associated value of the corresponding execname() in the reads array. Remember, these values are stored; they are not added to the associated values of each unique key, nor are they used to replace the current associated values. In a manner of speaking, think of it as having each unique key (execname()) having multiple associated values, accumulating with each probe handler run. +
Note
+ In the context of ExampleÂ 3.21, âstat-aggregates.stpâ, count returns the amount of data written by the returned execname() to the virtual file system. +
To extract data collected by statistical aggregates, use the syntax format @extractor(variable/array index expression). extractor can be any of the following integer extractors: -
count
- Returns the number of all values stored into the variable/array index expression. Given the sample probe in ExampleÂ 3.19, âstat-aggregates.stpâ, the expression @count(writes[execname()]) will return how many values are stored in each unique key in array writes. -
sum
- Returns the sum of all values stored into the variable/array index expression. Again, given sample probe in ExampleÂ 3.19, âstat-aggregates.stpâ, the expression @sum(writes[execname()]) will return the total of all values stored in each unique key in array writes. -
min
+
count
+ Returns the number of all values stored into the variable/array index expression. Given the sample probe in ExampleÂ 3.21, âstat-aggregates.stpâ, the expression @count(writes[execname()]) will return how many values are stored in each unique key in array writes. +
sum
+ Returns the sum of all values stored into the variable/array index expression. Again, given sample probe in ExampleÂ 3.21, âstat-aggregates.stpâ, the expression @sum(writes[execname()]) will return the total of all values stored in each unique key in array writes. +
min
Returns the smallest among all the values stored in the variable/array index expression. -
max
+
max
Returns the largest among all the values stored in the variable/array index expression. -
avg
+
avg
Returns the average of all values stored in the variable/array index expression.
When using statistical aggregates, you can also build array constructs that use multiple index expressions (to a maximum of 5). This is helpful in capturing additional contextual information during a probe. For example: -
-global reads +
ExampleÂ 3.22.Â Multiple Array Indexes
global reads probe vfs.read { reads[execname(),pid()] <<< 1 @@ -38,9 +35,8 @@ probe timer.s(3) { foreach([var1,var2] in reads) printf("%s (%d) : %d \n", var1, var2, @count(reads[var1,var2])) -} -
ExampleÂ 3.20.Â Multiple Array Indexes

- In ExampleÂ 3.20, âMultiple Array Indexesâ, the first probe tracks how many times each process performs a VFS read. What makes this different from earlier examples is that this array associates a performed read to both a process name and its corresponding process ID. +}

+ In ExampleÂ 3.22, âMultiple Array Indexesâ, the first probe tracks how many times each process performs a VFS read. What makes this different from earlier examples is that this array associates a performed read to both a process name and its corresponding process ID.
- The second probe in ExampleÂ 3.20, âMultiple Array Indexesâ demonstrates how to process and print the information collected by the array reads. Note how the foreach statement uses the same number of variables (i.e. var1 and var2) contained in the first instance of the array reads from the first probe. + The second probe in ExampleÂ 3.22, âMultiple Array Indexesâ demonstrates how to process and print the information collected by the array reads. Note how the foreach statement uses the same number of variables (i.e. var1 and var2) contained in the first instance of the array reads from the first probe.
Prev3.5.6.Â Using Arrays in Conditional Statements
Up
Home
Next3.6.Â Tapsets
diff --git a/SystemTap_Beginners_Guide/arrayops-conditionals.html b/SystemTap_Beginners_Guide/arrayops-conditionals.html index 0ddf98ef..cb7be2bd 100644 --- a/SystemTap_Beginners_Guide/arrayops-conditionals.html +++ b/SystemTap_Beginners_Guide/arrayops-conditionals.html @@ -1,9 +1,8 @@ -3.5.6.Â Using Arrays in Conditional Statements
Prev
Next
3.5.6.Â Using Arrays in Conditional Statements
+3.5.6.Â Using Arrays in Conditional Statements
Prev
Next
3.5.6.Â Using Arrays in Conditional Statements
You can also use associative arrays in if statements. This is useful if you want to execute a subroutine once a value in the array matches a certain condition. Consider the following example: -
-global reads +
ExampleÂ 3.19.Â vfsreads-print-if-1kb.stp
global reads probe vfs.read { reads[execname()] ++ @@ -17,17 +16,13 @@ probe timer.s(3) printf("%s : %dkB \n", count, reads[count]/1024) else printf("%s : %dB \n", count, reads[count]) -} -
ExampleÂ 3.17.Â vfsreads-print-if-1kb.stp

- Every three seconds, ExampleÂ 3.17, âvfsreads-print-if-1kb.stpâ prints out a list of all processes, along with how many times each process performed a VFS read. If the associated value of a process name is equal or greater than 1024, the if statement in the script converts and prints it out in kB. -
Testing for Membership
+}

+ Every three seconds, ExampleÂ 3.19, âvfsreads-print-if-1kb.stpâ prints out a list of all processes, along with how many times each process performed a VFS read. If the associated value of a process name is equal or greater than 1024, the if statement in the script converts and prints it out in kB. +
Testing for Membership
You can also test whether a specific unique key is a member of an array. Further, membership in an array can be used in if statements, as in: -
-if([index_expression] in array_name) statement -
+
if([index_expression] in array_name) statement
To illustrate this, consider the following example: -
-global reads +
ExampleÂ 3.20.Â vfsreads-stop-on-stapio2.stp
global reads probe vfs.read { @@ -37,13 +32,12 @@ probe vfs.read probe timer.s(3) { printf("=======\n") - foreach (count in reads+) + foreach (count in reads+) printf("%s : %d \n", count, reads[count]) if(["stapio"] in reads) { printf("stapio read detected, exiting\n") exit() } -} -
ExampleÂ 3.18.Â vfsreads-stop-on-stapio2.stp

+}

The if(["stapio"] in reads) statement instructs the script to print stapio read detected, exiting once the unique key stapio is added to the array reads.
Prev3.5.5.Â Clearing/Deleting Arrays and Array Elements
Up
Home
Next3.5.7.Â Computing for Statistical Aggregates
diff --git a/SystemTap_Beginners_Guide/arrayops-deleting.html b/SystemTap_Beginners_Guide/arrayops-deleting.html index 5b510029..1c8db5ef 100644 --- a/SystemTap_Beginners_Guide/arrayops-deleting.html +++ b/SystemTap_Beginners_Guide/arrayops-deleting.html @@ -1,27 +1,24 @@ -3.5.5.Â Clearing/Deleting Arrays and Array Elements
Prev
Next
3.5.5.Â Clearing/Deleting Arrays and Array Elements
- Sometimes, you may need to clear the associated values in array elements, or reset an entire array for re-use in another probe. ExampleÂ 3.15, âcumulative-vfsreads.stpâ in SectionÂ 3.5.4, âProcessing Multiple Elements in an Arrayâ allows you to track how the number of VFS reads per process grows over time, but it does not show you the number of VFS reads each process makes per 3-second period. -
+3.5.5.Â Clearing/Deleting Arrays and Array Elements
Prev
Next
3.5.5.Â Clearing/Deleting Arrays and Array Elements
+ Sometimes, you may need to clear the associated values in array elements, or reset an entire array for re-use in another probe. ExampleÂ 3.17, âcumulative-vfsreads.stpâ in SectionÂ 3.5.4, âProcessing Multiple Elements in an Arrayâ allows you to track how the number of VFS reads per process grows over time, but it does not show you the number of VFS reads each process makes per 3-second period. +
To do that, you will need to clear the values accumulated by the array. You can accomplish this using the delete operator to delete elements in an array, or an entire array. Consider the following example: -
-global reads +
ExampleÂ 3.18.Â noncumulative-vfsreads.stp
global reads probe vfs.read -{ +{ reads[execname()] ++ } probe timer.s(3) { foreach (count in reads) printf("%s : %d \n", count, reads[count]) - delete reads -} -
ExampleÂ 3.16.Â noncumulative-vfsreads.stp

- In ExampleÂ 3.16, ânoncumulative-vfsreads.stpâ, the second probe prints the number of VFS reads each process made within the probed 3-second period only. The delete reads statement clears the reads array within the probe. -
Note
- You can have multiple array operations within the same probe. Using the examples from SectionÂ 3.5.4, âProcessing Multiple Elements in an Arrayâ and SectionÂ 3.5.5, âClearing/Deleting Arrays and Array Elementsâ , you can track the number of VFS reads each process makes per 3-second period and tally the cumulative VFS reads of those same processes. Consider the following example: -
-global reads, totalreads + delete reads +}

+ In ExampleÂ 3.18, ânoncumulative-vfsreads.stpâ, the second probe prints the number of VFS reads each process made within the probed 3-second period only. The delete reads statement clears the reads array within the probe. +
Note
+ You can have multiple array operations within the same probe. Using the examples from SectionÂ 3.5.4, âProcessing Multiple Elements in an Arrayâ and SectionÂ 3.5.5, âClearing/Deleting Arrays and Array Elementsâ , you can track the number of VFS reads each process makes per 3-second period and tally the cumulative VFS reads of those same processes. Consider the following example: +
global reads, totalreads probe vfs.read { @@ -32,7 +29,7 @@ probe vfs.read probe timer.s(3) { printf("=======\n") - foreach (count in reads-) + foreach (count in reads-) printf("%s : %d \n", count, reads[count]) delete reads } @@ -42,7 +39,6 @@ probe end printf("TOTALS\n") foreach (total in totalreads-) printf("%s : %d \n", total, totalreads[total]) -} -
+}
In this example, the arrays reads and totalreads track the same information, and are printed out in a similar fashion. The only difference here is that reads is cleared every 3-second period, whereas totalreads keeps growing. -
Prev3.5.4.Â Processing Multiple Elements in an Array
Up
Home
Next3.5.6.Â Using Arrays in Conditional Statements
+
Prev3.5.4.Â Processing Multiple Elements in an Array
Up
Home
Next3.5.6.Â Using Arrays in Conditional Statements
diff --git a/SystemTap_Beginners_Guide/arrayops-foreach.html b/SystemTap_Beginners_Guide/arrayops-foreach.html index cc37dd95..5df5240b 100644 --- a/SystemTap_Beginners_Guide/arrayops-foreach.html +++ b/SystemTap_Beginners_Guide/arrayops-foreach.html @@ -1,34 +1,30 @@ -3.5.4.Â Processing Multiple Elements in an Array
Prev
Next
3.5.4.Â Processing Multiple Elements in an Array
- Once you've collected enough information in an array, you will need to retrieve and process all elements in that array to make it useful. Consider ExampleÂ 3.14, âvfsreads.stpâ: the script collects information about how many VFS reads each process performs, but does not specify what to do with it. The obvious means for making ExampleÂ 3.14, âvfsreads.stpâ useful is to print the key pairs in the array reads, but how? -
+3.5.4.Â Processing Multiple Elements in an Array
Prev
Next
3.5.4.Â Processing Multiple Elements in an Array
+ Once you've collected enough information in an array, you will need to retrieve and process all elements in that array to make it useful. Consider ExampleÂ 3.16, âvfsreads.stpâ: the script collects information about how many VFS reads each process performs, but does not specify what to do with it. The obvious means for making ExampleÂ 3.16, âvfsreads.stpâ useful is to print the key pairs in the array reads, but how? +
The best way to process all key pairs in an array (as an iteration) is to use the foreach statement. Consider the following example: -
-global reads +
ExampleÂ 3.17.Â cumulative-vfsreads.stp
global reads probe vfs.read -{ +{ reads[execname()] ++ } probe timer.s(3) { foreach (count in reads) printf("%s : %d \n", count, reads[count]) -} -
ExampleÂ 3.15.Â cumulative-vfsreads.stp

- In the second probe of ExampleÂ 3.15, âcumulative-vfsreads.stpâ, the foreach statement uses the variable count to reference each iteration of a unique key in the array reads. The reads[count] array statement in the same probe retrieves the associated value of each unique key. +}

+ In the second probe of ExampleÂ 3.17, âcumulative-vfsreads.stpâ, the foreach statement uses the variable count to reference each iteration of a unique key in the array reads. The reads[count] array statement in the same probe retrieves the associated value of each unique key.
- Given what we know about the first probe in ExampleÂ 3.15, âcumulative-vfsreads.stpâ, the script prints VFS-read statistics every 3 seconds, displaying names of processes that performed a VFS-read along with a corresponding VFS-read count. -
- Now, remember that the foreach statement in ExampleÂ 3.15, âcumulative-vfsreads.stpâ prints all iterations of process names in the array, and in no particular order. You can instruct the script to process the iterations in a particular order by using + (ascending) or - (descending). In addition, you can also limit the number of iterations the script needs to process with the limit value option. + Given what we know about the first probe in ExampleÂ 3.17, âcumulative-vfsreads.stpâ, the script prints VFS-read statistics every 3 seconds, displaying names of processes that performed a VFS-read along with a corresponding VFS-read count. +
+ Now, remember that the foreach statement in ExampleÂ 3.17, âcumulative-vfsreads.stpâ prints all iterations of process names in the array, and in no particular order. You can instruct the script to process the iterations in a particular order by using + (ascending) or - (descending). In addition, you can also limit the number of iterations the script needs to process with the limit value option.
For example, consider the following replacement probe: -
-probe timer.s(3) +
probe timer.s(3) { foreach (count in reads- limit 10) printf("%s : %d \n", count, reads[count]) -} -
+}
This foreach statement instructs the script to process the elements in the array reads in descending order (of associated value). The limit 10 option instructs the foreach to only process the first ten iterations (i.e. print the first 10, starting with the highest value).
Prev3.5.3.Â Incrementing Associated Values
Up
Home
Next3.5.5.Â Clearing/Deleting Arrays and Array Elements
diff --git a/SystemTap_Beginners_Guide/arrayops-increment.html b/SystemTap_Beginners_Guide/arrayops-increment.html index 902cf1c0..02b0b052 100644 --- a/SystemTap_Beginners_Guide/arrayops-increment.html +++ b/SystemTap_Beginners_Guide/arrayops-increment.html @@ -1,16 +1,12 @@ -3.5.3.Â Incrementing Associated Values
Prev
Next
3.5.3.Â Incrementing Associated Values
+3.5.3.Â Incrementing Associated Values
Prev
Next
3.5.3.Â Incrementing Associated Values
Use ++ to increment the associated value of a unique key in an array, as in: -
-array_name[index_expression] ++ -
+
array_name[index_expression] ++
Again, you can also use a handler function for your index_expression. For example, if you wanted to tally how many times a specific process performed a read to the virtual file system (using the event vfs.read), you can use the following probe: -
-probe vfs.read +
ExampleÂ 3.16.Â vfsreads.stp
probe vfs.read { reads[execname()] ++ -} -
ExampleÂ 3.14.Â vfsreads.stp

- In ExampleÂ 3.14, âvfsreads.stpâ, the first time that the probe returns the process name gnome-terminal (i.e. the first time gnome-terminal performs a VFS read), that process name is set as the unique key gnome-terminal with an associated value of 1. The next time that the probe returns the process name gnome-terminal, SystemTap increments the associated value of gnome-terminal by 1. SystemTap performs this operation for all process names as the probe returns them. +}

+ In ExampleÂ 3.16, âvfsreads.stpâ, the first time that the probe returns the process name gnome-terminal (i.e. the first time gnome-terminal performs a VFS read), that process name is set as the unique key gnome-terminal with an associated value of 1. The next time that the probe returns the process name gnome-terminal, SystemTap increments the associated value of gnome-terminal by 1. SystemTap performs this operation for all process names as the probe returns them.
Prev3.5.2.Â Reading Values From Arrays
Up
Home
Next3.5.4.Â Processing Multiple Elements in an Array
diff --git a/SystemTap_Beginners_Guide/arrayops-readvalues.html b/SystemTap_Beginners_Guide/arrayops-readvalues.html index 7732581f..1e1d6c10 100644 --- a/SystemTap_Beginners_Guide/arrayops-readvalues.html +++ b/SystemTap_Beginners_Guide/arrayops-readvalues.html @@ -1,13 +1,11 @@ -3.5.2.Â Reading Values From Arrays
Prev
Next
3.5.2.Â Reading Values From Arrays
+3.5.2.Â Reading Values From Arrays
Prev
Next
3.5.2.Â Reading Values From Arrays
You can also read values from an array the same way you would read the value of a variable. To do so, include the array_name[index_expression] statement as an element in a mathematical expression. For example: -
-delta = gettimeofday_s() - foo[tid()] -
ExampleÂ 3.13.Â Using Array Values in Simple Computations

- This example assumes that the array foo was built using the construct in ExampleÂ 3.12, âAssociating Timestamps to Process Namesâ (from SectionÂ 3.5.1, âAssigning an Associated Valueâ). This sets a timestamp that will serve as a reference point, to be used in computing for delta. +
ExampleÂ 3.15.Â Using Array Values in Simple Computations
delta = gettimeofday_s() - foo[tid()]

+ This example assumes that the array foo was built using the construct in ExampleÂ 3.14, âAssociating Timestamps to Process Namesâ (from SectionÂ 3.5.1, âAssigning an Associated Valueâ). This sets a timestamp that will serve as a reference point, to be used in computing for delta.
- The construct in ExampleÂ 3.13, âUsing Array Values in Simple Computationsâ computes a value for the variable delta by subtracting the associated value of the key tid() from the current gettimeofday_s(). The construct does this by reading the value of tid() from the array. This particular construct is useful for determining the time between two events, such as the start and completion of a read operation. -
Note
- If the index_expression cannot find the unique key, it returns a value of 0 (for numerical operations, such as ExampleÂ 3.13, âUsing Array Values in Simple Computationsâ) or a null/empty string value (for string operations) by default. -
Prev3.5.Â Array Operations in SystemTap
Up
Home
Next3.5.3.Â Incrementing Associated Values
+ The construct in ExampleÂ 3.15, âUsing Array Values in Simple Computationsâ computes a value for the variable delta by subtracting the associated value of the key tid() from the current gettimeofday_s(). The construct does this by reading the value of tid() from the array. This particular construct is useful for determining the time between two events, such as the start and completion of a read operation. +
Note
+ If the index_expression cannot find the unique key, it returns a value of 0 (for numerical operations, such as ExampleÂ 3.15, âUsing Array Values in Simple Computationsâ) or a null/empty string value (for string operations) by default. +
Prev3.5.Â Array Operations in SystemTap
Up
Home
Next3.5.3.Â Incrementing Associated Values
diff --git a/SystemTap_Beginners_Guide/associativearrays.html b/SystemTap_Beginners_Guide/associativearrays.html index 47ed86e3..940925a2 100644 --- a/SystemTap_Beginners_Guide/associativearrays.html +++ b/SystemTap_Beginners_Guide/associativearrays.html @@ -1,21 +1,15 @@ -3.4.Â Associative Arrays
Prev
Next
3.4.Â Associative Arrays
+3.4.Â Associative Arrays
Prev
Next
3.4.Â Associative Arrays
SystemTap also supports the use of associative arrays. While an ordinary variable represents a single value, associative arrays can represent a collection of values. Simply put, an associative array is a collection of unique keys; each key in the array has a value associated with it. -
- Since associative arrays are normally processed in multiple probes (as we will demonstrate later), they are declared as global variables in the SystemTap script. The syntax for accessing an element in an associative array is similar to that of awk, and is as follows: -
-array_name[index_expression] -
+
+ Since associative arrays are normally processed in multiple probes (as we will demonstrate later), they should be declared as global variables in the SystemTap script. The syntax for accessing an element in an associative array is similar to that of awk, and is as follows: +
array_name[index_expression]
Here, the array_name is any arbitrary name the array uses. The index_expression is used to refer to a specific unique key in the array. To illustrate, let us try to build an array named foo that specifies the ages of three people (i.e. the unique keys): tom, dick, and harry. To assign them the ages (i.e. associated values) of 23, 24, and 25 respectively, we'd use the following array statements: -
-foo["tom"] = 23 +
ExampleÂ 3.13.Â Basic Array Statements
foo["tom"] = 23 foo["dick"] = 24 -foo["harry"] = 25 -
ExampleÂ 3.11.Â Basic Array Statements

- You can specify up to 5 index expressons in an array statement, each one delimited by a comma (,). This is useful if you wish to have a key that contains multiple pieces of information. The following line from disktop.stp uses 5 elements for the key: process ID, executable name, user ID, parent process ID, and string "W". It associates the value of devname with that key. -
-device[pid(),execname(),uid(),ppid(),"W"] = devname -
Important
+foo["harry"] = 25

+ You can specify up to nine index expressons in an array statement, each one delimited by a comma (,). This is useful if you wish to have a key that contains multiple pieces of information. The following line from disktop.stp uses 5 elements for the key: process ID, executable name, user ID, parent process ID, and string "W". It associates the value of devname with that key. +
device[pid(),execname(),uid(),ppid(),"W"] = devname
Important
All associate arrays must be declared as global, regardless of whether the associate array is used in one or multiple probes. -
Prev3.3.3.Â Command-Line Arguments
Up
Home
Next3.5.Â Array Operations in SystemTap
+
Prev3.3.4.Â Command-Line Arguments
Up
Home
Next3.5.Â Array Operations in SystemTap
diff --git a/SystemTap_Beginners_Guide/commandlineargssect.html b/SystemTap_Beginners_Guide/commandlineargssect.html index 0516f85e..1d0e7ca3 100644 --- a/SystemTap_Beginners_Guide/commandlineargssect.html +++ b/SystemTap_Beginners_Guide/commandlineargssect.html @@ -1,10 +1,8 @@ -3.3.3.Â Command-Line Arguments
Prev
Next
3.3.3.Â Command-Line Arguments
+3.3.4.Â Command-Line Arguments
Prev
Next
3.3.4.Â Command-Line Arguments
You can also allow a SystemTap script to accept simple command-line arguments using a $ or @ immediately followed by the number of the argument on the command line. Use $ if you are expecting the user to enter an integer as a command-line argument, and @ if you are expecting a string. -
-probe kernel.function(@1) { } -probe kernel.function(@1).return { } -
ExampleÂ 3.10.Â commandlineargs.stp

- ExampleÂ 3.10, âcommandlineargs.stpâ is similar to ExampleÂ 3.1, âwildcards.stpâ, except that it allows you to pass the kernel function to be probed as a command-line argument (as in stap commandlineargs.stp kernel function). You can also specify the script to accept multiple command-line arguments, noting them as @1, @2, and so on, in the order they are entered by the user. -
Prev3.3.2.Â Conditional Statements
Up
Home
Next3.4.Â Associative Arrays
+
ExampleÂ 3.12.Â commandlineargs.stp
probe kernel.function(@1) { } +probe kernel.function(@1).return { }

+ ExampleÂ 3.12, âcommandlineargs.stpâ is similar to ExampleÂ 3.1, âwildcards.stpâ, except that it allows you to pass the kernel function to be probed as a command-line argument (as in stap commandlineargs.stp kernel function). You can also specify the script to accept multiple command-line arguments, noting them as @1, @2, and so on, in the order they are entered by the user. +
Prev3.3.3.Â Conditional Statements
Up
Home
Next3.4.Â Associative Arrays
diff --git a/SystemTap_Beginners_Guide/cross-compiling.html b/SystemTap_Beginners_Guide/cross-compiling.html index f172ea36..60294bf4 100644 --- a/SystemTap_Beginners_Guide/cross-compiling.html +++ b/SystemTap_Beginners_Guide/cross-compiling.html @@ -1,56 +1,52 @@ -2.2.Â Generating Instrumentation for Other Computers
Prev
Next
2.2.Â Generating Instrumentation for Other Computers
- When users run a SystemTap script, SystemTap builds a kernel module out of that script. SystemTap then loads the module into the kernel, allowing it to extract the specified data directly from the kernel (refer to ProcedureÂ 3.1, âSystemTap Sessionâ in SectionÂ 3.1, âArchitectureâ for more information). +2.2.Â Generating Instrumentation for Other Computers
Prev
Next
2.2.Â Generating Instrumentation for Other Computers
+ When users run a SystemTap script, SystemTap builds a kernel module out of that script. SystemTap then loads the module into the kernel, allowing it to extract the specified data directly from the kernel (refer to ProcedureÂ 3.1, âSystemTap Sessionâ in SectionÂ 3.1, âArchitectureâ for more information).
- Normally, however, SystemTap scripts can only be run on systems where SystemTap is deployed (as in SectionÂ 2.1, âInstallation and Setupâ). This could mean that if you want to run SystemTap on ten systems, you would need to deploy SystemTap on all those systems. In some cases, this may be neither feasible nor desired. For instance, corporate policy may prohibit an administrator from installing RPMs that provide compilers or debug information on specific machines, which will prevent the deployment of SystemTap. + Normally, however, SystemTap scripts can only be run on systems where SystemTap is deployed (as in SectionÂ 2.1, âInstallation and Setupâ). This could mean that if you want to run SystemTap on ten systems, you would need to deploy SystemTap on all those systems. In some cases, this may be neither feasible nor desired. For instance, corporate policy may prohibit an administrator from installing RPMs that provide compilers or debug information on specific machines, which will prevent the deployment of SystemTap.
To work around this, you can resort to cross-instrumentation. Cross-instrumentation is the process of generating SystemTap instrumentation module from a SystemTap script on one computer to be used on another computer. This process offers the following benefits: -
+
The kernel information packages for various machines can be installed on a single host machine.
- Each target machine only needs one RPM to installed to use the generated SystemTap instrumentation module: systemtap-runtime. -
Note
+ Each target machine only needs one RPM to be installed to use the generated SystemTap instrumentation module: systemtap-runtime. +
Note
For the sake of simplicity, we will be using the following terms throughout this section:
- - - instrumentation module â the kernel module built from a SystemTap script; i.e. the SystemTap module is built on the host system, and will be loaded on the target kernel of target system. + + + instrumentation module â the kernel module built from a SystemTap script; i.e. the SystemTap module is built on the host system, and will be loaded on the target kernel of target system.
- - - host system â the system on which you compile the instrumentation modules (from SystemTap scripts), to be loaded on target systems. + + + host system â the system on which you compile the instrumentation modules (from SystemTap scripts), to be loaded on target systems.
- - - target system â the system for which you are building the instrumentation module (from SystemTap scripts). + + + target system â the system for which you are building the instrumentation module (from SystemTap scripts).
- - - target kernel â the kernel of the target system. This is the kernel on which you wish to load/run the instrumentation module. -
ProcedureÂ 2.1.Â Configuring a Host System and Target Systems
+ + + target kernel â the kernel of the target system. This is the kernel on which you wish to load/run the instrumentation module. +
ProcedureÂ 2.1.Â Configuring a Host System and Target Systems
Install the systemtap-runtime RPM on each target system. -
+
Determine the kernel running on each target system by running uname -r on each target system. -
- Install SystemTap on the host system. You will be building the instrumentation module for the target systems on the host system. For instructions on how to install SystemTap, refer to SectionÂ 2.1.1, âInstalling SystemTapâ. -
- Using the target kernel version determined earlier, install the target kernel and related RPMs on the host system by the method described in SectionÂ 2.1.2, âInstalling Required Kernel Information RPMsâ. If multiple target systems use different target kernels, you will need to repeat this step for each different kernel used on the target systems. +
+ Install SystemTap on the host system. You will be building the instrumentation module for the target systems on the host system. For instructions on how to install SystemTap, refer to SectionÂ 2.1.1, âInstalling SystemTapâ. +
+ Using the target kernel version determined earlier, install the target kernel and related RPMs on the host system by the method described in SectionÂ 2.1.2, âInstalling Required Kernel Information RPMsâ. If multiple target systems use different target kernels, you will need to repeat this step for each different kernel used on the target systems.
- After performing ProcedureÂ 2.1, âConfiguring a Host System and Target Systemsâ, you can now build the instrumentation module (for any target system) on the host system. + After performing ProcedureÂ 2.1, âConfiguring a Host System and Target Systemsâ, you can now build the instrumentation module (for any target system) on the host system.
To build the instrumentation module, run the following command on the host system (be sure to specify the appropriate values): -
-stap -r kernel_version script -m module_name -
+
stap -r kernel_version script -m module_name
Here, kernel_version refers to the version of the target kernel (the output of uname -r on the target machine), script refers to the script to be converted into an instrumentation module, and module_name is the desired name of the instrumentation module. -
Note
+
Note
To determine the architecture notation of a running kernel, run uname -m. -
+
Once the the instrumentation module is compiled, copy it to the target system and then load it using: -
-staprun module_name.ko -
+
staprun module_name.ko
For example, to create the instrumentation module simple.ko from a SystemTap script named simple.stp for the target kernel 2.6.18-92.1.10.el5 (on x86_64 architecture), use the following command:
stap -r 2.6.18-92.1.10.el5 -e 'probe vfs.read {exit()}' -m simple @@ -58,6 +54,6 @@ staprun module_name.ko This will create a module named simple.ko. To use the instrumentation module simple.ko, copy it to the target system and run the following command (on the target system):
staprun simple.ko -
Important
+
Important
The host system must be the same architecture and running the same distribution of Linux as the target system in order for the built instrumentation module to work. -
PrevChapterÂ 2.Â Using SystemTap
Up
Home
Next2.3.Â Running SystemTap Scripts
+
PrevChapterÂ 2.Â Using SystemTap
Up
Home
Next2.3.Â Running SystemTap Scripts
diff --git a/SystemTap_Beginners_Guide/errors.html b/SystemTap_Beginners_Guide/errors.html index 70bbd0d4..5d4ba8fd 100644 --- a/SystemTap_Beginners_Guide/errors.html +++ b/SystemTap_Beginners_Guide/errors.html @@ -1,64 +1,56 @@ -ChapterÂ 5.Â Understanding SystemTap Errors
Prev
Next
ChapterÂ 5.Â Understanding SystemTap Errors
5.1. Parse and Semantic Errors
5.2. Run Time Errors and Warnings
+ChapterÂ 6.Â Understanding SystemTap Errors
Prev
Next
ChapterÂ 6.Â Understanding SystemTap Errors
6.1. Parse and Semantic Errors
6.2. Run Time Errors and Warnings
This chapter explains the most common errors you may encounter while using SystemTap. -
5.1.Â Parse and Semantic Errors
+
6.1.Â Parse and Semantic Errors
These types of errors occur while SystemTap attempts to parse and translate the script into C, prior to being converted into a kernel module. For example type errors result from operations that assign invalid values to variables or arrays. -
parse error: expected foo, saw bar
+
parse error: expected foo, saw bar
The script contains a grammatical/typographical error. SystemTap detected type of construct that is incorrect, given the context of the probe.
The following invalid SystemTap script is missing its probe handlers:
-
-probe vfs.read +
probe vfs.read probe vfs.write -

It results in the following error message showing that the parser was expecting something other than the probe keyword in column 1 of line 2:
-
-parse error: expected one of '. , ( ? ! { = +=' +
parse error: expected one of '. , ( ? ! { = +=' saw: keyword at perror.stp:2:1 -1 parse error(s). -
-
parse error: embedded code in unprivileged script
- The script contains unsafe embedded C code (blocks of code surrounded by %{ %}. SystemTap allows you to embed C code in a script, which is useful if there are no tapsets to suit your purposes. However, embedded C constructs are not be safe; as such, SystemTap warns you with this error if such constructs appear in the script. -
+1 parse error(s). +
parse error: embedded code in unprivileged script
+ The script contains unsafe embedded C code (blocks of code surrounded by %{ %}. SystemTap allows you to embed C code in a script, which is useful if there are no tapsets to suit your purposes. However, embedded C constructs are not safe; as such, SystemTap warns you with this error if such constructs appear in the script. +
If you are sure of the safety of any similar constructs in the script and are member of stapdev group (or have root privileges), run the script in "guru" mode by using the option -g (i.e. stap -g script). -
semantic error: type mismatch for identifier 'foo' ... string vs. long
- The function foo in the script used the wrong type (i.e. %s or %d). This error will present itself in ExampleÂ 5.1, âerror-variable.stpâ, because the function execname() returns a string the format specifier should be a %s, not %d. -
-probe syscall.open +
semantic error: type mismatch for identifier 'foo' ... string vs. long
+ The function foo in the script used the wrong type (i.e. %s or %d). This error will present itself in ExampleÂ 6.1, âerror-variable.stpâ, because the function execname() returns a string the format specifier should be a %s, not %d. +
ExampleÂ 6.1.Â error-variable.stp
probe syscall.open { printf ("%d(%d) open\n", execname(), pid()) -} -
ExampleÂ 5.1.Â error-variable.stp

semantic error: unresolved type for identifier 'foo'
+}

semantic error: unresolved type for identifier 'foo'
The identifier (e.g. a variable) was used, but no type (integer or string) could be determined. This occurs, for instance, if you use a variable in a printf statement while the script never assigns a value to the variable. -
semantic error: Expecting symbol or array index expression
+
semantic error: Expecting symbol or array index expression
SystemTap could not assign a value to a variable or to a location in an array. The destination for the assignment is not a valid destination. The following example code would generate this error:
-
-probe begin { printf("x") = 1 } - +
probe begin { printf("x") = 1 }
-
while searching for arity N function, semantic error: unresolved function call
+
while searching for arity N function, semantic error: unresolved function call
A function call or array index expression in the script used an invalid number of arguments/parameters. In SystemTap arity can either refer to the number of indices for an array, or the number of parameters to a function. -
semantic error: array locals not supported, missing global declaration?
- The script used an array operation without declaring the array as a global variable (global variables can be declared after their use in Systemtap scripts). Similar messages appear if an array is used, but with inconsistent arities. -
semantic error: variable âfooâ modiï¬ed during âforeachâ iteration
+
semantic error: array locals not supported, missing global declaration?
+ The script used an array operation without declaring the array as a global variable (global variables can be declared after their use in SystemTap scripts). Similar messages appear if an array is used, but with inconsistent arities. +
semantic error: variable âfooâ modiï¬ed during âforeachâ iteration
The array foo is being modifed (being assigned to or deleted from) within an active foreach loop. This error also displays if an operation within the script performs a function call within the foreach loop. -
semantic error: probe point mismatch at position N, while resolving probe point foo
+
semantic error: probe point mismatch at position N, while resolving probe point foo
SystemTap did not understand what the event or SystemTap function foo refers to. This usually means that SystemTap could not find a match for foo in the tapset library. The N refers to the line and column of the error. -
semantic error: no match for probe point, while resolving probe point foo
- The events / handler function foo could not be resolved altogether, for a variety of reasons. This error occurs when the script contains the event kernel.function("blah"), and blah does not exist. In some cases, the error could also mean the script contains an invalid kernel file name or source line number. -
semantic error: unresolved target-symbol expression
+
semantic error: no match for probe point, while resolving probe point foo
+ The events/handler function foo could not be resolved altogether, for a variety of reasons. This error occurs when the script contains the event kernel.function("blah"), and blah does not exist. In some cases, the error could also mean the script contains an invalid kernel file name or source line number. +
semantic error: unresolved target-symbol expression
A handler in the script references a target variable, but the value of the variable could not be resolved. This error could also mean that a handler is referencing a target variable that is not valid in the context when it was referenced. This may be a result of compiler optimization of the generated code. -
semantic error: libdwï¬ failure
- There was a problem processing the debugging information. In most cases, this error results from the installation of a kernel-debuginfo RPM whose version does not match the probed kernel exactly. The installed kernel-debuginfo RPM itself may have some consistency / correctness problems. -
semantic error: cannot find foo debuginfo
+
semantic error: libdwï¬ failure
+ There was a problem processing the debugging information. In most cases, this error results from the installation of a kernel-debuginfo RPM whose version does not match the probed kernel exactly. The installed kernel-debuginfo RPM itself may have some consistency/correctness problems. +
semantic error: cannot find foo debuginfo
SystemTap could not find a suitable kernel-debuginfo at all. -
Prev4.4.Â Identifying Contended User-Space Locks
Up
Home
Next5.2.Â Run Time Errors and Warnings
+
Prev5.4.Â Identifying Contended User-Space Locks
Up
Home
Next6.2.Â Run Time Errors and Warnings
diff --git a/SystemTap_Beginners_Guide/futexcontentionsect.html b/SystemTap_Beginners_Guide/futexcontentionsect.html index 97f45e65..8f47a57c 100644 --- a/SystemTap_Beginners_Guide/futexcontentionsect.html +++ b/SystemTap_Beginners_Guide/futexcontentionsect.html @@ -1,43 +1,28 @@ -4.4.Â Identifying Contended User-Space Locks
Prev
Next
4.4.Â Identifying Contended User-Space Locks
- This section describes how to identify contended user-space locks throughout the system within a specific time period. The ability to identify contended user-space locks can help you investigate hangs that you suspect may be caused by futex contentions. -
- Simply put, a futex contention occurs when multiple processes are trying to access the same region of memory. In some cases, this can result in a deadlock between the processes in contention, thereby appearing as an application hang. -
- To do this, futexes.stp probes the futex system call. -
futexes.stp
+5.4.Â Identifying Contended User-Space Locks
Prev
Next
5.4.Â Identifying Contended User-Space Locks
+ This section describes how to identify contended user-space locks throughout the system within a specific time period. The ability to identify contended user-space locks can help you investigate poor program performance that you suspect may be caused by futex contentions. +
+ Simply put, futex contention occurs when multiple processes are trying to access the same lock variable at the same time. This can result in a poor performance because the lock serializes execution; one process obtains the lock while the other processes must wait for the lock variable to become available again. +
+ The futexes.stp script probes the futex system call to show lock contention. +
futexes.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # This script tries to identify contended user-space locks by hooking # into the futex system call. -global thread_thislock # short -global thread_blocktime # global FUTEX_WAIT = 0 /*, FUTEX_WAKE = 1 */ global lock_waits # long-lived stats on (tid,lock) blockage elapsed time global process_names # long-lived pid-to-execname mapping -probe syscall.futex { - if (op != FUTEX_WAIT) next # don't care about WAKE event originator - t = tid () - process_names[pid()] = execname() - thread_thislock[t] = $uaddr - thread_blocktime[t] = gettimeofday_us() -} - probe syscall.futex.return { - t = tid() - ts = thread_blocktime[t] - if (ts) { - elapsed = gettimeofday_us() - ts - lock_waits[pid(), thread_thislock[t]] <<< elapsed - delete thread_blocktime[t] - delete thread_thislock[t] - } + if ($op != FUTEX_WAIT) next # don't care about WAKE event originator + process_names[pid()] = execname() + elapsed = gettimeofday_us() - @entry(gettimeofday_us()) + lock_waits[pid(), $uaddr] <<< elapsed } probe end { @@ -46,25 +31,22 @@ probe end { process_names[pid], pid, lock, @count(lock_waits[pid,lock]), @avg(lock_waits[pid,lock])) } -

- futexes.stp needs to be manually stopped; upon exit, it prints the following information: + futexes.stp needs to be manually stopped; upon exit, it prints the following information:
Name and ID of the process responsible for a contention
- The region of memory it contested + The location of the contested lock variable
- How many times the region of memory was contended + How many times the lock variable was contended
Average time of contention throughout the probe
- ExampleÂ 4.19, âfutexes.stp Sample Outputâ contains an excerpt from the output of futexes.stp upon exiting the script (after approximately 20 seconds). -
-[...] + ExampleÂ 5.19, âfutexes.stp Sample Outputâ contains an excerpt from the output of futexes.stp upon exiting the script (after approximately 20 seconds). +
ExampleÂ 5.19.Â futexes.stp Sample Output
[...] automount[2825] lock 0x00bc7784 contended 18 times, 999931 avg us synergyc[3686] lock 0x0861e96c contended 192 times, 101991 avg us synergyc[3758] lock 0x08d98744 contended 192 times, 101990 avg us synergyc[3938] lock 0x0982a8b4 contended 192 times, 101997 avg us -[...] -
ExampleÂ 4.19.Â futexes.stp Sample Output

Prev4.3.6.Â Tracking System Call Volume Per Process
Up
Home
NextChapterÂ 5.Â Understanding SystemTap Errors
+[...]

Prev5.3.6.Â Tracking System Call Volume Per Process
Up
Home
NextChapterÂ 6.Â Understanding SystemTap Errors
diff --git a/SystemTap_Beginners_Guide/handlerconditionalstatements.html b/SystemTap_Beginners_Guide/handlerconditionalstatements.html index 5e8bd0f1..4b33e175 100644 --- a/SystemTap_Beginners_Guide/handlerconditionalstatements.html +++ b/SystemTap_Beginners_Guide/handlerconditionalstatements.html @@ -1,59 +1,49 @@ -3.3.2.Â Conditional Statements
Prev
Next
3.3.2.Â Conditional Statements
+3.3.3.Â Conditional Statements
Prev
Next
3.3.3.Â Conditional Statements
In some cases, the output of a SystemTap script may be too big. To address this, you need to further refine the script's logic in order to delimit the output into something more relevant or useful to your probe.
You can do this by using conditionals in handlers. SystemTap accepts the following types of conditional statements: -
If/Else Statements
+
If/Else Statements
Format: -
-if (condition) +
if (condition) statement1 else - statement2 -
- The statement1 is executed if the condition expression is non-zero. The statement2 is executed if the condition expression is zero. The else clause (else statement2)is optional. Both statement1 and statement2 can be statement blocks. -
-global countread, countnonread + statement2
+ The statement1 is executed if the condition expression is non-zero. The statement2 is executed if the condition expression is zero. The else clause (else statement2) is optional. Both statement1 and statement2 can be statement blocks. +
ExampleÂ 3.11.Â ifelse.stp
global countread, countnonread probe kernel.function("vfs_read"),kernel.function("vfs_write") { - if (probefunc()=="vfs_read") - countread ++ - else + if (probefunc()=="vfs_read") + countread ++ + else countnonread ++ } probe timer.s(5) { exit() } -probe end +probe end { printf("VFS reads total %d\n VFS writes total %d\n", countread, countnonread) -} -
ExampleÂ 3.9.Â ifelse.stp

- ExampleÂ 3.9, âifelse.stpâ is a script that counts how many virtual file system reads (vfs_read) and writes (vfs_write) the system performs within a 5-second span. When run, the script increments the value of the variable countread by 1 if the name of the function it probed matches vfs_read (as noted by the condition if (probefunc()=="vfs_read")); otherwise, it increments countnonread (else {countnonread ++}). -
While Loops
+}

+ ExampleÂ 3.11, âifelse.stpâ is a script that counts how many virtual file system reads (vfs_read) and writes (vfs_write) the system performs within a 5-second span. When run, the script increments the value of the variable countread by 1 if the name of the function it probed matches vfs_read (as noted by the condition if (probefunc()=="vfs_read")); otherwise, it increments countnonread (else {countnonread ++}). +
While Loops
Format: -
-while (condition) - statement -
+
while (condition) + statement
So long as condition is non-zero the block of statements in statement are executed. The statement is often a statement block and it must change a value so condition will eventually be zero. -
For Loops
+
For Loops
Format: -
-for (initialization; conditional; increment) statement -
+
for (initialization; conditional; increment) statement
The for loop is simply shorthand for a while loop. The following is the equivalent while loop: -
-initialization +
initialization while (conditional) { statement increment -} -
Conditional Operators
+}
Conditional Operators
Aside from == ("is equal to"), you can also use the following operators in your conditional statements: -
>=
+
>=
Greater than or equal to -
<=
+
<=
Less than or equal to -
!=
+
!=
Is not equal to -
Prev3.3.Â Basic SystemTap Handler Constructs
Up
Home
Next3.3.3.Â Command-Line Arguments
+
Prev3.3.2.3.Â Checking Target Variable Availablility
Up
Home
Next3.3.4.Â Command-Line Arguments
diff --git a/SystemTap_Beginners_Guide/index.html b/SystemTap_Beginners_Guide/index.html index 11225b44..16c808a4 100644 --- a/SystemTap_Beginners_Guide/index.html +++ b/SystemTap_Beginners_Guide/index.html @@ -1,10 +1,11 @@ -SystemTap Beginners Guide
Next
Systemtap 1.0
SystemTap Beginners Guide
Introduction to SystemTap (for Fedora 10)
Edition 2.0
+SystemTap Beginners Guide
Next
Systemtap 1.4
SystemTap Beginners Guide
Introduction to SystemTap (for Fedora)
Edition 2.0
-
+ +
Red Hat, Inc. -
Don Domingo
Engineering Services and Operations Content Services
ddomingo@redhat.com
William Cohen
Engineering Services and Operations Performance Tools
wcohen@redhat.com
Copyright Â© 2009 Red Hat, Inc
Legal Notice
+
Don Domingo
Engineering Services and Operations Content Services
ddomingo@redhat.com
William Cohen
Engineering Services and Operations Performance Tools
wcohen@redhat.com
Copyright Â© 2010 Red Hat, Inc
Legal Notice
This documentation is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -12,7 +13,6 @@ You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
For more details see the file COPYING in the source distribution of Linux. -
Abstract
-This guide provides basic instructions on how to use SystemTap to -monitor different subsystems of Fedora 10 in finer detail. The SystemTap Beginners Guideis recommended for users who have taken RHCTor have a similar level of expertise in Fedora 10. -
Preface
1. Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. We Need Feedback!
1. Introduction
1.1. Documentation Goals
1.2. SystemTap Capabilities
2. Using SystemTap
2.1. Installation and Setup
2.1.1. Installing SystemTap
2.1.2. Installing Required Kernel Information RPMs
2.1.3. Initial Testing
2.2. Generating Instrumentation for Other Computers
2.3. Running SystemTap Scripts
3. Understanding How SystemTap Works
3.1. Architecture
3.2. SystemTap Scripts
3.2.1. Event
3.2.2. Systemtap Handler/Body
3.3. Basic SystemTap Handler Constructs
3.3.1. Variables
3.3.2. Conditional Statements
3.3.3. Command-Line Arguments
3.4. Associative Arrays
3.5. Array Operations in SystemTap
3.5.1. Assigning an Associated Value
3.5.2. Reading Values From Arrays
3.5.3. Incrementing Associated Values
3.5.4. Processing Multiple Elements in an Array
3.5.5. Clearing/Deleting Arrays and Array Elements
3.5.6. Using Arrays in Conditional Statements
3.5.7. Computing for Statistical Aggregates
3.6. Tapsets
4. Useful SystemTap Scripts
4.1. Network
4.1.1. Network Profiling
4.1.2. Tracing Functions Called in Network Socket Code
4.1.3. Monitoring Incoming TCP Connections
4.1.4. Monitoring TCP Packets
4.1.5. Monitoring Network Packets Drops in Kernel
4.2. Disk
4.2.1. Summarizing Disk Read/Write Traffic
4.2.2. Tracking I/O Time For Each File Read or Write
4.2.3. Track Cumulative IO
4.2.4. I/O Monitoring (By Device)
4.2.5. Monitoring Reads and Writes to a File
4.2.6. Monitoring Changes to File Attributes
4.2.7. Periodically Print I/O Block Time
4.3. Profiling
4.3.1. Counting Function Calls Made
4.3.2. Call Graph Tracing
4.3.3. Determining Time Spent in Kernel and User Space
4.3.4. Monitoring Polling Applications
4.3.5. Tracking Most Frequently Used System Calls
4.3.6. Tracking System Call Volume Per Process
4.4. Identifying Contended User-Space Locks
5. Understanding SystemTap Errors
5.1. Parse and Semantic Errors
5.2. Run Time Errors and Warnings
6. References
A. Revision History
Index
NextPreface
+
Abstract
+ This guide provides basic instructions on how to use SystemTap to monitor different subsystems of Fedora in finer detail. The SystemTap Beginners Guide is recommended for users who have taken RHCT or have a similar level of expertise in Fedora. +
Preface
1. Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. We Need Feedback!
1. Introduction
1.1. Documentation Goals
1.2. SystemTap Capabilities
2. Using SystemTap
2.1. Installation and Setup
2.1.1. Installing SystemTap
2.1.2. Installing Required Kernel Information RPMs
2.1.3. Initial Testing
2.2. Generating Instrumentation for Other Computers
2.3. Running SystemTap Scripts
2.3.1. SystemTap Flight Recorder Mode
3. Understanding How SystemTap Works
3.1. Architecture
3.2. SystemTap Scripts
3.2.1. Event
3.2.2. Systemtap Handler/Body
3.3. Basic SystemTap Handler Constructs
3.3.1. Variables
3.3.2. Target Variables
3.3.3. Conditional Statements
3.3.4. Command-Line Arguments
3.4. Associative Arrays
3.5. Array Operations in SystemTap
3.5.1. Assigning an Associated Value
3.5.2. Reading Values From Arrays
3.5.3. Incrementing Associated Values
3.5.4. Processing Multiple Elements in an Array
3.5.5. Clearing/Deleting Arrays and Array Elements
3.5.6. Using Arrays in Conditional Statements
3.5.7. Computing for Statistical Aggregates
3.6. Tapsets
4. User-Space Probing
4.1. User-Space Events
4.2. Accessing User-Space Target Variables
4.3. User-Space Stack Backtraces
5. Useful SystemTap Scripts
5.1. Network
5.1.1. Network Profiling
5.1.2. Tracing Functions Called in Network Socket Code
5.1.3. Monitoring Incoming TCP Connections
5.1.4. Monitoring TCP Packets
5.1.5. Monitoring Network Packets Drops in Kernel
5.2. Disk
5.2.1. Summarizing Disk Read/Write Traffic
5.2.2. Tracking I/O Time For Each File Read or Write
5.2.3. Track Cumulative IO
5.2.4. I/O Monitoring (By Device)
5.2.5. Monitoring Reads and Writes to a File
5.2.6. Monitoring Changes to File Attributes
5.2.7. Periodically Print I/O Block Time
5.3. Profiling
5.3.1. Counting Function Calls Made
5.3.2. Call Graph Tracing
5.3.3. Determining Time Spent in Kernel and User Space
5.3.4. Monitoring Polling Applications
5.3.5. Tracking Most Frequently Used System Calls
5.3.6. Tracking System Call Volume Per Process
5.4. Identifying Contended User-Space Locks
6. Understanding SystemTap Errors
6.1. Parse and Semantic Errors
6.2. Run Time Errors and Warnings
7. References
A. Revision History
Index
NextPreface
diff --git a/SystemTap_Beginners_Guide/inodewatch2sect.html b/SystemTap_Beginners_Guide/inodewatch2sect.html index 828ded17..ef655f9a 100644 --- a/SystemTap_Beginners_Guide/inodewatch2sect.html +++ b/SystemTap_Beginners_Guide/inodewatch2sect.html @@ -1,29 +1,29 @@ -4.2.6.Â Monitoring Changes to File Attributes
Prev
Next
4.2.6.Â Monitoring Changes to File Attributes
+5.2.6.Â Monitoring Changes to File Attributes
Prev
Next
5.2.6.Â Monitoring Changes to File Attributes
This section describes how to monitor if any processes are changing the attributes of a targeted file, in real time. -
inodewatch2-simple.stp
+
inodewatch2.stp
-
-global ATTR_MODE = 1 - -probe kernel.function("inode_setattr") { - dev_nr = $inode->i_sb->s_dev - inode_nr = $inode->i_ino - - if (dev_nr == ($1 << 20 | $2) # major/minor device - && inode_nr == $3 - && $attr->ia_valid & ATTR_MODE) - printf ("%s(%d) %s 0x%x/%u %o %d\n", - execname(), pid(), probefunc(), dev_nr, inode_nr, $attr->ia_mode, uid()) -} +
#! /usr/bin/env stap +global ATTR_MODE = 1 + +probe kernel.function("setattr_copy")!, + kernel.function("generic_setattr")!, + kernel.function("inode_setattr") { + dev_nr = $inode->i_sb->s_dev + inode_nr = $inode->i_ino + + if (dev_nr == MKDEV($1,$2) # major/minor device + && inode_nr == $3 + && $attr->ia_valid & ATTR_MODE) + printf ("%s(%d) %s 0x%x/%u %o %d\n", + execname(), pid(), probefunc(), dev_nr, inode_nr, $attr->ia_mode, uid()) +}

- Like inodewatch-simple.stp from SectionÂ 4.2.5, âMonitoring Reads and Writes to a Fileâ, inodewatch2-simple.stp takes the targeted file's device number (in integer format) and inode number as arguments. For more information on how to retrieve this information, refer to SectionÂ 4.2.5, âMonitoring Reads and Writes to a Fileâ. + Like inodewatch.stp from SectionÂ 5.2.5, âMonitoring Reads and Writes to a Fileâ, inodewatch2.stp takes the targeted file's device number (in integer format) and inode number as arguments. For more information on how to retrieve this information, refer to SectionÂ 5.2.5, âMonitoring Reads and Writes to a Fileâ.
- The output for inodewatch2-simple.stp is similar to that of inodewatch-simple.stp, except that inodewatch2-simple.stp also contains the attribute changes to the monitored file, as well as the ID of the user responsible (uid()). ExampleÂ 4.11, âinodewatch2-simple.stp Sample Outputâ contains shows the output of inodewatch2-simple.stp while monitoring /home/joe/bigfile when user joe executes chmod 777 /home/joe/bigfile and chmod 666 /home/joe/bigfile. -
-chmod(17448) inode_setattr 0x800005/6011835 100777 500 -chmod(17449) inode_setattr 0x800005/6011835 100666 500 -
ExampleÂ 4.11.Â inodewatch2-simple.stp Sample Output

Prev4.2.5.Â Monitoring Reads and Writes to a File
Up
Home
Next4.2.7.Â Periodically Print I/O Block Time
+ The output for inodewatch2.stp is similar to that of inodewatch.stp, except that inodewatch2.stp also contains the attribute changes to the monitored file, as well as the ID of the user responsible (uid()). ExampleÂ 5.11, âinodewatch2.stp Sample Outputâ shows the output of inodewatch2.stp while monitoring /home/joe/bigfile when user joe executes chmod 777 /home/joe/bigfile and chmod 666 /home/joe/bigfile. +
ExampleÂ 5.11.Â inodewatch2.stp Sample Output
chmod(17448) inode_setattr 0x800005/6011835 100777 500 +chmod(17449) inode_setattr 0x800005/6011835 100666 500

Prev5.2.5.Â Monitoring Reads and Writes to a File
Up
Home
Next5.2.7.Â Periodically Print I/O Block Time
diff --git a/SystemTap_Beginners_Guide/inodewatchsect.html b/SystemTap_Beginners_Guide/inodewatchsect.html index a218f79b..79746724 100644 --- a/SystemTap_Beginners_Guide/inodewatchsect.html +++ b/SystemTap_Beginners_Guide/inodewatchsect.html @@ -1,40 +1,35 @@ -4.2.5.Â Monitoring Reads and Writes to a File
Prev
Next
4.2.5.Â Monitoring Reads and Writes to a File
+5.2.5.Â Monitoring Reads and Writes to a File
Prev
Next
5.2.5.Â Monitoring Reads and Writes to a File
This section describes how to monitor reads from and writes to a file in real time. -
inodewatch-simple.stp
+
inodewatch.stp
-
-probe vfs.write, vfs.read -{ - dev_nr = $file->f_dentry->d_inode->i_sb->s_dev - inode_nr = $file->f_dentry->d_inode->i_ino - if (dev_nr == ($1 << 20 | $2) # major/minor device - && inode_nr == $3) - printf ("%s(%d) %s 0x%x/%u\n", - execname(), pid(), probefunc(), dev_nr, inode_nr) -} +
#! /usr/bin/env stap +probe vfs.write, vfs.read +{ + # dev and ino are defined by vfs.write and vfs.read + if (dev == MKDEV($1,$2) # major/minor device + && ino == $3) + printf ("%s(%d) %s 0x%x/%u\n", + execname(), pid(), probefunc(), dev, ino) +}

- inodewatch-simple.stp takes the following information about the file as arguments on the command line: -
+ inodewatch.stp takes the following information about the file as arguments on the command line: +
The file's major device number.
The file's minor device number.
The file's inode number. -
+
To get this information, use stat -c '%D %i' filename, where filename is an absolute path.
For instance: if you wish to monitor /etc/crontab, run stat -c '%D %i' /etc/crontab first. This gives the following output: -
-805 1078319 -
+
805 1078319
805 is the base-16 (hexadecimal) device number. The lower two digits are the minor device number and the upper digits are the major number. 1078319 is the inode number. To start monitoring /etc/crontab, run stap inodewatch.stp 0x8 0x05 1078319 (The 0x prefixes indicate base-16 values.
- The output of this command contains the name and ID of any process performing a read/write, the function it is performing (i.e. vfs_read or vfs_write), the device number (in hex format), and the inode number. ExampleÂ 4.10, âinodewatch-simple.stp Sample Outputâ contains the output of stap inodewatch.stp 0x8 0x05 1078319 (when cat /etc/crontab is executed while the script is running) : -
-cat(16437) vfs_read 0x800005/1078319 -cat(16437) vfs_read 0x800005/1078319 -
ExampleÂ 4.10.Â inodewatch-simple.stp Sample Output

Prev4.2.4.Â I/O Monitoring (By Device)
Up
Home
Next4.2.6.Â Monitoring Changes to File Attributes
+ The output of this command contains the name and ID of any process performing a read/write, the function it is performing (i.e. vfs_read or vfs_write), the device number (in hex format), and the inode number. ExampleÂ 5.10, âinodewatch.stp Sample Outputâ contains the output of stap inodewatch.stp 0x8 0x05 1078319 (when cat /etc/crontab is executed while the script is running) : +
ExampleÂ 5.10.Â inodewatch.stp Sample Output
cat(16437) vfs_read 0x800005/1078319 +cat(16437) vfs_read 0x800005/1078319

Prev5.2.4.Â I/O Monitoring (By Device)
Up
Home
Next5.2.6.Â Monitoring Changes to File Attributes
diff --git a/SystemTap_Beginners_Guide/intro-systemtap-vs-others.html b/SystemTap_Beginners_Guide/intro-systemtap-vs-others.html index 273e4a13..22b0475a 100644 --- a/SystemTap_Beginners_Guide/intro-systemtap-vs-others.html +++ b/SystemTap_Beginners_Guide/intro-systemtap-vs-others.html @@ -1,13 +1,13 @@ -1.2.Â SystemTap Capabilities
Prev
Next
1.2.Â SystemTap Capabilities
- SystemTap was originally developed to provide functionality for Fedora 10 similar to previous Linux probing tools such as dprobes and the Linux Trace Toolkit. SystemTap aims to supplement the existing suite of Linux monitoring tools by providing users with the infrastructure to track kernel activity. In addition, SystemTap combines this capability with two things: +1.2.Â SystemTap Capabilities
Prev
Next
1.2.Â SystemTap Capabilities
+ SystemTap was originally developed to provide functionality for Fedora similar to previous Linux probing tools such as dprobes and the Linux Trace Toolkit. SystemTap aims to supplement the existing suite of Linux monitoring tools by providing users with the infrastructure to track kernel activity. In addition, SystemTap combines this capability with two things:
Flexibility: SystemTap's framework allows users to develop simple scripts for investigating and monitoring a wide variety of kernel functions, system calls, and other events that occur in kernel-space. With this, SystemTap is not so much a tool as it is a system that allows you to develop your own kernel-specific forensic and monitoring tools.
Ease-Of-Use: as mentioned earlier, SystemTap allows users to probe kernel-space events without having to resort to instrument, recompile, install, and reboot the kernel.
- Most of the SystemTap scripts enumerated in ChapterÂ 4, Useful SystemTap Scripts demonstrate system forensics and monitoring capabilities not natively available with other similar tools (such as top, oprofile, or ps). These scripts are provided to give readers extensive examples of the application of SystemTap, which in turn will educate them further on the capabilities they can employ when writing their own SystemTap scripts. -
Limitations
- The current iteration of SystemTap allows for a multitude of options when probing kernel-space events. However, SystemTap's ability to probe user-space events is quite limited. At present, the developmental efforts of the SystemTap community are geared towards improving SystemTap's user-space probing capabilities. + Most of the SystemTap scripts enumerated in ChapterÂ 5, Useful SystemTap Scripts demonstrate system forensics and monitoring capabilities not natively available with other similar tools (such as top, oprofile, or ps). These scripts are provided to give readers extensive examples of the application of SystemTap, which in turn will educate them further on the capabilities they can employ when writing their own SystemTap scripts. +
Limitations
+ The current iteration of SystemTap allows for a multitude of options when probing kernel-space events for a wide range of kernels. However, SystemTap's ability to probe user-space events is dependent on kernel support (the Utrace mechanism) that is unavailable in many kernels. Thus, only some kernel versions support user-space probing. At present, the developmental efforts of the SystemTap community are geared towards improving SystemTap's user-space probing capabilities.
PrevChapterÂ 1.Â Introduction
Up
Home
NextChapterÂ 2.Â Using SystemTap
diff --git a/SystemTap_Beginners_Guide/introduction.html b/SystemTap_Beginners_Guide/introduction.html index 876742e4..156da7a8 100644 --- a/SystemTap_Beginners_Guide/introduction.html +++ b/SystemTap_Beginners_Guide/introduction.html @@ -1,10 +1,10 @@ -ChapterÂ 1.Â Introduction
Prev
Next
ChapterÂ 1.Â Introduction
1.1. Documentation Goals
1.2. SystemTap Capabilities
- SystemTap is a tracing and probing tool that allows users to study and monitor the activities of the operating system (particularly, the kernel) in fine detail. It provides information similar to the output of tools like netstat, ps, top, and iostat; however, SystemTap is designed to provide more filtering and analysis options for collected information. +ChapterÂ 1.Â Introduction
Prev
Next
ChapterÂ 1.Â Introduction
1.1. Documentation Goals
1.2. SystemTap Capabilities
+ SystemTap is a tracing and probing tool that allows users to study and monitor the activities of the computer system (particularly, the kernel) in fine detail. It provides information similar to the output of tools like netstat, ps, top, and iostat; however, SystemTap is designed to provide more filtering and analysis options for collected information.
- For system administrators, SystemTap can be used as a performance monitoring tool for Fedora 10. It is most useful when other similar tools cannot precisely pinpoint a bottleneck in the system, requiring a deep analysis of kernel activity. In the same manner, application developers can also use SystemTap to monitor, in finer detail, how their application behaves within the Linux system. -
1.1.Â Documentation Goals
+ For system administrators, SystemTap can be used as a performance monitoring tool for Fedora. It is most useful when other similar tools cannot precisely pinpoint a bottleneck in the system, requiring a deep analysis of kernel activity. In the same manner, application developers can also use SystemTap to monitor, in finer detail, how their application behaves within the Linux system. +
1.1.Â Documentation Goals
SystemTap provides the infrastructure to monitor the running Linux kernel for detailed analysis. This can assist administrators and developers in identifying the underlying cause of a bug or performance problem.
Without SystemTap, monitoring the activity of a running kernel would require a tedious instrument, recompile, install, and reboot sequence. SystemTap is designed to eliminate this, allowing users to gather the same information by simply running user-written SystemTap scripts. @@ -13,7 +13,7 @@
To lower these barriers the SystemTap Beginners Guide was written with the following goals:
- To introduce users to SystemTap, familiarize them with its architecture, and provide setup instructions for all kernel types. + To introduce users to SystemTap, familiarize them with its architecture, and provide setup instructions.
To provide pre-written SystemTap scripts for monitoring detailed activity in different components of the system, along with instructions on how to run them and analyze their output.
Prev2.Â We Need Feedback!
Up
Home
Next1.2.Â SystemTap Capabilities
diff --git a/SystemTap_Beginners_Guide/ioblktimesect.html b/SystemTap_Beginners_Guide/ioblktimesect.html index b4025c7f..764fc51f 100644 --- a/SystemTap_Beginners_Guide/ioblktimesect.html +++ b/SystemTap_Beginners_Guide/ioblktimesect.html @@ -1,11 +1,10 @@ -4.2.7.Â Periodically Print I/O Block Time
Prev
Next
4.2.7.Â Periodically Print I/O Block Time
+5.2.7.Â Periodically Print I/O Block Time
Prev
Next
5.2.7.Â Periodically Print I/O Block Time
This section describes how to track the amount of time each block I/O requests spends waiting for completion. This is useful in determining whether there are too many outstanding block I/O operations at any given time. -
ioblktime.stp
+
ioblktime.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap global req_time, etimes @@ -34,18 +33,15 @@ probe timer.s(10), end { } delete etimes } -

- ioblktime.stp computes the average waiting time for block I/O per device, and prints a list every 10 seconds. As always, you can revise this refresh rate by editing the specified value in probe timer.s(10), end {. + ioblktime.stp computes the average waiting time for block I/O per device, and prints a list every 10 seconds. As always, you can revise this refresh rate by editing the specified value in probe timer.s(10), end {.
In some cases, there can be too many outstanding block I/O operations, at which point the script can exceed the default number of MAXMAPENTRIES. MAXMAPENTRIES is the maximum number of rows in an array if the array size is not specified explicitly when declared. If the script exceeds the default MAXMAPENTRIES value of 2048, run the script again with the stap option -DMAXMAPENTRIES=10000. -
- device rw total (us) count avg (us) +
ExampleÂ 5.12.Â ioblktime.stp Sample Output
device rw total (us) count avg (us) sda W 9659 6 1609 dm-0 W 20278 6 3379 dm-0 R 20524 5 4104 - sda R 19277 5 3855 -
ExampleÂ 4.12.Â ioblktime.stp Sample Output

- ExampleÂ 4.12, âioblktime.stp Sample Outputâ displays the device name, operations performed (rw), total wait time of all operations (total(us)), number of operations (count), and average wait time for all those operations (avg (us)). The times tallied by the script are in microseconds. -
Prev4.2.6.Â Monitoring Changes to File Attributes
Up
Home
Next4.3.Â Profiling
+ sda R 19277 5 3855

+ ExampleÂ 5.12, âioblktime.stp Sample Outputâ displays the device name, operations performed (rw), total wait time of all operations (total(us)), number of operations (count), and average wait time for all those operations (avg (us)). The times tallied by the script are in microseconds. +
Prev5.2.6.Â Monitoring Changes to File Attributes
Up
Home
Next5.3.Â Profiling
diff --git a/SystemTap_Beginners_Guide/iotimesect.html b/SystemTap_Beginners_Guide/iotimesect.html index a6b6ceda..6bb1ec0e 100644 --- a/SystemTap_Beginners_Guide/iotimesect.html +++ b/SystemTap_Beginners_Guide/iotimesect.html @@ -1,81 +1,85 @@ -4.2.2.Â Tracking I/O Time For Each File Read or Write
Prev
Next
4.2.2.Â Tracking I/O Time For Each File Read or Write
+5.2.2.Â Tracking I/O Time For Each File Read or Write
Prev
Next
5.2.2.Â Tracking I/O Time For Each File Read or Write
This section describes how to monitor the amount of time it takes for each process to read from or write to any file. This is useful if you wish to determine what files are slow to load on a given system. -
iotime.stp
+
iotime.stp
-
+
#! /usr/bin/env stap + +/* + * Copyright (C) 2006-2007 Red Hat Inc. + * + * This copyrighted material is made available to anyone wishing to use, + * modify, copy, or redistribute it subject to the terms and conditions + * of the GNU General Public License v.2. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software Foundation, + * Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + * + * Print out the amount of time spent in the read and write systemcall + * when each file opened by the process is closed. Note that the systemtap + * script needs to be running before the open operations occur for + * the script to record data. + * + * This script could be used to to find out which files are slow to load + * on a machine. e.g. + * + * stap iotime.stp -c 'firefox' + * + * Output format is: + * timestamp pid (executabable) info_type path ... + * + * 200283135 2573 (cupsd) access /etc/printcap read: 0 write: 7063 + * 200283143 2573 (cupsd) iotime /etc/printcap time: 69 + * + */ + global start -global entry_io -global fd_io global time_io -function timestamp:long() { - return gettimeofday_us() - start -} - -function proc:string() { - return sprintf("%d (%s)", pid(), execname()) -} +function timestamp:long() { return gettimeofday_us() - start } -probe begin { - start = gettimeofday_us() -} +function proc:string() { return sprintf("%d (%s)", pid(), execname()) } -global filenames -global filehandles -global fileread -global filewrite +probe begin { start = gettimeofday_us() } -probe syscall.open { - filenames[pid()] = user_string($filename) -} +global filehandles, fileread, filewrite probe syscall.open.return { + filename = user_string($filename) if ($return != -1) { - filehandles[pid(), $return] = filenames[pid()] - fileread[pid(), $return] = 0 - filewrite[pid(), $return] = 0 + filehandles[pid(), $return] = filename } else { - printf("%d %s access %s fail\n", timestamp(), proc(), filenames[pid()]) + printf("%d %s access %s fail\n", timestamp(), proc(), filename) } - delete filenames[pid()] -} - -probe syscall.read { - if ($count > 0) { - fileread[pid(), $fd] += $count - } - t = gettimeofday_us(); p = pid() - entry_io[p] = t - fd_io[p] = $fd } probe syscall.read.return { - t = gettimeofday_us(); p = pid() - fd = fd_io[p] - time_io[p,fd] <<< t - entry_io[p] -} - -probe syscall.write { - if ($count > 0) { - filewrite[pid(), $fd] += $count - } - t = gettimeofday_us(); p = pid() - entry_io[p] = t - fd_io[p] = $fd + p = pid() + fd = $fd + bytes = $return + time = gettimeofday_us() - @entry(gettimeofday_us()) + if (bytes > 0) + fileread[p, fd] += bytes + time_io[p, fd] <<< time } probe syscall.write.return { - t = gettimeofday_us(); p = pid() - fd = fd_io[p] - time_io[p,fd] <<< t - entry_io[p] + p = pid() + fd = $fd + bytes = $return + time = gettimeofday_us() - @entry(gettimeofday_us()) + if (bytes > 0) + filewrite[p, fd] += bytes + time_io[p, fd] <<< time } probe syscall.close { - if (filehandles[pid(), $fd] != "") { - printf("%d %s access %s read: %d write: %d\n", timestamp(), proc(), - filehandles[pid(), $fd], fileread[pid(), $fd], filewrite[pid(), $fd]) + if ([pid(), $fd] in filehandles) { + printf("%d %s access %s read: %d write: %d\n", + timestamp(), proc(), filehandles[pid(), $fd], + fileread[pid(), $fd], filewrite[pid(), $fd]) if (@count(time_io[pid(), $fd])) printf("%d %s iotime %s time: %d\n", timestamp(), proc(), filehandles[pid(), $fd], @sum(time_io[pid(), $fd])) @@ -83,18 +87,14 @@ probe syscall.close { delete fileread[pid(), $fd] delete filewrite[pid(), $fd] delete filehandles[pid(), $fd] - delete fd_io[pid()] - delete entry_io[pid()] delete time_io[pid(),$fd] } -

- iotime.stp tracks each time a system call opens, closes, reads from, and writes to a file. For each file any system call accesses, iotime.stp counts the number of microseconds it takes for any reads or writes to finish and tracks the amount of data (in bytes) read from or written to the file. -
- iotime.stp also uses the local variable $count to track the amount of data (in bytes) that any system call attempts to read or write. Note that $return (as used in disktop.stp from SectionÂ 4.2.1, âSummarizing Disk Read/Write Trafficâ) stores the actual amount of data read/written. $count can only be used on probes that track data reads or writes (e.g. syscall.read and syscall.write). -
-[...] + iotime.stp tracks each time a system call opens, closes, reads from, and writes to a file. For each file any system call accesses, iotime.stp counts the number of microseconds it takes for any reads or writes to finish and tracks the amount of data (in bytes) read from or written to the file. +
+ iotime.stp also uses the local variable $count to track the amount of data (in bytes) that any system call attempts to read or write. Note that $return (as used in disktop.stp from SectionÂ 5.2.1, âSummarizing Disk Read/Write Trafficâ) stores the actual amount of data read/written. $count can only be used on probes that track data reads or writes (e.g. syscall.read and syscall.write). +
ExampleÂ 5.7.Â iotime.stp Sample Output
[...] 825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0 825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9 [...] @@ -103,19 +103,18 @@ probe syscall.close { [...] 3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0 3973744 2886 (sendmail) iotime /proc/loadavg time: 11 -[...] -
ExampleÂ 4.7.Â iotime.stp Sample Output

- ExampleÂ 4.7, âiotime.stp Sample Outputâ prints out the following data: +[...]

+ ExampleÂ 5.7, âiotime.stp Sample Outputâ prints out the following data:
- A timestamp, in microseconds + A timestamp, in microseconds.
- Process ID and process name + Process ID and process name.
- An access or iotime flag + An access or iotime flag.
- The file accessed + The file accessed.
- If a process was able to read or write any data, a pair of access and iotime lines should appear together. The access line's timestamp refer to the time that a given process started accessing a file; at the end of the line, it will show the amount of data read/written (in bytes). The iotime line will show the amount of time (in microseconds) that the process took in order to perform the read or write. + If a process was able to read or write any data, a pair of access and iotime lines should appear together. The access line's timestamp refers to the time that a given process started accessing a file; at the end of the line, it will show the amount of data read/written (in bytes). The iotime line will show the amount of time (in microseconds) that the process took in order to perform the read or write.
If an access line is not followed by an iotime line, it simply means that the process did not read or write any data. -
Prev4.2.Â Disk
Up
Home
Next4.2.3.Â Track Cumulative IO
+
Prev5.2.Â Disk
Up
Home
Next5.2.3.Â Track Cumulative IO
diff --git a/SystemTap_Beginners_Guide/ix01.html b/SystemTap_Beginners_Guide/ix01.html index bed3e782..59511a48 100644 --- a/SystemTap_Beginners_Guide/ix01.html +++ b/SystemTap_Beginners_Guide/ix01.html @@ -1,3 +1,3 @@ -Index
Prev
Index
Symbols
$count
sample usage
local variables, Tracking I/O Time For Each File Read or Write
$return
sample usage
local variables, Summarizing Disk Read/Write Traffic, Track Cumulative IO
@avg (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@count (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@max (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@min (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@sum (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
A
adding values to statistical aggregates
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
advantages of cross-instrumentation, Generating Instrumentation for Other Computers
aggregate element not found
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
aggregates (statistical)
array operations, Computing for Statistical Aggregates
aggregation overflow
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
algebraic formulas using arrays
reading values from arrays
array operations, Reading Values From Arrays
architecture notation, determining, Generating Instrumentation for Other Computers
architecture of SystemTap, Architecture
array locals not supported
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
array operations
assigning associated values, Assigning an Associated Value
associating timestamps to process names, Assigning an Associated Value
associative arrays, Array Operations in SystemTap
clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements
delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
computing for statistical aggregates, Computing for Statistical Aggregates
@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates
conditional statements, using arrays in, Using Arrays in Conditional Statements
testing for array membership, Using Arrays in Conditional Statements
deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements
incrementing associated values, Incrementing Associated Values
tallying virtual file system reads (VFS reads), Incrementing Associated Values
multiple elements in an array, Processing Multiple Elements in an Array
processing multiple elements in an array, Processing Multiple Elements in an Array
cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array
reading values from arrays, Reading Values From Arrays
computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays
arrays, Associative Arrays
(see also associative arrays)
assigning associated values
array operations, Assigning an Associated Value
associating timestamps to process names, Assigning an Associated Value
associating timestamps to process names
array operations, Assigning an Associated Value
associated values
introduction
arrays, Associative Arrays
associating timestamps to process names
assigning associated values
array operations, Assigning an Associated Value
associative arrays
introduction, Associative Arrays
associated values, Associative Arrays
example, Associative Arrays
index expression, Associative Arrays
key pairs, Associative Arrays
syntax, Associative Arrays
unique keys, Associative Arrays
asynchronous events
Events, Event
B
begin
Events, Event
building instrumentation modules from SystemTap scripts, Generating Instrumentation for Other Computers
building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
C
call graph tracing
examples of SystemTap scripts, Call Graph Tracing
capabilities of SystemTap
Introduction, SystemTap Capabilities
changes to file attributes, monitoring
examples of SystemTap scripts, Monitoring Changes to File Attributes
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
command-line arguments
SystemTap handler constructs
handlers, Command-Line Arguments
compiling instrumentation/kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
components
SystemTap scripts
introduction, SystemTap Scripts
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates
computing for timestamp deltas
reading values from arrays
array operations, Reading Values From Arrays
conditional operators
conditional statements
handlers, Conditional Statements
conditional statements, using arrays in
array operations, Using Arrays in Conditional Statements
testing for array membership, Using Arrays in Conditional Statements
CONFIG_HZ, computing for, Variables
contended user-space locks (futex contentions), identifying
examples of SystemTap scripts, Identifying Contended User-Space Locks
copy fault
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
count operator
computing for statistical aggregates
array (operator), Computing for Statistical Aggregates
counting function calls
examples of SystemTap scripts, Counting Function Calls Made
CPU ticks
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
cpu()
functions, Systemtap Handler/Body
cross-compiling, Generating Instrumentation for Other Computers
cross-instrumentation
advantages of, Generating Instrumentation for Other Computers
building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
configuration
host system and target system, Generating Instrumentation for Other Computers
generating instrumentation from SystemTap scripts, Generating Instrumentation for Other Computers
host system, Generating Instrumentation for Other Computers
instrumentation module, Generating Instrumentation for Other Computers
target kernel, Generating Instrumentation for Other Computers
target system, Generating Instrumentation for Other Computers
ctime()
functions, Systemtap Handler/Body
ctime(), example of usage
script examples, Summarizing Disk Read/Write Traffic
cumulative I/O, tracking
examples of SystemTap scripts, Track Cumulative IO
cumulative virtual file system reads, tallying
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
D
delete operator
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
determining architecture notation, Generating Instrumentation for Other Computers
determining the kernel version, Installing Required Kernel Information RPMs
determining time spent in kernel and user space
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
device I/O, monitoring
examples of SystemTap scripts, I/O Monitoring (By Device)
device number of a file (integer format)
examples of SystemTap scripts, Monitoring Reads and Writes to a File
disk I/O traffic, summarizing
script examples, Summarizing Disk Read/Write Traffic
division by 0
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
documentation goals
Introduction, Documentation Goals
E
embedded code in unprivileged script
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
empty unique keys
reading values from arrays
array operations, Reading Values From Arrays
end
Events, Event
errors
parse/semantics error, Parse and Semantic Errors
embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors
runtime errors/warnings, Run Time Errors and Warnings
aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings
event types
Understanding How SystemTap Works, Understanding How SystemTap Works
Events
asynchronous events, Event
begin, Event
end, Event
examples of synchronous and asynchronous events, Event
introduction, Event
kernel.function("function"), Event
kernel.trace("tracepoint"), Event
module("module"), Event
synchronous events, Event
syscall.system_call, Event
timer events, Event
vfs.file_operation, Event
wildcards, Event
events and handlers, Understanding How SystemTap Works
events wildcards, Event
example
introduction
arrays, Associative Arrays
example of multiple command-line arguments
examples of SystemTap scripts, Call Graph Tracing
examples of synchronous and asynchronous events
Events, Event
examples of SystemTap scripts, Useful SystemTap Scripts
call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
process deadlocks (arising from futex contentions), Identifying Contended User-Space Locks
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)
exceeded MAXACTION
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
exceeded MAXNESTING
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
exit()
functions, Systemtap Handler/Body
expected symbol/array index expression
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
extracting data collected by statistical aggregates
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
F
fedoradebugurl.sh, Installing Required Kernel Information RPMs
feedback
contact information for this manual, We Need Feedback!
file attributes, monitoring changes to
examples of SystemTap scripts, Monitoring Changes to File Attributes
file device number (integer format)
examples of SystemTap scripts, Monitoring Reads and Writes to a File
file reads/writes, monitoring
examples of SystemTap scripts, Monitoring Reads and Writes to a File
for loops
conditional statements
handlers, Conditional Statements
foreach
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
format
introduction
arrays, Associative Arrays
format and syntax
printf(), Systemtap Handler/Body
SystemTap handler constructs
handlers, Variables
SystemTap scripts
introduction, SystemTap Scripts
format specifiers
printf(), Systemtap Handler/Body
format strings
printf(), Systemtap Handler/Body
function call (unresolved)
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
function calls (incoming/outgoing), tracing
examples of SystemTap scripts, Call Graph Tracing
function calls, tallying
examples of SystemTap scripts, Counting Function Calls Made
functions, Systemtap Handler/Body
cpu(), Systemtap Handler/Body
ctime(), Systemtap Handler/Body
gettimeofday_s(), Systemtap Handler/Body
pp(), Systemtap Handler/Body
SystemTap scripts
introduction, SystemTap Scripts
target(), Systemtap Handler/Body
thread_indent(), Systemtap Handler/Body
tid(), Systemtap Handler/Body
uid(), Systemtap Handler/Body
functions (used in handlers)
exit(), Systemtap Handler/Body
functions called in network socket code, tracing
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
futex contention, definition
examples of SystemTap scripts, Identifying Contended User-Space Locks
futex contentions, identifying
examples of SystemTap scripts, Identifying Contended User-Space Locks
futex system call
examples of SystemTap scripts, Identifying Contended User-Space Locks
G
gettimeofday_s()
functions, Systemtap Handler/Body
global
SystemTap handler constructs
handlers, Variables
goals, documentation
Introduction, Documentation Goals
grammatical/typographical script error
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
guru mode
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
H
handler functions, Systemtap Handler/Body
handlers
conditional statements, Conditional Statements
conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements
introduction, Systemtap Handler/Body
SystemTap handler constructs, Basic SystemTap Handler Constructs
command-line arguments, Command-Line Arguments
global, Variables
syntax and format, Basic SystemTap Handler Constructs
variable notations, Command-Line Arguments
variables, Variables
handlers and events, Understanding How SystemTap Works
SystemTap scripts
introduction, SystemTap Scripts
heaviest disk reads/writes, identifying
script examples, Summarizing Disk Read/Write Traffic
host system
cross-instrumentation, Generating Instrumentation for Other Computers
host system and target system
cross-instrumentation
configuration, Generating Instrumentation for Other Computers
I
I/O block time, monitoring
examples of SystemTap scripts, Periodically Print I/O Block Time
I/O monitoring (by device)
examples of SystemTap scripts, I/O Monitoring (By Device)
I/O time, monitoring
examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
I/O traffic, summarizing
script examples, Summarizing Disk Read/Write Traffic
identifier type mismatch
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
identifying contended user-space locks (i.e. futex contentions)
examples of SystemTap scripts, Identifying Contended User-Space Locks
identifying heaviest disk reads/writes
script examples, Summarizing Disk Read/Write Traffic
if/else
conditional statements
handlers, Conditional Statements
if/else conditionals, alternative syntax
examples of SystemTap scripts, Network Profiling
if/else statements, using arrays in
array operations, Using Arrays in Conditional Statements
incoming TCP connections, monitoring
examples of SystemTap scripts, Monitoring Incoming TCP Connections
incoming/outgoing function calls, tracing
examples of SystemTap scripts, Call Graph Tracing
incrementing associated values
array operations, Incrementing Associated Values
tallying virtual file system reads (VFS reads), Incrementing Associated Values
index expression
introduction
arrays, Associative Arrays
initial testing, Initial Testing
inode number
examples of SystemTap scripts, Monitoring Reads and Writes to a File
Installation
fedoradebugurl.sh, Installing Required Kernel Information RPMs
initial testing, Initial Testing
kernel information packages, Installing Required Kernel Information RPMs
kernel version, determining the, Installing Required Kernel Information RPMs
required packages, Installing Required Kernel Information RPMs
Setup and Installation, Installation and Setup
systemtap package, Installing SystemTap
systemtap-runtime package, Installing SystemTap
instrumentation module
cross-instrumentation, Generating Instrumentation for Other Computers
instrumentation modules from SystemTap scripts, building, Generating Instrumentation for Other Computers
integer extractors
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
Introduction
capabilities of SystemTap, SystemTap Capabilities
documentation goals, Documentation Goals
goals, documentation, Documentation Goals
limitations of SystemTap, SystemTap Capabilities
performance monitoring, Introduction
invalid division
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
invalid values to variables/arrays
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
iterations, processing elements in an array as
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
K
kernel and user space, determining time spent in
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
kernel information packages, Installing Required Kernel Information RPMs
kernel modules from SystemTap scripts, building, Generating Instrumentation for Other Computers
kernel version, determining the, Installing Required Kernel Information RPMs
kernel.function("function")
Events, Event
kernel.trace("tracepoint")
Events, Event
key pairs
introduction
arrays, Associative Arrays
L
libdwfl failure
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
limitations of SystemTap
Introduction, SystemTap Capabilities
limiting the output of foreach
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
local variables
name, Systemtap Handler/Body
sample usage
$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO
M
MAXACTION exceeded
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
MAXNESTING exceeded
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
membership (in array), testing for
conditional statements, using arrays in
array operations, Using Arrays in Conditional Statements
module("module")
Events, Event
monitoring changes to file attributes
examples of SystemTap scripts, Monitoring Changes to File Attributes
monitoring cumulative I/O
examples of SystemTap scripts, Track Cumulative IO
monitoring device I/O
examples of SystemTap scripts, I/O Monitoring (By Device)
monitoring I/O block time
examples of SystemTap scripts, Periodically Print I/O Block Time
monitoring I/O time
examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections
examples of SystemTap scripts, Monitoring Incoming TCP Connections
monitoring polling applications
examples of SystemTap scripts, Monitoring Polling Applications
monitoring reads and writes to a file
examples of SystemTap scripts, Monitoring Reads and Writes to a File
monitoring system calls
examples of SystemTap scripts, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process)
examples of SystemTap scripts, Tracking System Call Volume Per Process
monitoring TCP packets
examples of SystemTap scripts, Monitoring TCP Packets
multiple array operations within the same probe
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
multiple command-line arguments, example of
examples of SystemTap scripts, Call Graph Tracing
multiple elements in an array
array operations, Processing Multiple Elements in an Array
N
name
local variables, Systemtap Handler/Body
net/socket.c, tracing functions from
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
network profiling
examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
network socket code, tracing functions called in
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
network traffic, monitoring
examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
no match for probe point
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
non-global arrays
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
number of errors: N, skipped probes: M
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
O
operations
assigning associated values
associating timestamps to process names, Assigning an Associated Value
associative arrays, Array Operations in SystemTap
clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements
delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
computing for statistical aggregates, Computing for Statistical Aggregates
@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates
conditional statements, using arrays in, Using Arrays in Conditional Statements
testing for array membership, Using Arrays in Conditional Statements
deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements
incrementing associated values, Incrementing Associated Values
tallying virtual file system reads (VFS reads), Incrementing Associated Values
multiple elements in an array, Processing Multiple Elements in an Array
processing multiple elements in an array, Processing Multiple Elements in an Array
cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array
reading values from arrays, Reading Values From Arrays
computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays
options, stap
Usage, Running SystemTap Scripts
ordering the output of foreach
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
overflow of aggregation
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
P
packages required to run SystemTap, Installing Required Kernel Information RPMs
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors
performance monitoring
Introduction, Introduction
pointer dereference fault
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
polling applications, monitoring
examples of SystemTap scripts, Monitoring Polling Applications
pp()
functions, Systemtap Handler/Body
printf()
format specifiers, Systemtap Handler/Body
format strings, Systemtap Handler/Body
syntax and format, Systemtap Handler/Body
printing I/O activity (cumulative)
examples of SystemTap scripts, Track Cumulative IO
printing I/O block time (periodically)
examples of SystemTap scripts, Periodically Print I/O Block Time
probe mismatch
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
probe point (no match for)
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
probes
SystemTap scripts
introduction, SystemTap Scripts
process deadlocks (arising from futex contentions)
examples of SystemTap scripts, Identifying Contended User-Space Locks
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
cumulative virtual file system reads, tallying
array operations, Processing Multiple Elements in an Array
foreach
array operations, Processing Multiple Elements in an Array
limiting the output of foreach
array operations, Processing Multiple Elements in an Array
ordering the output of foreach
array operations, Processing Multiple Elements in an Array
profiling the network
examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
R
reading values from arrays
array operations, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays
computing for timestamp deltas
array operations, Reading Values From Arrays
reads/writes to a file, monitoring
examples of SystemTap scripts, Monitoring Reads and Writes to a File
required packages, Installing Required Kernel Information RPMs
RPMs required to run SystemTap, Installing Required Kernel Information RPMs
running scripts from standard input, Running SystemTap Scripts
running SystemTap scripts
Usage, Running SystemTap Scripts
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings
S
script examples
call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
process deadlocks (arising from futex contentions), Identifying Contended User-Space Locks
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)
scripts
introduction, SystemTap Scripts
components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts
sessions, SystemTap, Architecture
Setup and Installation, Installation and Setup
standard input, running scripts from
Usage, Running SystemTap Scripts
stap
Usage, Running SystemTap Scripts
stap options, Running SystemTap Scripts
stapdev
Usage, Running SystemTap Scripts
staprun
Usage, Running SystemTap Scripts
stapusr
Usage, Running SystemTap Scripts
stat -c, determining file device number (integer format)
examples of SystemTap scripts, Monitoring Reads and Writes to a File
stat -c, determining whole device number
examples of SystemTap scripts, I/O Monitoring (By Device)
statement blocks
SystemTap scripts
introduction, SystemTap Scripts
statistical aggregates
array operations, Computing for Statistical Aggregates
summarizing disk I/O traffic
script examples, Summarizing Disk Read/Write Traffic
synchronous events
Events, Event
syntax
introduction
arrays, Associative Arrays
syntax and format
printf(), Systemtap Handler/Body
SystemTap handler constructs
handlers, Basic SystemTap Handler Constructs
SystemTap scripts
introduction, SystemTap Scripts
syscall.system_call
Events, Event
system calls volume (per process), monitoring
examples of SystemTap scripts, Tracking System Call Volume Per Process
system calls, monitoring
examples of SystemTap scripts, Tracking Most Frequently Used System Calls
SystemTap architecture, Architecture
SystemTap handlers
SystemTap handler constructs, Basic SystemTap Handler Constructs
syntax and format, Basic SystemTap Handler Constructs
systemtap package, Installing SystemTap
SystemTap script functions, Systemtap Handler/Body
SystemTap scripts
introduction, SystemTap Scripts
components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts
useful examples, Useful SystemTap Scripts
SystemTap scripts, how to run, Running SystemTap Scripts
SystemTap sessions, Architecture
SystemTap statements
conditional statements, Conditional Statements
conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements
SystemTap handler constructs
command-line arguments, Command-Line Arguments
global, Variables
variable notations, Command-Line Arguments
variables, Variables
systemtap-runtime package, Installing SystemTap
systemtap-testsuite package
sample scripts, Useful SystemTap Scripts
T
tallying function calls
examples of SystemTap scripts, Counting Function Calls Made
tallying virtual file system reads (VFS reads)
incrementing associated values
array operations, Incrementing Associated Values
Tapsets
definition of, Tapsets
target kernel
cross-instrumentation, Generating Instrumentation for Other Computers
target system
cross-instrumentation, Generating Instrumentation for Other Computers
target system and host system
configuration, Generating Instrumentation for Other Computers
target()
functions, Systemtap Handler/Body
target-symbol expression, unresolved
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
TCP connections (incoming), monitoring
examples of SystemTap scripts, Monitoring Incoming TCP Connections
TCP packets, monitoring
examples of SystemTap scripts, Monitoring TCP Packets
testing for array membership
conditional statements, using arrays in
array operations, Using Arrays in Conditional Statements
testing, initial, Initial Testing
thread_indent()
functions, Systemtap Handler/Body
thread_indent(), sample usage
examples of SystemTap scripts, Call Graph Tracing
tid()
functions, Systemtap Handler/Body
time of I/O
examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
time spent in kernel/user space, determining
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
timer events
Events, Event
timer.ms(), sample usage
examples of SystemTap scripts, Counting Function Calls Made
timer.s(), sample usage
examples of SystemTap scripts, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
timestamp deltas, computing for
reading values from arrays
array operations, Reading Values From Arrays
timestamps, association thereof to process names
assigning associated values
array operations, Assigning an Associated Value
tracepoint, Event, Monitoring Network Packets Drops in Kernel
tracing call graph
examples of SystemTap scripts, Call Graph Tracing
tracing functions called in network socket code
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
tracing incoming/outgoing function calls
examples of SystemTap scripts, Call Graph Tracing
tracking cumulative I/O
examples of SystemTap scripts, Track Cumulative IO
trigger function
examples of SystemTap scripts, Call Graph Tracing
type mismatch for identifier
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
typographical script error
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
U
uid()
functions, Systemtap Handler/Body
uname -m, Generating Instrumentation for Other Computers
uname -r, Installing Required Kernel Information RPMs
understainding SystemTap errors
runtime errors/warnings, Run Time Errors and Warnings
aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings
Understanding How SystemTap Works, Understanding How SystemTap Works
architecture, Architecture
event types, Understanding How SystemTap Works
events and handlers, Understanding How SystemTap Works
SystemTap sessions, Architecture
understanding SystemTap errors
parse/semantics error, Parse and Semantic Errors
embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors
unique keys
introduction
arrays, Associative Arrays
unprivileged script, embedded code in
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unresolved function call
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unresolved target-symbol expression
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unresolved type for identifier
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unsafe embedded code in unprivileged script
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
Usage
options, stap, Running SystemTap Scripts
running SystemTap scripts, Running SystemTap Scripts
standard input, running scripts from, Running SystemTap Scripts
stap, Running SystemTap Scripts
stapdev, Running SystemTap Scripts
staprun, Running SystemTap Scripts
stapusr, Running SystemTap Scripts
useful examples of SystemTap scripts, Useful SystemTap Scripts
user and kernel space, determining time spent in
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
using arrays in simple computations
reading values from arrays
array operations, Reading Values From Arrays
Using SystemTap, Using SystemTap
usrdev2kerndev()
examples of SystemTap scripts, I/O Monitoring (By Device)
V
values, assignment of
array operations, Assigning an Associated Value
variable modified during 'foreach'
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
variable notations
SystemTap handler constructs
handlers, Command-Line Arguments
variables
SystemTap handler constructs
handlers, Variables
variables (local)
name, Systemtap Handler/Body
sample usage
$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO
VFS reads, tallying of
incrementing associated values
array operations, Incrementing Associated Values
vfs.file_operation
Events, Event
virtual file system reads (cumulative), tallying
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
virtual file system reads (non-cumulative), tallying
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
W
while loops
conditional statements
handlers, Conditional Statements
whole device number (usage as a command-line argument)
examples of SystemTap scripts, I/O Monitoring (By Device)
wildcards in events, Event
writes/reads to a file, monitoring
examples of SystemTap scripts, Monitoring Reads and Writes to a File
PrevAppendixÂ A.Â Revision History
Up
Home
+Index
Prev
Index
Symbols
$count
sample usage
local variables, Tracking I/O Time For Each File Read or Write
$return
sample usage
local variables, Summarizing Disk Read/Write Traffic, Track Cumulative IO
@avg (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@count (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@max (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@min (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@sum (integer extractor)
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
A
adding values to statistical aggregates
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
advantages of cross-instrumentation, Generating Instrumentation for Other Computers
aggregate element not found
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
aggregates (statistical)
array operations, Computing for Statistical Aggregates
aggregation overflow
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
algebraic formulas using arrays
reading values from arrays
array operations, Reading Values From Arrays
architecture notation, determining, Generating Instrumentation for Other Computers
architecture of SystemTap, Architecture
array locals not supported
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
array operations
assigning associated values, Assigning an Associated Value
associating timestamps to process names, Assigning an Associated Value
associative arrays, Array Operations in SystemTap
clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements
delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
computing for statistical aggregates, Computing for Statistical Aggregates
@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates
conditional statements, using arrays in, Using Arrays in Conditional Statements
testing for array membership, Using Arrays in Conditional Statements
deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements
incrementing associated values, Incrementing Associated Values
tallying virtual file system reads (VFS reads), Incrementing Associated Values
multiple elements in an array, Processing Multiple Elements in an Array
processing multiple elements in an array, Processing Multiple Elements in an Array
cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array
reading values from arrays, Reading Values From Arrays
computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays
arrays, Associative Arrays
(see also associative arrays)
assigning associated values
array operations, Assigning an Associated Value
associating timestamps to process names, Assigning an Associated Value
associating timestamps to process names
array operations, Assigning an Associated Value
associated values
introduction
arrays, Associative Arrays
associating timestamps to process names
assigning associated values
array operations, Assigning an Associated Value
associative arrays
introduction, Associative Arrays
associated values, Associative Arrays
example, Associative Arrays
index expression, Associative Arrays
key pairs, Associative Arrays
syntax, Associative Arrays
unique keys, Associative Arrays
asynchronous events
Events, Event
B
begin
Events, Event
building instrumentation modules from SystemTap scripts, Generating Instrumentation for Other Computers
building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
C
call graph tracing
examples of SystemTap scripts, Call Graph Tracing
capabilities of SystemTap
Introduction, SystemTap Capabilities
changes to file attributes, monitoring
examples of SystemTap scripts, Monitoring Changes to File Attributes
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
command-line arguments
SystemTap handler constructs
handlers, Command-Line Arguments
compiling instrumentation/kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
components
SystemTap scripts
introduction, SystemTap Scripts
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates
computing for timestamp deltas
reading values from arrays
array operations, Reading Values From Arrays
conditional operators
conditional statements
handlers, Conditional Statements
conditional statements, using arrays in
array operations, Using Arrays in Conditional Statements
testing for array membership, Using Arrays in Conditional Statements
CONFIG_HZ, computing for, Variables
contended user-space locks (futex contentions), identifying
examples of SystemTap scripts, Identifying Contended User-Space Locks
copy fault
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
count operator
computing for statistical aggregates
array (operator), Computing for Statistical Aggregates
counting function calls
examples of SystemTap scripts, Counting Function Calls Made
CPU ticks
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
cpu()
functions, Systemtap Handler/Body
cross-compiling, Generating Instrumentation for Other Computers
cross-instrumentation
advantages of, Generating Instrumentation for Other Computers
building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
configuration
host system and target system, Generating Instrumentation for Other Computers
generating instrumentation from SystemTap scripts, Generating Instrumentation for Other Computers
host system, Generating Instrumentation for Other Computers
instrumentation module, Generating Instrumentation for Other Computers
target kernel, Generating Instrumentation for Other Computers
target system, Generating Instrumentation for Other Computers
ctime()
functions, Systemtap Handler/Body
ctime(), example of usage
script examples, Summarizing Disk Read/Write Traffic
cumulative I/O, tracking
examples of SystemTap scripts, Track Cumulative IO
cumulative virtual file system reads, tallying
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
D
delete operator
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
determining architecture notation, Generating Instrumentation for Other Computers
determining the kernel version, Installing Required Kernel Information RPMs
determining time spent in kernel and user space
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
device I/O, monitoring
examples of SystemTap scripts, I/O Monitoring (By Device)
device number of a file (integer format)
examples of SystemTap scripts, Monitoring Reads and Writes to a File
disk I/O traffic, summarizing
script examples, Summarizing Disk Read/Write Traffic
division by 0
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
documentation goals
Introduction, Documentation Goals
E
embedded code in unprivileged script
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
empty unique keys
reading values from arrays
array operations, Reading Values From Arrays
end
Events, Event
errors
parse/semantics error, Parse and Semantic Errors
embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors
runtime errors/warnings, Run Time Errors and Warnings
aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings
event types
Understanding How SystemTap Works, Understanding How SystemTap Works
Events
asynchronous events, Event
begin, Event
end, Event
examples of synchronous and asynchronous events, Event
introduction, Event
kernel.function("function"), Event
kernel.trace("tracepoint"), Event
module("module"), Event
synchronous events, Event
syscall.system_call, Event
timer events, Event
user-space, User-Space Events
vfs.file_operation, Event
wildcards, Event
events and handlers, Understanding How SystemTap Works
events wildcards, Event
example
introduction
arrays, Associative Arrays
example of multiple command-line arguments
examples of SystemTap scripts, Call Graph Tracing
examples of synchronous and asynchronous events
Events, Event
examples of SystemTap scripts, Useful SystemTap Scripts
call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex (lock) contentions, Identifying Contended User-Space Locks
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)
exceeded MAXACTION
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
exceeded MAXNESTING
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
exit()
functions, Systemtap Handler/Body
expected symbol/array index expression
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
extracting data collected by statistical aggregates
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
F
fedoradebugurl.sh, Installing Required Kernel Information RPMs
feedback
contact information for this manual, We Need Feedback!
file attributes, monitoring changes to
examples of SystemTap scripts, Monitoring Changes to File Attributes
file device number (integer format)
examples of SystemTap scripts, Monitoring Reads and Writes to a File
file reads/writes, monitoring
examples of SystemTap scripts, Monitoring Reads and Writes to a File
flight recorder mode, SystemTap Flight Recorder Mode
file mode, File Flight Recorder
in-memory mode, In-memory Flight Recorder
for loops
conditional statements
handlers, Conditional Statements
foreach
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
format
introduction
arrays, Associative Arrays
format and syntax
printf(), Systemtap Handler/Body
SystemTap handler constructs
handlers, Variables
SystemTap scripts
introduction, SystemTap Scripts
format specifiers
printf(), Systemtap Handler/Body
format strings
printf(), Systemtap Handler/Body
function call (unresolved)
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
function calls (incoming/outgoing), tracing
examples of SystemTap scripts, Call Graph Tracing
function calls, tallying
examples of SystemTap scripts, Counting Function Calls Made
functions, Systemtap Handler/Body
cpu(), Systemtap Handler/Body
ctime(), Systemtap Handler/Body
gettimeofday_s(), Systemtap Handler/Body
pp(), Systemtap Handler/Body
SystemTap scripts
introduction, SystemTap Scripts
target(), Systemtap Handler/Body
thread_indent(), Systemtap Handler/Body
tid(), Systemtap Handler/Body
uid(), Systemtap Handler/Body
functions (used in handlers)
exit(), Systemtap Handler/Body
functions called in network socket code, tracing
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
futex (lock) contentions
examples of SystemTap scripts, Identifying Contended User-Space Locks
futex contention, definition
examples of SystemTap scripts, Identifying Contended User-Space Locks
futex contentions, identifying
examples of SystemTap scripts, Identifying Contended User-Space Locks
futex system call
examples of SystemTap scripts, Identifying Contended User-Space Locks
G
gettimeofday_s()
functions, Systemtap Handler/Body
global
SystemTap handler constructs
handlers, Variables
goals, documentation
Introduction, Documentation Goals
grammatical/typographical script error
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
guru mode
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
H
handler functions, Systemtap Handler/Body
handlers
conditional statements, Conditional Statements
conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements
introduction, Systemtap Handler/Body
SystemTap handler constructs, Basic SystemTap Handler Constructs
command-line arguments, Command-Line Arguments
global, Variables
syntax and format, Basic SystemTap Handler Constructs
variable notations, Command-Line Arguments
variables, Variables
target variables, Target Variables
handlers and events, Understanding How SystemTap Works
SystemTap scripts
introduction, SystemTap Scripts
heaviest disk reads/writes, identifying
script examples, Summarizing Disk Read/Write Traffic
host system
cross-instrumentation, Generating Instrumentation for Other Computers
host system and target system
cross-instrumentation
configuration, Generating Instrumentation for Other Computers
I
I/O block time, monitoring
examples of SystemTap scripts, Periodically Print I/O Block Time
I/O monitoring (by device)
examples of SystemTap scripts, I/O Monitoring (By Device)
I/O time, monitoring
examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
I/O traffic, summarizing
script examples, Summarizing Disk Read/Write Traffic
identifier type mismatch
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
identifying contended user-space locks (i.e. futex contentions)
examples of SystemTap scripts, Identifying Contended User-Space Locks
identifying heaviest disk reads/writes
script examples, Summarizing Disk Read/Write Traffic
if/else
conditional statements
handlers, Conditional Statements
if/else conditionals, alternative syntax
examples of SystemTap scripts, Network Profiling
if/else statements, using arrays in
array operations, Using Arrays in Conditional Statements
incoming TCP connections, monitoring
examples of SystemTap scripts, Monitoring Incoming TCP Connections
incoming/outgoing function calls, tracing
examples of SystemTap scripts, Call Graph Tracing
incrementing associated values
array operations, Incrementing Associated Values
tallying virtual file system reads (VFS reads), Incrementing Associated Values
index expression
introduction
arrays, Associative Arrays
initial testing, Initial Testing
inode number
examples of SystemTap scripts, Monitoring Reads and Writes to a File
Installation
fedoradebugurl.sh, Installing Required Kernel Information RPMs
initial testing, Initial Testing
kernel information packages, Installing Required Kernel Information RPMs
kernel version, determining the, Installing Required Kernel Information RPMs
required packages, Installing Required Kernel Information RPMs
Setup and Installation, Installation and Setup
systemtap package, Installing SystemTap
systemtap-runtime package, Installing SystemTap
instrumentation module
cross-instrumentation, Generating Instrumentation for Other Computers
instrumentation modules from SystemTap scripts, building, Generating Instrumentation for Other Computers
integer extractors
computing for statistical aggregates
array operations, Computing for Statistical Aggregates
Introduction
capabilities of SystemTap, SystemTap Capabilities
documentation goals, Documentation Goals
goals, documentation, Documentation Goals
limitations of SystemTap, SystemTap Capabilities
performance monitoring, Introduction
invalid division
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
invalid values to variables/arrays
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
iterations, processing elements in an array as
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
K
kernel and user space, determining time spent in
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
kernel information packages, Installing Required Kernel Information RPMs
kernel modules from SystemTap scripts, building, Generating Instrumentation for Other Computers
kernel version, determining the, Installing Required Kernel Information RPMs
kernel.function("function")
Events, Event
kernel.trace("tracepoint")
Events, Event
key pairs
introduction
arrays, Associative Arrays
L
libdwfl failure
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
limitations of SystemTap
Introduction, SystemTap Capabilities
limiting the output of foreach
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
local variables
name, Systemtap Handler/Body
sample usage
$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO
M
MAXACTION exceeded
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
MAXNESTING exceeded
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
membership (in array), testing for
conditional statements, using arrays in
array operations, Using Arrays in Conditional Statements
module("module")
Events, Event
monitoring changes to file attributes
examples of SystemTap scripts, Monitoring Changes to File Attributes
monitoring cumulative I/O
examples of SystemTap scripts, Track Cumulative IO
monitoring device I/O
examples of SystemTap scripts, I/O Monitoring (By Device)
monitoring I/O block time
examples of SystemTap scripts, Periodically Print I/O Block Time
monitoring I/O time
examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections
examples of SystemTap scripts, Monitoring Incoming TCP Connections
monitoring polling applications
examples of SystemTap scripts, Monitoring Polling Applications
monitoring reads and writes to a file
examples of SystemTap scripts, Monitoring Reads and Writes to a File
monitoring system calls
examples of SystemTap scripts, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process)
examples of SystemTap scripts, Tracking System Call Volume Per Process
monitoring TCP packets
examples of SystemTap scripts, Monitoring TCP Packets
multiple array operations within the same probe
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
multiple command-line arguments, example of
examples of SystemTap scripts, Call Graph Tracing
multiple elements in an array
array operations, Processing Multiple Elements in an Array
N
name
local variables, Systemtap Handler/Body
net/socket.c, tracing functions from
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
network profiling
examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
network socket code, tracing functions called in
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
network traffic, monitoring
examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
no match for probe point
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
non-global arrays
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
number of errors: N, skipped probes: M
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
O
operations
assigning associated values
associating timestamps to process names, Assigning an Associated Value
associative arrays, Array Operations in SystemTap
clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements
delete operator, Clearing/Deleting Arrays and Array Elements
multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
computing for statistical aggregates, Computing for Statistical Aggregates
@avg (integer extractor), Computing for Statistical Aggregates
@count (integer extractor), Computing for Statistical Aggregates
@max (integer extractor), Computing for Statistical Aggregates
@min (integer extractor), Computing for Statistical Aggregates
@sum (integer extractor), Computing for Statistical Aggregates
adding values to statistical aggregates, Computing for Statistical Aggregates
count (operator), Computing for Statistical Aggregates
extracting data collected by statistical aggregates, Computing for Statistical Aggregates
conditional statements, using arrays in, Using Arrays in Conditional Statements
testing for array membership, Using Arrays in Conditional Statements
deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements
incrementing associated values, Incrementing Associated Values
tallying virtual file system reads (VFS reads), Incrementing Associated Values
multiple elements in an array, Processing Multiple Elements in an Array
processing multiple elements in an array, Processing Multiple Elements in an Array
cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
foreach, Processing Multiple Elements in an Array
iterations, processing elements in an array as, Processing Multiple Elements in an Array
limiting the output of foreach, Processing Multiple Elements in an Array
ordering the output of foreach, Processing Multiple Elements in an Array
reading values from arrays, Reading Values From Arrays
computing for timestamp deltas, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays
options, stap
Usage, Running SystemTap Scripts
ordering the output of foreach
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
overflow of aggregation
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
P
packages required to run SystemTap, Installing Required Kernel Information RPMs
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors
performance monitoring
Introduction, Introduction
pointer dereference fault
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
polling applications, monitoring
examples of SystemTap scripts, Monitoring Polling Applications
pp()
functions, Systemtap Handler/Body
printf()
format specifiers, Systemtap Handler/Body
format strings, Systemtap Handler/Body
syntax and format, Systemtap Handler/Body
printing I/O activity (cumulative)
examples of SystemTap scripts, Track Cumulative IO
printing I/O block time (periodically)
examples of SystemTap scripts, Periodically Print I/O Block Time
probe mismatch
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
probe point (no match for)
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
probes
SystemTap scripts
introduction, SystemTap Scripts
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
cumulative virtual file system reads, tallying
array operations, Processing Multiple Elements in an Array
foreach
array operations, Processing Multiple Elements in an Array
limiting the output of foreach
array operations, Processing Multiple Elements in an Array
ordering the output of foreach
array operations, Processing Multiple Elements in an Array
profiling the network
examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
R
reading values from arrays
array operations, Reading Values From Arrays
empty unique keys, Reading Values From Arrays
using arrays in simple computations, Reading Values From Arrays
computing for timestamp deltas
array operations, Reading Values From Arrays
reads/writes to a file, monitoring
examples of SystemTap scripts, Monitoring Reads and Writes to a File
required packages, Installing Required Kernel Information RPMs
RPMs required to run SystemTap, Installing Required Kernel Information RPMs
running scripts from standard input, Running SystemTap Scripts
running SystemTap scripts
Usage, Running SystemTap Scripts
runtime errors/warnings
understainding SystemTap errors, Run Time Errors and Warnings
aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings
S
script examples
call graph tracing, Call Graph Tracing
CPU ticks, Determining Time Spent in Kernel and User Space
ctime(), example of usage, Summarizing Disk Read/Write Traffic
determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
file device number (integer format), Monitoring Reads and Writes to a File
futex (lock) contentions, Identifying Contended User-Space Locks
futex system call, Identifying Contended User-Space Locks
identifying contended user-space locks (i.e. futex contentions), Identifying Contended User-Space Locks
if/else conditionals, alternative syntax, Network Profiling
inode number, Monitoring Reads and Writes to a File
monitoring changes to file attributes, Monitoring Changes to File Attributes
monitoring device I/O, I/O Monitoring (By Device)
monitoring I/O block time, Periodically Print I/O Block Time
monitoring I/O time, Tracking I/O Time For Each File Read or Write
monitoring incoming TCP connections, Monitoring Incoming TCP Connections
monitoring polling applications, Monitoring Polling Applications
monitoring reads and writes to a file, Monitoring Reads and Writes to a File
monitoring system calls, Tracking Most Frequently Used System Calls
monitoring system calls (volume per process), Tracking System Call Volume Per Process
monitoring TCP packets, Monitoring TCP Packets
multiple command-line arguments, example of, Call Graph Tracing
net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
stat -c, determining whole device number, I/O Monitoring (By Device)
summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
tallying function calls, Counting Function Calls Made
thread_indent(), sample usage, Call Graph Tracing
timer.ms(), sample usage, Counting Function Calls Made
timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
tracking cumulative I/O, Track Cumulative IO
trigger function, Call Graph Tracing
usrdev2kerndev(), I/O Monitoring (By Device)
whole device number (usage as a command-line argument), I/O Monitoring (By Device)
scripts
introduction, SystemTap Scripts
components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts
sessions, SystemTap, Architecture
Setup and Installation, Installation and Setup
Stack backtrace
user-space, User-Space Stack Backtraces
standard input, running scripts from
Usage, Running SystemTap Scripts
stap
Usage, Running SystemTap Scripts
stap options, Running SystemTap Scripts
stapdev
Usage, Running SystemTap Scripts
staprun
Usage, Running SystemTap Scripts
stapusr
Usage, Running SystemTap Scripts
stat -c, determining file device number (integer format)
examples of SystemTap scripts, Monitoring Reads and Writes to a File
stat -c, determining whole device number
examples of SystemTap scripts, I/O Monitoring (By Device)
statement blocks
SystemTap scripts
introduction, SystemTap Scripts
statistical aggregates
array operations, Computing for Statistical Aggregates
summarizing disk I/O traffic
script examples, Summarizing Disk Read/Write Traffic
synchronous events
Events, Event
syntax
introduction
arrays, Associative Arrays
syntax and format
printf(), Systemtap Handler/Body
SystemTap handler constructs
handlers, Basic SystemTap Handler Constructs
SystemTap scripts
introduction, SystemTap Scripts
syscall.system_call
Events, Event
system calls volume (per process), monitoring
examples of SystemTap scripts, Tracking System Call Volume Per Process
system calls, monitoring
examples of SystemTap scripts, Tracking Most Frequently Used System Calls
SystemTap architecture, Architecture
SystemTap handlers
SystemTap handler constructs, Basic SystemTap Handler Constructs
syntax and format, Basic SystemTap Handler Constructs
systemtap package, Installing SystemTap
SystemTap script functions, Systemtap Handler/Body
SystemTap scripts
introduction, SystemTap Scripts
components, SystemTap Scripts
events and handlers, SystemTap Scripts
format and syntax, SystemTap Scripts
functions, SystemTap Scripts
probes, SystemTap Scripts
statement blocks, SystemTap Scripts
useful examples, Useful SystemTap Scripts
SystemTap scripts, how to run, Running SystemTap Scripts
SystemTap sessions, Architecture
SystemTap statements
conditional statements, Conditional Statements
conditional operators, Conditional Statements
for loops, Conditional Statements
if/else, Conditional Statements
while loops, Conditional Statements
SystemTap handler constructs
command-line arguments, Command-Line Arguments
global, Variables
variable notations, Command-Line Arguments
variables, Variables
systemtap-runtime package, Installing SystemTap
systemtap-testsuite package
sample scripts, Useful SystemTap Scripts
T
tallying function calls
examples of SystemTap scripts, Counting Function Calls Made
tallying virtual file system reads (VFS reads)
incrementing associated values
array operations, Incrementing Associated Values
Tapsets
definition of, Tapsets
target kernel
cross-instrumentation, Generating Instrumentation for Other Computers
target system
cross-instrumentation, Generating Instrumentation for Other Computers
target system and host system
configuration, Generating Instrumentation for Other Computers
target variables, Target Variables
pretty printing, Pretty Printing Target Variables
typecasting, Typecasting
user-space, Accessing User-Space Target Variables
variable availability, Checking Target Variable Availablility
target()
functions, Systemtap Handler/Body
target-symbol expression, unresolved
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
TCP connections (incoming), monitoring
examples of SystemTap scripts, Monitoring Incoming TCP Connections
TCP packets, monitoring
examples of SystemTap scripts, Monitoring TCP Packets
testing for array membership
conditional statements, using arrays in
array operations, Using Arrays in Conditional Statements
testing, initial, Initial Testing
thread_indent()
functions, Systemtap Handler/Body
thread_indent(), sample usage
examples of SystemTap scripts, Call Graph Tracing
tid()
functions, Systemtap Handler/Body
time of I/O
examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
time spent in kernel/user space, determining
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
timer events
Events, Event
timer.ms(), sample usage
examples of SystemTap scripts, Counting Function Calls Made
timer.s(), sample usage
examples of SystemTap scripts, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
timestamp deltas, computing for
reading values from arrays
array operations, Reading Values From Arrays
timestamps, association thereof to process names
assigning associated values
array operations, Assigning an Associated Value
tracepoint, Event, Monitoring Network Packets Drops in Kernel
tracing call graph
examples of SystemTap scripts, Call Graph Tracing
tracing functions called in network socket code
examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
tracing incoming/outgoing function calls
examples of SystemTap scripts, Call Graph Tracing
tracking cumulative I/O
examples of SystemTap scripts, Track Cumulative IO
trigger function
examples of SystemTap scripts, Call Graph Tracing
type mismatch for identifier
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
typographical script error
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
U
uid()
functions, Systemtap Handler/Body
uname -m, Generating Instrumentation for Other Computers
uname -r, Installing Required Kernel Information RPMs
understainding SystemTap errors
runtime errors/warnings, Run Time Errors and Warnings
aggregate element not found, Run Time Errors and Warnings
aggregation overflow, Run Time Errors and Warnings
copy fault, Run Time Errors and Warnings
division by 0, Run Time Errors and Warnings
MAXACTION exceeded, Run Time Errors and Warnings
MAXNESTING exceeded, Run Time Errors and Warnings
number of errors: N, skipped probes: M, Run Time Errors and Warnings
pointer dereference fault, Run Time Errors and Warnings
Understanding How SystemTap Works, Understanding How SystemTap Works
architecture, Architecture
event types, Understanding How SystemTap Works
events and handlers, Understanding How SystemTap Works
SystemTap sessions, Architecture
understanding SystemTap errors
parse/semantics error, Parse and Semantic Errors
embedded code in unprivileged script, Parse and Semantic Errors
expected symbol/array index expression, Parse and Semantic Errors
grammatical/typographical script error, Parse and Semantic Errors
guru mode, Parse and Semantic Errors
invalid values to variables/arrays, Parse and Semantic Errors
libdwfl failure, Parse and Semantic Errors
no match for probe point, Parse and Semantic Errors
non-global arrays, Parse and Semantic Errors
probe mismatch, Parse and Semantic Errors
type mismatch for identifier, Parse and Semantic Errors
unresolved function call, Parse and Semantic Errors
unresolved target-symbol expression, Parse and Semantic Errors
unresolved type for identifier, Parse and Semantic Errors
variable modified during 'foreach', Parse and Semantic Errors
unique keys
introduction
arrays, Associative Arrays
unprivileged script, embedded code in
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unresolved function call
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unresolved target-symbol expression
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unresolved type for identifier
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
unsafe embedded code in unprivileged script
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
Usage
options, stap, Running SystemTap Scripts
running SystemTap scripts, Running SystemTap Scripts
standard input, running scripts from, Running SystemTap Scripts
stap, Running SystemTap Scripts
stapdev, Running SystemTap Scripts
staprun, Running SystemTap Scripts
stapusr, Running SystemTap Scripts
useful examples of SystemTap scripts, Useful SystemTap Scripts
user and kernel space, determining time spent in
examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
using arrays in simple computations
reading values from arrays
array operations, Reading Values From Arrays
Using SystemTap, Using SystemTap
usrdev2kerndev()
examples of SystemTap scripts, I/O Monitoring (By Device)
V
values, assignment of
array operations, Assigning an Associated Value
variable modified during 'foreach'
parse/semantics error
understanding SystemTap errors, Parse and Semantic Errors
variable notations
SystemTap handler constructs
handlers, Command-Line Arguments
variables
SystemTap handler constructs
handlers, Variables
variables (local)
name, Systemtap Handler/Body
sample usage
$count, Tracking I/O Time For Each File Read or Write
$return, Summarizing Disk Read/Write Traffic, Track Cumulative IO
VFS reads, tallying of
incrementing associated values
array operations, Incrementing Associated Values
vfs.file_operation
Events, Event
virtual file system reads (cumulative), tallying
processing multiple elements in an array
array operations, Processing Multiple Elements in an Array
virtual file system reads (non-cumulative), tallying
clearing arrays/array elements
array operations, Clearing/Deleting Arrays and Array Elements
W
while loops
conditional statements
handlers, Conditional Statements
whole device number (usage as a command-line argument)
examples of SystemTap scripts, I/O Monitoring (By Device)
wildcards in events, Event
writes/reads to a file, monitoring
examples of SystemTap scripts, Monitoring Reads and Writes to a File
PrevAppendixÂ A.Â Revision History
Up
Home
diff --git a/SystemTap_Beginners_Guide/mainsect-disk.html b/SystemTap_Beginners_Guide/mainsect-disk.html index ce20f1a5..a243168d 100644 --- a/SystemTap_Beginners_Guide/mainsect-disk.html +++ b/SystemTap_Beginners_Guide/mainsect-disk.html @@ -1,13 +1,12 @@ -4.2.Â Disk
Prev
Next
4.2.Â Disk
+5.2.Â Disk
Prev
Next
5.2.Â Disk
The following sections showcase scripts that monitor disk and I/O activity. -
4.2.1.Â Summarizing Disk Read/Write Traffic
+
5.2.1.Â Summarizing Disk Read/Write Traffic
This section describes how to identify which processes are performing the heaviest disk reads/writes to the system. -
disktop.stp
+
disktop.stp
-
-#!/usr/bin/env stap +
#!/usr/bin/env stap # # Copyright (C) 2007 Oracle Corp. # @@ -78,10 +77,9 @@ probe end{ delete read_bytes delete write_bytes } -

- disktop.stp outputs the top ten processes responsible for the heaviest reads/writes to disk. ExampleÂ 4.6, âdisktop.stp Sample Outputâ displays a sample output for this script, and includes the following data per listed process: + disktop.stp outputs the top ten processes responsible for the heaviest reads/writes to disk. ExampleÂ 5.6, âdisktop.stp Sample Outputâ displays a sample output for this script, and includes the following data per listed process:
UID â user ID. A user ID of 0 refers to the root user.
@@ -96,12 +94,11 @@ probe end{ T â the type of action performed by the listed process; W refers to write, while R refers to read.
BYTES â the amount of data read to or written from disk. -
- The time and date in the output of disktop.stp is returned by the functions ctime() and gettimeofday_s(). ctime() derives calendar time in terms of seconds passed since the Unix epoch (January 1, 1970). gettimeofday_s() counts the actual number of seconds since Unix epoch, which gives a fairly accurate human-readable timestamp for the output. -
+
+ The time and date in the output of disktop.stp is returned by the functions ctime() and gettimeofday_s(). ctime() derives calendar time in terms of seconds passed since the Unix epoch (January 1, 1970). gettimeofday_s() counts the actual number of seconds since Unix epoch, which gives a fairly accurate human-readable timestamp for the output. +
In this script, the $return is a local variable that stores the actual number of bytes each process reads or writes from the virtual file system. $return can only be used in return probes (e.g. vfs.read.return and vfs.read.return). -
-[...] +
ExampleÂ 5.6.Â disktop.stp Sample Output
[...] Mon Sep 29 03:38:28 2008 , Average: 19Kb/sec, Read: 7Kb, Write: 89Kb UID PID PPID CMD DEVICE T BYTES @@ -113,5 +110,4 @@ Mon Sep 29 03:38:38 2008 , Average: 1Kb/sec, Read: 7Kb, Write: 1Kb UID PID PPID CMD DEVICE T BYTES 0 2758 2757 pam_timestamp_c sda5 R 8064 -0 2885 1 cupsd sda5 W 1678 -
ExampleÂ 4.6.Â disktop.stp Sample Output

PrevChapterÂ 4.Â Useful SystemTap Scripts
Up
Home
Next4.2.2.Â Tracking I/O Time For Each File Read or Wr...
+0 2885 1 cupsd sda5 W 1678

PrevChapterÂ 5.Â Useful SystemTap Scripts
Up
Home
Next5.2.2.Â Tracking I/O Time For Each File Read or Wr...
diff --git a/SystemTap_Beginners_Guide/mainsect-profiling.html b/SystemTap_Beginners_Guide/mainsect-profiling.html index b1dce9fb..34017c68 100644 --- a/SystemTap_Beginners_Guide/mainsect-profiling.html +++ b/SystemTap_Beginners_Guide/mainsect-profiling.html @@ -1,13 +1,12 @@ -4.3.Â Profiling
Prev
Next
4.3.Â Profiling
+5.3.Â Profiling
Prev
Next
5.3.Â Profiling
The following sections showcase scripts that profile kernel activity by monitoring function calls. -
4.3.1.Â Counting Function Calls Made
+
5.3.1.Â Counting Function Calls Made
This section describes how to identify how many times the system called a specific kernel function in a 30-second sample. Depending on your use of wildcards, you can also use this script to target multiple kernel functions. -
functioncallcount.stp
+
functioncallcount.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # The following line command will probe all the functions # in kernel's memory management code: # @@ -25,14 +24,12 @@ probe end { printf("%s %d\n", fn, @count(called[fn])) exit() } -

- functioncallcount.stp takes the targeted kernel function as an argument. The argument supports wildcards, which enables you to target multiple kernel functions up to a certain extent. -
- You can increase the sample time by editing the timer in the second probe (timer.ms()). The output of functioncallcount.stp contains the name of the function called and how many times it was called during the sample time (in alphabetical order). ExampleÂ 4.13, âfunctioncallcount.stp Sample Outputâ contains an excerpt from the output of stap countcalls.stp "*@mm/*.c": -
-[...] + functioncallcount.stp takes the targeted kernel function as an argument. The argument supports wildcards, which enables you to target multiple kernel functions up to a certain extent. +
+ The output of functioncallcount.stp contains the name of the function called and how many times it was called during the sample time (in alphabetical order). ExampleÂ 5.13, âfunctioncallcount.stp Sample Outputâ contains an excerpt from the output of stap functioncallcount.stp "*@mm/*.c": +
ExampleÂ 5.13.Â functioncallcount.stp Sample Output
[...] __vma_link 97 __vma_link_file 66 __vma_link_list 97 @@ -59,5 +56,4 @@ atomic_add_negative 97 atomic_dec_and_test 5153 atomic_inc 470 atomic_inc_and_test 1 -[...] -
ExampleÂ 4.13.Â functioncallcount.stp Sample Output

Prev4.2.7.Â Periodically Print I/O Block Time
Up
Home
Next4.3.2.Â Call Graph Tracing
+[...]

Prev5.2.7.Â Periodically Print I/O Block Time
Up
Home
Next5.3.2.Â Call Graph Tracing
diff --git a/SystemTap_Beginners_Guide/paracallgraph.html b/SystemTap_Beginners_Guide/paracallgraph.html index e61f7344..2032d832 100644 --- a/SystemTap_Beginners_Guide/paracallgraph.html +++ b/SystemTap_Beginners_Guide/paracallgraph.html @@ -1,89 +1,55 @@ -4.3.2.Â Call Graph Tracing
Prev
Next
4.3.2.Â Call Graph Tracing
+5.3.2.Â Call Graph Tracing
Prev
Next
5.3.2.Â Call Graph Tracing
This section describes how to trace incoming and outgoing function calls. -
para-callgraph-simple.stp
+
para-callgraph.stp
-
-function trace(entry_p) { - if(tid() in trace) - printf("%s%s%s\n",thread_indent(entry_p), - (entry_p>0?"->":"<-"), - probefunc()) -} - -global trace -probe kernel.function(@1).call { - if (execname() == "stapio") next # skip our own helper process - trace[tid()] = 1 - trace(1) -} -probe kernel.function(@1).return { - trace(-1) - delete trace[tid()] -} - -probe kernel.function(@2).call { trace(1) } -probe kernel.function(@2).return { trace(-1) } -function trace(entry_p) { - if(tid() in trace) - printf("%s%s%s\n",thread_indent(entry_p), - (entry_p>0?"->":"<-"), - probefunc()) -} - -global trace -probe kernel.function(@1).call { - if (execname() == "stapio") next # skip our own helper process - trace[tid()] = 1 - trace(1) -} -probe kernel.function(@1).return { - trace(-1) - delete trace[tid()] -} - -probe kernel.function(@2).call { trace(1) } -probe kernel.function(@2).return { trace(-1) } +
#! /usr/bin/env stap +function trace(entry_p, extra) { + %( $# > 1 %? if (tid() in trace) %) + printf("%s%s%s %s\n", + thread_indent (entry_p), + (entry_p>0?"->":"<-"), + probefunc (), + extra) +} + + +%( $# > 1 %? +global trace +probe $2.call { + trace[tid()] = 1 +} +probe $2.return { + delete trace[tid()] +} +%) + +probe $1.call { trace(1, $$parms) } +probe $1.return { trace(-1, $$return) }
-
- para-callgraph-simple.stp takes two command-line arguments: -
- A trigger function (@1), which enables or disables tracing on a per-thread basis. Tracing in each thread will continue as long as the trigger function has not exited yet. +
+ para-callgraph.stp takes two command-line arguments: +
+ The function/s whose entry/exit call you'd like to trace ($1).
- The kernel function/s whose entry/exit call you'd like to trace (@2). -
- para-callgraph-simple.stp uses thread_indent(); as such, its output contains the timestamp, process name, and thread ID of @2 (i.e. the probe function you are tracing). For more information about thread_indent(), refer to its entry in SystemTap Functions. + A second optional trigger function ($2), which enables or disables tracing on a per-thread basis. Tracing in each thread will continue as long as the trigger function has not exited yet. +
+ para-callgraph.stp uses thread_indent(); as such, its output contains the timestamp, process name, and thread ID of $1 (i.e. the probe function you are tracing). For more information about thread_indent(), refer to its entry in SystemTap Functions.
- The following example contains an excerpt from the output for stap para-callgraph.stp sys_read '*@fs/*.c': -
-[...] - 0 klogd(1391):->sys_read - 14 klogd(1391): ->fget_light - 22 klogd(1391): <-fget_light - 27 klogd(1391): ->vfs_read - 35 klogd(1391): ->rw_verify_area - 43 klogd(1391): <-rw_verify_area - 49 klogd(1391): ->kmsg_read - 0 sendmail(1696):->sys_read - 17 sendmail(1696): ->fget_light - 26 sendmail(1696): <-fget_light - 34 sendmail(1696): ->vfs_read - 44 sendmail(1696): ->rw_verify_area - 52 sendmail(1696): <-rw_verify_area - 58 sendmail(1696): ->proc_file_read - 70 sendmail(1696): ->loadavg_read_proc - 84 sendmail(1696): ->proc_calc_metrics - 92 sendmail(1696): <-proc_calc_metrics - 95 sendmail(1696): <-loadavg_read_proc - 101 sendmail(1696): <-proc_file_read - 106 sendmail(1696): ->dnotify_parent - 115 sendmail(1696): <-dnotify_parent - 119 sendmail(1696): ->inotify_dentry_parent_queue_event - 127 sendmail(1696): <-inotify_dentry_parent_queue_event - 133 sendmail(1696): ->inotify_inode_queue_event - 141 sendmail(1696): <-inotify_inode_queue_event - 146 sendmail(1696): <-vfs_read - 151 sendmail(1696):<-sys_read -
ExampleÂ 4.14.Â para-callgraph-simple.stp Sample Output

Prev4.3.Â Profiling
Up
Home
Next4.3.3.Â Determining Time Spent in Kernel and User ...
+ The following example contains an excerpt from the output for stap para-callgraph.stp 'kernel.function("*@fs/*.c")' 'kernel.function("sys_read")': +
ExampleÂ 5.14.Â para-callgraph.stp Sample Output
[...] + 267 gnome-terminal(2921): <-do_sync_read return=0xfffffffffffffff5 + 269 gnome-terminal(2921):<-vfs_read return=0xfffffffffffffff5 + 0 gnome-terminal(2921):->fput file=0xffff880111eebbc0 + 2 gnome-terminal(2921):<-fput + 0 gnome-terminal(2921):->fget_light fd=0x3 fput_needed=0xffff88010544df54 + 3 gnome-terminal(2921):<-fget_light return=0xffff8801116ce980 + 0 gnome-terminal(2921):->vfs_read file=0xffff8801116ce980 buf=0xc86504 count=0x1000 pos=0xffff88010544df48 + 4 gnome-terminal(2921): ->rw_verify_area read_write=0x0 file=0xffff8801116ce980 ppos=0xffff88010544df48 count=0x1000 + 7 gnome-terminal(2921): <-rw_verify_area return=0x1000 + 12 gnome-terminal(2921): ->do_sync_read filp=0xffff8801116ce980 buf=0xc86504 len=0x1000 ppos=0xffff88010544df48 + 15 gnome-terminal(2921): <-do_sync_read return=0xfffffffffffffff5 + 18 gnome-terminal(2921):<-vfs_read return=0xfffffffffffffff5 + 0 gnome-terminal(2921):->fput file=0xffff8801116ce980

Prev5.3.Â Profiling
Up
Home
Next5.3.3.Â Determining Time Spent in Kernel and User ...
diff --git a/SystemTap_Beginners_Guide/pr01s02.html b/SystemTap_Beginners_Guide/pr01s02.html index da92bb33..a662c7a7 100644 --- a/SystemTap_Beginners_Guide/pr01s02.html +++ b/SystemTap_Beginners_Guide/pr01s02.html @@ -1,9 +1,9 @@ -2.Â We Need Feedback!
Prev
Next
2.Â We Need Feedback!
- If you find a typographical error in this manual, or if you have thought of a way to make this manual better, we would love to hear from you! Please submit a report in Bugzilla: http://bugzilla.redhat.com/bugzilla/ against the product Red_Hat_Enterprise_Linux 5. +2.Â We Need Feedback!
Prev
Next
2.Â We Need Feedback!
+ If you find a typographical error in this manual, or if you have thought of a way to make this manual better, we would love to hear from you! Please submit a report in Bugzilla: http://bugzilla.redhat.com/bugzilla/ against the product systemtap.
- When submitting a bug report, be sure to mention the manual's identifier: SystemTap_Beginners_Guide + When submitting a report, be sure to include the specific file or URL the report refers to and the manual's identifier: SystemTap_Beginners_Guide
If you have a suggestion for improving the documentation, try to be as specific as possible when describing it. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.
PrevPreface
Up
Home
NextChapterÂ 1.Â Introduction
diff --git a/SystemTap_Beginners_Guide/references.html b/SystemTap_Beginners_Guide/references.html index da962fde..9499b84e 100644 --- a/SystemTap_Beginners_Guide/references.html +++ b/SystemTap_Beginners_Guide/references.html @@ -1,21 +1,23 @@ -ChapterÂ 6.Â References
Prev
Next
ChapterÂ 6.Â References
+ChapterÂ 7.Â References
Prev
Next
ChapterÂ 7.Â References
This chapter enumerates other references for more information about SystemTap. It is advisable that you refer to these sources in the course of writing advanced probes and tapsets. -
SystemTap Wiki
- The SystemTap Wiki is a collection of links and articles related to the deployment, usage, and development of SystemTap. You can find it in http://sourceware.org/systemtap/wiki/HomePage. -
SystemTap Tutorial
+
SystemTap Wiki
+ The SystemTap Wiki is a collection of links and articles related to the deployment, usage, and development of SystemTap. You can find it at http://sourceware.org/systemtap/wiki/HomePage. +
SystemTap Tutorial
Much of the content in this book comes from the SystemTap Tutorial. The SystemTap Tutorial is a more appropriate reference for users with intermediate to advanced knowledge of C++ and kernel development, and can be found at http://sourceware.org/systemtap/tutorial/. -
man stapprobes
- The stapprobes man page enumerates a variety of probe points supported by SystemTap, along with additional aliases defined by the SystemTap tapset library. The bottom of the man page includes a list of other man pages enumerating similar probe points for specific system components, such as stapprobes.scsi, stapprobes.kprocess, stapprobes.signal, etc. -
man stapfuncs
+
man stapprobes
+ The stapprobes man page enumerates a variety of probe points supported by SystemTap, along with additional aliases defined by the SystemTap tapset library. The bottom of the man page includes a list of other man pages enumerating similar probe points for specific system components, such as tapset::scsi, tapset::kprocess, tapset::signal, etc. +
man stapfuncs
The stapfuncs man page enumerates numerous functions supported by the SystemTap tapset library, along with the prescribed syntax for each one. Note, however, that this is not a complete list of all supported functions; there are more undocumented functions available. -
SystemTap Language Reference
+
SystemTap Tapset Reference Manual
+ The SystemTap Tapset Reference Manual at http://sourceware.org/systemtap/tapsets/ describes the individual predefined functions and probe points of the tapsets greater detail. +
SystemTap Language Reference
This document is a comprehensive reference of SystemTap's language constructs and syntax. It is recommended for users with a rudimentary to intermediate knowledge of C++ and other similar programming languages. The SystemTap Language Reference is available to all users at http://sourceware.org/systemtap/langref/ -
Tapset Developers Guide
+
Tapset Developers Guide
Once you have sufficient proficiency in writing SystemTap scripts, you can then try your hand out on writing your own tapsets. The Tapset Developers Guide describes how to add functions to your tapset library. -
Test Suite
- The systemtap-testsuite package allows you to test the entire SystemTap toolchain without having to build from source. In addition, it also contains numerous examples of SystemTap scripts you can study and test; some of these scripts are also documented in ChapterÂ 4, Useful SystemTap Scripts. +
Test Suite
+ The systemtap-testsuite package allows you to test the entire SystemTap toolchain without having to build from source. In addition, it also contains numerous examples of SystemTap scripts you can study and test; some of these scripts are also documented in ChapterÂ 5, Useful SystemTap Scripts.
By default, the example scripts included in systemtap-testsuite are located in /usr/share/systemtap/testsuite/systemtap.examples. -
Prev5.2.Â Run Time Errors and Warnings
Up
Home
NextAppendixÂ A.Â Revision History
+
Prev6.2.Â Run Time Errors and Warnings
Up
Home
NextAppendixÂ A.Â Revision History
diff --git a/SystemTap_Beginners_Guide/runtimeerror.html b/SystemTap_Beginners_Guide/runtimeerror.html index db97603c..c3071850 100644 --- a/SystemTap_Beginners_Guide/runtimeerror.html +++ b/SystemTap_Beginners_Guide/runtimeerror.html @@ -1,21 +1,21 @@ -5.2.Â Run Time Errors and Warnings
Prev
Next
5.2.Â Run Time Errors and Warnings
+6.2.Â Run Time Errors and Warnings
Prev
Next
6.2.Â Run Time Errors and Warnings
Runtime errors and warnings occur when the SystemTap instrumentation has been installed and is collecting data on the system. -
WARNING: Number of errors: N, skipped probes: M
+
WARNING: Number of errors: N, skipped probes: M
Errors and/or skipped probes occurred during this run. Both N and M are the counts of the number of probes that were not executed due to conditions such as too much time required to execute event handlers over an interval of time. -
division by 0
+
division by 0
The script code performed an invalid division. -
aggregate element not found
- An statistics extractor function other than @count was invoked on an aggregate that has not had any values accumulated yet. This is similar to a division by zero. -
aggregation overflow
+
aggregate element not found
+ A statistics extractor function other than @count was invoked on an aggregate that has not had any values accumulated yet. This is similar to a division by zero. +
aggregation overflow
An array containing aggregate values contains too many distinct key pairs at this time. -
MAXNESTING exceeded
+
MAXNESTING exceeded
Too many levels of function call nesting were attempted. The default nesting of function calls allowed is 10. -
MAXACTION exceeded
- The probe handler attempted to execute too many statements in the probe handler. The default number of actions allow in a probe handler is 1000. -
kernel/user string copy fault at ADDR
- The probe handler attempted to copy a string from kernel or user space at an invalid address (ADDR). -
pointer dereference fault
+
MAXACTION exceeded
+ The probe handler attempted to execute too many statements in the probe handler. The default number of actions allowed in a probe handler is 1000. +
kernel/user string copy fault at ADDR
+ The probe handler attempted to copy a string from kernel or user-space at an invalid address (ADDR). +
pointer dereference fault
There was a fault encountered during a pointer dereference operation such as a target variable evaluation. -
PrevChapterÂ 5.Â Understanding SystemTap Errors
Up
Home
NextChapterÂ 6.Â References
+
PrevChapterÂ 6.Â Understanding SystemTap Errors
Up
Home
NextChapterÂ 7.Â References
diff --git a/SystemTap_Beginners_Guide/scriptconstructions.html b/SystemTap_Beginners_Guide/scriptconstructions.html index 90376e2a..5e7d2bdd 100644 --- a/SystemTap_Beginners_Guide/scriptconstructions.html +++ b/SystemTap_Beginners_Guide/scriptconstructions.html @@ -1,13 +1,12 @@ -3.3.Â Basic SystemTap Handler Constructs
Prev
Next
3.3.Â Basic SystemTap Handler Constructs
+3.3.Â Basic SystemTap Handler Constructs
Prev
Next
3.3.Â Basic SystemTap Handler Constructs
SystemTap supports the use of several basic constructs in handlers. The syntax for most of these handler constructs are mostly based on C and awk syntax. This section describes several of the most useful SystemTap handler constructs, which should provide you with enough information to write simple yet useful SystemTap scripts. -
3.3.1.Â Variables
- Variables can be used freely throughout a handler; simply choose a name, assign a value from a function or expression to it, and use it in an expression. SystemTap automatically identifies whether a variable should be typed as a string or integer, based on the type of the values assigned to it. For instance, if you use set the variable foo to gettimeofday_s() (as in foo = gettimeofday_s()), then foo is typed as an number and can be printed in a printf() with the integer format specifier (%d). -
+
3.3.1.Â Variables
+ Variables can be used freely throughout a handler; simply choose a name, assign a value from a function or expression to it, and use it in an expression. SystemTap automatically identifies whether a variable should be typed as a string or integer, based on the type of the values assigned to it. For instance, if you use set the variable foo to gettimeofday_s() (as in foo = gettimeofday_s()), then foo is typed as a number and can be printed in a printf() with the integer format specifier (%d). +
Note, however, that by default variables are only local to the probe they are used in. This means that variables are initialized, used and disposed at each probe handler invocation. To share a variable between probes, declare the variable name using global outside of the probes. Consider the following example: -
-global count_jiffies, count_ms +
ExampleÂ 3.8.Â timer-jiffies.stp
global count_jiffies, count_ms probe timer.jiffies(100) { count_jiffies ++ } probe timer.ms(100) { count_ms ++ } probe timer.ms(12345) @@ -16,13 +15,10 @@ probe timer.ms(12345) printf ("jiffies:ms ratio %d:%d => CONFIG_HZ=%d\n", count_jiffies, count_ms, hz) exit () -} -
ExampleÂ 3.8.Â timer-jiffies.stp

- ExampleÂ 3.8, âtimer-jiffies.stpâ computes the CONFIG_HZ setting of the kernel using timers that count jiffies and milliseconds, then computing accordingly. The global statement allows the script to use the variables count_jiffies and count_ms (set in their own respective probes) to be shared with probe timer.ms(12345). -
Note
- The ++ notation in ExampleÂ 3.8, âtimer-jiffies.stpâ (i.e. count_jiffies ++ and count_ms ++) is used to increment the value of a variable by 1. In the following probe, count_jiffies is incremented by 1 every 100 jiffies: -
-probe timer.jiffies(100) { count_jiffies ++ } -
+}

+ ExampleÂ 3.8, âtimer-jiffies.stpâ computes the CONFIG_HZ setting of the kernel using timers that count jiffies and milliseconds, then computing accordingly. The global statement allows the script to use the variables count_jiffies and count_ms (set in their own respective probes) to be shared with probe timer.ms(12345). +
Note
+ The ++ notation in ExampleÂ 3.8, âtimer-jiffies.stpâ (i.e. count_jiffies ++ and count_ms ++) is used to increment the value of a variable by 1. In the following probe, count_jiffies is incremented by 1 every 100 jiffies: +
probe timer.jiffies(100) { count_jiffies ++ }
In this instance, SystemTap understands that count_jiffies is an integer. Because no initial value was assigned to count_jiffies, its initial value is zero by default. -
Prev3.2.2.Â Systemtap Handler/Body
Up
Home
Next3.3.2.Â Conditional Statements
+
Prev3.2.2.Â Systemtap Handler/Body
Up
Home
Next3.3.2.Â Target Variables
diff --git a/SystemTap_Beginners_Guide/scripts.html b/SystemTap_Beginners_Guide/scripts.html index 77839d1a..10070eb1 100644 --- a/SystemTap_Beginners_Guide/scripts.html +++ b/SystemTap_Beginners_Guide/scripts.html @@ -1,83 +1,73 @@ -3.2.Â SystemTap Scripts
Prev
Next
3.2.Â SystemTap Scripts
+3.2.Â SystemTap Scripts
Prev
Next
3.2.Â SystemTap Scripts
For the most part, SystemTap scripts are the foundation of each SystemTap session. SystemTap scripts instruct SystemTap on what type of information to collect, and what to do once that information is collected. -
- As stated in ChapterÂ 3, Understanding How SystemTap Works, SystemTap scripts are made up of two components: events and handlers. Once a SystemTap session is underway, SystemTap monitors the operating system for the specified events and executes the handlers as they occur. -
Note
+
+ As stated in ChapterÂ 3, Understanding How SystemTap Works, SystemTap scripts are made up of two components: events and handlers. Once a SystemTap session is underway, SystemTap monitors the operating system for the specified events and executes the handlers as they occur. +
Note
An event and its corresponding handler is collectively called a probe. A SystemTap script can have multiple probes.
A probe's handler is commonly referred to as a probe body. -
+
In terms of application development, using events and handlers is similar to instrumenting the code by inserting diagnostic print statements in a program's sequence of commands. These diagnostic print statements allow you to view a history of commands executed once the program is run.
SystemTap scripts allow insertion of the instrumentation code without recompilation of the code and allows more flexibility with regard to handlers. Events serve as the triggers for handlers to run; handlers can be specified to record specified data and print it in a certain manner. -
Format
+
Format
SystemTap scripts use the file extension .stp, and contains probes written in the following format: -
-probe event {statements} -
+
probe event {statements}
SystemTap supports multiple events per probe; multiple events are delimited by a comma (,). If multiple events are specified in a single probe, SystemTap will execute the handler when any of the specified events occur. -
+
Each probe has a corresponding statement block. This statement block is enclosed in braces ({ }) and contains the statements to be executed per event. SystemTap executes these statements in sequence; special separators or terminators are generally not necessary between multiple statements. -
Note
+
Note
Statement blocks in SystemTap scripts follow the same syntax and semantics as the C programming language. A statement block can be nested within another statement block. -
+
Systemtap allows you to write functions to factor out code to be used by a number of probes. Thus, rather than repeatedly writing the same series of statements in multiple probes, you can just place the instructions in a function, as in: -
-function function_name(arguments) {statements} -probe event {function_name(arguments)} -
+
function function_name(arguments) {statements} +probe event {function_name(arguments)}
The statements in function_name are executed when the probe for event executes. The arguments are optional values passed into the function. -
Important
- SectionÂ 3.2, âSystemTap Scriptsâ is designed to introduce readers to the basics of SystemTap scripts. To understand SystemTap scripts better, it is advisable that you refer to ChapterÂ 4, Useful SystemTap Scripts; each section therein provides a detailed explanation of the script, its events, handlers, and expected output. -
3.2.1.Â Event
+
Important
+ SectionÂ 3.2, âSystemTap Scriptsâ is designed to introduce readers to the basics of SystemTap scripts. To understand SystemTap scripts better, it is advisable that you refer to ChapterÂ 5, Useful SystemTap Scripts; each section therein provides a detailed explanation of the script, its events, handlers, and expected output. +
3.2.1.Â Event
SystemTap events can be broadly classified into two types: synchronous and asynchronous. -
Synchronous Events
+
Synchronous Events
A synchronous event occurs when any process executes an instruction at a particular location in kernel code. This gives other events a reference point from which more contextual data may be available. -
+
Examples of synchronous events include: -
syscall.system_call
+
syscall.system_call
The entry to the system call system_call. If the exit from a syscall is desired, appending a .return to the event monitor the exit of the system call instead. For example, to specify the entry and exit of the system call close, use syscall.close and syscall.close.return respectively. -
vfs.file_operation
+
vfs.file_operation
The entry to the file_operation event for Virtual File System (VFS). Similar to syscall event, appending a .return to the event monitors the exit of the file_operation operation. -
kernel.function("function")
+
kernel.function("function")
The entry to the kernel function function. For example, kernel.function("sys_open") refers to the "event" that occurs when the kernel function sys_open is called by any thread in the system. To specify the return of the kernel function sys_open, append the return string to the event statement; i.e. kernel.function("sys_open").return. -
+
When defining probe events, you can use asterisk (*) for wildcards. You can also trace the entry or exit of a function in a kernel source file. Consider the following example: -
-probe kernel.function("*@net/socket.c") { } -probe kernel.function("*@net/socket.c").return { } -
ExampleÂ 3.1.Â wildcards.stp

+
ExampleÂ 3.1.Â wildcards.stp
probe kernel.function("*@net/socket.c") { } +probe kernel.function("*@net/socket.c").return { }

In the previous example, the first probe's event specifies the entry of ALL functions in the kernel source file net/socket.c. The second probe specifies the exit of all those functions. Note that in this example, there are no statements in the handler; as such, no information will be collected or displayed. -
kernel.trace("tracepoint")
+
kernel.trace("tracepoint")
The static probe for tracepoint. Recent kernels (2.6.30 and newer) include instrumentation for specific events in the kernel. These events are statically marked with tracepoints. One example of a tracepoint available in systemtap is kernel.trace("kfree_skb") which indicates each time a network buffer is freed in the kernel. -
module("module").function("function")
+
module("module").function("function")
Allows you to probe functions within modules. For example: -
-probe module("ext3").function("*") { } -probe module("ext3").function("*").return { } -
ExampleÂ 3.2.Â moduleprobe.stp

- The first probe in ExampleÂ 3.2, âmoduleprobe.stpâ points to the entry of all functions for the ext3 module. The second probe points to the exits of all functions for that same module; the use of the .return suffix is similar to kernel.function(). Note that the probes in ExampleÂ 3.2, âmoduleprobe.stpâ do not contain statements in the probe handlers, and as such will not print any useful data (as in ExampleÂ 3.1, âwildcards.stpâ). +
ExampleÂ 3.2.Â moduleprobe.stp
probe module("ext3").function("*") { } +probe module("ext3").function("*").return { }

+ The first probe in ExampleÂ 3.2, âmoduleprobe.stpâ points to the entry of all functions for the ext3 module. The second probe points to the exits of all functions for that same module; the use of the .return suffix is similar to kernel.function(). Note that the probes in ExampleÂ 3.2, âmoduleprobe.stpâ do not contain statements in the probe handlers, and as such will not print any useful data (as in ExampleÂ 3.1, âwildcards.stpâ).
- A system's kernel modules are typically located in /lib/modules/kernel_version, where kernel_version refers to the currently loaded kernel version. Modules use the filename extension .ko. -
Asynchronous Events
+ A system's kernel modules are typically located in /lib/modules/kernel_version, where kernel_version refers to the currently loaded kernel version. Modules use the file name extension .ko. +
Asynchronous Events
Asynchronous events are not tied to a particular instruction or location in code. This family of probe points consists mainly of counters, timers, and similar constructs.
Examples of asynchronous events include: -
begin
+
begin
The startup of a SystemTap session; i.e. as soon as the SystemTap script is run. -
end
+
end
The end of a SystemTap session. -
timer events
+
timer events
An event that specifies a handler to be executed periodically. For example: -
-probe timer.s(4) +
ExampleÂ 3.3.Â timer-s.stp
probe timer.s(4) { printf("hello world\n") -} -
ExampleÂ 3.3.Â timer-s.stp

- ExampleÂ 3.3, âtimer-s.stpâ is an example of a probe that prints hello world every 4 seconds. Note that you can also use the following timer events: +}

+ ExampleÂ 3.3, âtimer-s.stpâ is an example of a probe that prints hello world every 4 seconds. Note that you can also use the following timer events:
timer.ms(milliseconds)
@@ -90,6 +80,6 @@ probe timer.s(4) timer.jiffies(jiffies)
When used in conjunction with other probes that collect information, timer events allows you to print out get periodic updates and see how that information changes over time. -
Important
+
Important
SystemTap supports the use of a large collection of probe events. For more information about supported events, refer to man stapprobes. The SEE ALSO section of man stapprobes also contains links to other man pages that discuss supported events for specific subsystems and components. -
PrevChapterÂ 3.Â Understanding How SystemTap Works
Up
Home
Next3.2.2.Â Systemtap Handler/Body
+
PrevChapterÂ 3.Â Understanding How SystemTap Works
Up
Home
Next3.2.2.Â Systemtap Handler/Body
diff --git a/SystemTap_Beginners_Guide/syscallsbyprocpidsect.html b/SystemTap_Beginners_Guide/syscallsbyprocpidsect.html index fca74b39..d634981d 100644 --- a/SystemTap_Beginners_Guide/syscallsbyprocpidsect.html +++ b/SystemTap_Beginners_Guide/syscallsbyprocpidsect.html @@ -1,11 +1,10 @@ -4.3.6.Â Tracking System Call Volume Per Process
Prev
Next
4.3.6.Â Tracking System Call Volume Per Process
- This section illustrates how to determine which processes are performing the highest volume of system calls. In previous sections, we've described how to monitor the top system calls used by the system over time (SectionÂ 4.3.5, âTracking Most Frequently Used System Callsâ). We've also described how to identify which applications use a specific set of "polling suspect" system calls the most (SectionÂ 4.3.4, âMonitoring Polling Applicationsâ). Monitoring the volume of system calls made by each process provides more data in investigating your system for polling processes and other resource hogs. -
syscalls_by_proc.stp
+5.3.6.Â Tracking System Call Volume Per Process
Prev
Next
5.3.6.Â Tracking System Call Volume Per Process
+ This section illustrates how to determine which processes are performing the highest volume of system calls. In previous sections, we've described how to monitor the top system calls used by the system over time (SectionÂ 5.3.5, âTracking Most Frequently Used System Callsâ). We've also described how to identify which applications use a specific set of "polling suspect" system calls the most (SectionÂ 5.3.4, âMonitoring Polling Applicationsâ). Monitoring the volume of system calls made by each process provides more data in investigating your system for polling processes and other resource hogs. +
syscalls_by_proc.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # Copyright (C) 2006 IBM Corp. # @@ -33,12 +32,10 @@ probe end { foreach (proc in syscalls-) printf("%-10d %-s\n", syscalls[proc], proc) } -

- syscalls_by_proc.stp lists the top 20 processes performing the highest number of system calls. It also lists how many system calls each process performed during the time period. Refer to ExampleÂ 4.18, âtopsys.stp Sample Outputâ for a sample output. -
-Collecting data... Type Ctrl-C to exit and display results + syscalls_by_proc.stp lists the top 20 processes performing the highest number of system calls. It also lists how many system calls each process performed during the time period. Refer to ExampleÂ 5.18, âtopsys.stp Sample Outputâ for a sample output. +
ExampleÂ 5.18.Â topsys.stp Sample Output
Collecting data... Type Ctrl-C to exit and display results #SysCalls Process Name 1577 multiload-apple 692 synergyc @@ -53,13 +50,11 @@ Collecting data... Type Ctrl-C to exit and display results 84 scim-bridge 78 gnome-screensav 66 scim-launcher -[...] -
ExampleÂ 4.18.Â topsys.stp Sample Output

+[...]

If you prefer the output to display the process IDs instead of the process names, use the following script instead. -
syscalls_by_pid.stp
+
syscalls_by_pid.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # Copyright (C) 2006 IBM Corp. # @@ -87,13 +82,10 @@ probe end { foreach (pid in syscalls-) printf("%-10d %-d\n", syscalls[pid], pid) } -

As indicated in the output, you need to manually exit the script in order to display the results. You can add a timed expiration to either script by simply adding a timer.s() probe; for example, to instruct the script to expire after 5 seconds, add the following probe to the script: -
-probe timer.s(5) +
probe timer.s(5) { exit() -} -
Prev4.3.5.Â Tracking Most Frequently Used System Calls
Up
Home
Next4.4.Â Identifying Contended User-Space Locks
+}
Prev5.3.5.Â Tracking Most Frequently Used System Calls
Up
Home
Next5.4.Â Identifying Contended User-Space Locks
diff --git a/SystemTap_Beginners_Guide/systemtapscript-handler.html b/SystemTap_Beginners_Guide/systemtapscript-handler.html index 6758783c..e49d9c0f 100644 --- a/SystemTap_Beginners_Guide/systemtapscript-handler.html +++ b/SystemTap_Beginners_Guide/systemtapscript-handler.html @@ -1,62 +1,55 @@ -3.2.2.Â Systemtap Handler/Body
Prev
Next
3.2.2.Â Systemtap Handler/Body
+3.2.2.Â Systemtap Handler/Body
Prev
Next
3.2.2.Â Systemtap Handler/Body
Consider the following sample script: -
-probe begin +
ExampleÂ 3.4.Â helloworld.stp
probe begin { printf ("hello world\n") exit () -} -
ExampleÂ 3.4.Â helloworld.stp

- In ExampleÂ 3.4, âhelloworld.stpâ, the event begin (i.e. the start of the session) triggers the handler enclosed in { }, which simply prints hello world followed by a new-line, then exits. -
Note
+}

+ In ExampleÂ 3.4, âhelloworld.stpâ, the event begin (i.e. the start of the session) triggers the handler enclosed in { }, which simply prints hello world followed by a new-line, then exits. +
Note
SystemTap scripts continue to run until the exit() function executes. If the users wants to stop the execution of the script, it can interrupted manually with Ctrl+C. -
printf ( ) Statements
+
printf ( ) Statements
The printf () statement is one of the simplest functions for printing data. printf () can also be used to display data using a wide variety of SystemTap functions in the following format: -
- printf ("format string\n", arguments) -
- The format string specifies how arguments should be printed. The format string of ExampleÂ 3.4, âhelloworld.stpâ simply instructs SystemTap to print hello world, and contains no format specifiers. -
+
printf ("format string\n", arguments) +
+ The format string specifies how arguments should be printed. The format string of ExampleÂ 3.4, âhelloworld.stpâ simply instructs SystemTap to print hello world, and contains no format specifiers. +
You can use the format specifiers %s (for strings) and %d (for numbers) in format strings, depending on your list of arguments. Format strings can have multiple format specifiers, each matching a corresponding argument; multiple arguments are delimited by a comma (,). -
Note
+
Note
Semantically, the SystemTap printf function is very similar to its C language counterpart. The aforementioned syntax and format for SystemTap's printf function is identical to that of the C-style printf. -
+
To illustrate this, consider the following probe example: -
-probe syscall.open +
ExampleÂ 3.5.Â variables-in-printf-statements.stp
probe syscall.open { printf ("%s(%d) open\n", execname(), pid()) -} -
ExampleÂ 3.5.Â variables-in-printf-statements.stp

- ExampleÂ 3.5, âvariables-in-printf-statements.stpâ instructs SystemTap to probe all entries to the system call open; for each event, it prints the current execname() (a string with the executable name) and pid() (the current process ID number), followed by the word open. A snippet of this probe's output would look like: -
-vmware-guestd(2206) open +}

+ ExampleÂ 3.5, âvariables-in-printf-statements.stpâ instructs SystemTap to probe all entries to the system call open; for each event, it prints the current execname() (a string with the executable name) and pid() (the current process ID number), followed by the word open. A snippet of this probe's output would look like: +
vmware-guestd(2206) open hald(2360) open hald(2360) open hald(2360) open df(3433) open df(3433) open df(3433) open -hald(2360) open -
SystemTap Functions
- SystemTap supports a wide variety of functions that can be used as printf () arguments. ExampleÂ 3.5, âvariables-in-printf-statements.stpâ uses the SystemTap functions execname() (name of the process that called a kernel function/performed a system call) and pid() (current process ID). +hald(2360) open
SystemTap Functions
+ SystemTap supports a wide variety of functions that can be used as printf () arguments. ExampleÂ 3.5, âvariables-in-printf-statements.stpâ uses the SystemTap functions execname() (name of the process that called a kernel function/performed a system call) and pid() (current process ID).
The following is a list of commonly-used SystemTap functions: -
tid()
+
tid()
The ID of the current thread. -
uid()
+
uid()
The ID of the current user. -
cpu()
+
cpu()
The current CPU number. -
gettimeofday_s()
+
gettimeofday_s()
The number of seconds since UNIX epoch (January 1, 1970). -
ctime()
+
ctime()
Convert number of seconds since UNIX epoch to date. -
pp()
+
pp()
A string describing the probe point currently being handled. -
thread_indent()
+
thread_indent()
This particular function is quite useful, providing you with a way to better organize your print results. The function takes one argument, an indentation delta, which indicates how many spaces to add or remove from a thread's "indentation counter". It then returns a string with some generic trace data along with an appropriate number of indentation spaces.
The generic data included in the returned string includes a timestamp (number of microseconds since the first call to thread_indent() by the thread), a process name, and the thread ID. This allows you to identify what functions were called, who called them, and the duration of each function call. @@ -64,19 +57,16 @@ hald(2360) open If call entries and exits immediately precede each other, it is easy to match them. However, in most cases, after a first function call entry is made several other call entries and exits may be made before the first call exits. The indentation counter helps you match an entry with its corresponding exit by indenting the next function call if it is not the exit of the previous one.
Consider the following example on the use of thread_indent(): -
-probe kernel.function("*@net/socket.c") +
ExampleÂ 3.6.Â thread_indent.stp
probe kernel.function("*@net/socket.c") { printf ("%s -> %s\n", thread_indent(1), probefunc()) } -probe kernel.function("*@net/socket.c").return +probe kernel.function("*@net/socket.c").return { printf ("%s <- %s\n", thread_indent(-1), probefunc()) -} -
ExampleÂ 3.6.Â thread_indent.stp

- ExampleÂ 3.6, âthread_indent.stpâ prints out the thread_indent() and probe functions at each event in the following format: -
-0 ftp(7223): -> sys_socketcall +}

+ ExampleÂ 3.6, âthread_indent.stpâ prints out the thread_indent() and probe functions at each event in the following format: +
0 ftp(7223): -> sys_socketcall 1159 ftp(7223): -> sys_socket 2173 ftp(7223): -> __sock_create 2286 ftp(7223): -> sock_alloc_inode @@ -91,28 +81,25 @@ probe kernel.function("*@net/socket.c").return 4699 ftp(7223): <- sock_map_file 4715 ftp(7223): <- sock_map_fd 4732 ftp(7223): <- sys_socket -4775 ftp(7223): <- sys_socketcall -
+4775 ftp(7223): <- sys_socketcall
This sample output contains the following information:
- The time (in microseconds) since the initial thread_ident() call for the thread (included in the string from thread_ident()). + The time (in microseconds) since the initial thread_indent() call for the thread (included in the string from thread_indent()).
- The process name (and its corresponding ID) that made the function call (included in the string from thread_ident()). + The process name (and its corresponding ID) that made the function call (included in the string from thread_indent()).
An arrow signifying whether the call was an entry (<-) or an exit (->); the indentations help you match specific function call entries with their corresponding exits.
The name of the function called by the process. -
name
+
name
Identifies the name of a specific system call. This variable can only be used in probes that use the event syscall.system_call. -
target()
+
target()
Used in conjunction with stap script -x process ID or stap script -c command. If you want to specify a script to take an argument of a process ID or command, use target() as the variable in the script to refer to it. For example: -
-probe syscall.* { +
ExampleÂ 3.7.Â targetexample.stp
probe syscall.* { if (pid() == target()) printf("%s/n", name) -} -
ExampleÂ 3.7.Â targetexample.stp

- When ExampleÂ 3.7, âtargetexample.stpâ is run with the argument -x process ID, it watches all system calls (as specified by the event syscall.*) and prints out the name of all system calls made by the specified process. +}

+ When ExampleÂ 3.7, âtargetexample.stpâ is run with the argument -x process ID, it watches all system calls (as specified by the event syscall.*) and prints out the name of all system calls made by the specified process.
This has the same effect as specifying if (pid() == process ID) each time you wish to target a specific process. However, using target() makes it easier for you to re-use the script, giving you the ability to simply pass a process ID as an argument each time you wish to run the script (e.g. stap targetexample.stp -x process ID).
diff --git a/SystemTap_Beginners_Guide/targetavailable.html b/SystemTap_Beginners_Guide/targetavailable.html new file mode 100644 index 00000000..efa2bcf5 --- /dev/null +++ b/SystemTap_Beginners_Guide/targetavailable.html @@ -0,0 +1,15 @@ + + +3.3.2.3.Â Checking Target Variable Availablility
Prev
Next
3.3.2.3.Â Checking Target Variable Availablility
+ As code evolves the target variables available may change. The @defined makes it easier to handle those variations in the available target variables. The @defined provides a test to see if a particular target variable is available. The result of this test can be used to select the appropriate expression. +
+ The ExampleÂ 3.10, âTesting target variable available Exampleâ from the memory.stp tapset provides an probe event alias. Some version of the kernel functions being probed have an argument $flags. When available, the $flags argument is used to generate the local variable write_access. The versions of the probed functions that do not have the $flags argument have a $write argument and that is used instead for the local variable write_access. +
ExampleÂ 3.10.Â Testing target variable available Example
probe vm.pagefault = kernel.function("__handle_mm_fault@mm/memory.c") ?, + kernel.function("handle_mm_fault@mm/memory.c") ? +{ + name = "pagefault" + write_access = (@defined($flags) + ? $flags & FAULT_FLAG_WRITE : $write_access) + address = $address +} +

Prev3.3.2.2.Â Typecasting
Up
Home
Next3.3.3.Â Conditional Statements
diff --git a/SystemTap_Beginners_Guide/targetvariables.html b/SystemTap_Beginners_Guide/targetvariables.html new file mode 100644 index 00000000..55222128 --- /dev/null +++ b/SystemTap_Beginners_Guide/targetvariables.html @@ -0,0 +1,47 @@ + + +3.3.2.Â Target Variables
Prev
Next
3.3.2.Â Target Variables
+ The probe events that map to actual locations in the code (for example kernel.function("function") and kernel.statement("statement")) allow the use of target variables to obtain the value of variables visible at that location in the code. You can use the -L option to list the target variable available at a probe point. If the debug information is installed for the running kernel, you can run the following command to find out what target variables are available for the vfs_read function: +
stap -L 'kernel.function("vfs_read")'
+ This will yield something similar to the following: +
kernel.function("vfs_read@fs/read_write.c:277") $file:struct file* $buf:char* $count:size_t $pos:loff_t*
+ Each target variable is proceeded by a â$â and the type of the target variable follows the â:â. The kernel's vfs_read function has $file (pointer to structure describing the file), $buf (pointer to the user-space memory to store the read data), $count (number of bytes to read), and $pos (position to start reading from in the file) target variables at the entry to the function. +
+ SystemTap tracks the typing information of the target variable and can examine the fields of a structure with the -> operator. The -> operator can be chained to look at data structures contained within data structures and follow pointers to other data structures. The -> operator will obtain the value in the field of the structure. The -> operator is used regardless whether accessing a field in a substructure or accessing another structure through a pointer. +
+ For pointers to base types such as integers and strings there are a number of functions listed below to access kernel-space data. The first argument for each functions is the pointer to the data item. Similar functions are described in SectionÂ 4.2, âAccessing User-Space Target Variablesâ for accessing target variables in user-space code. +
kernel_char(address)
+ Obtain the character at address from kernel memory. +
kernel_short(address)
+ Obtain the short at address from kernel memory. +
kernel_int(address)
+ Obtain the int at address from kernel memory. +
kernel_long(address)
+ Obtain the long at address from kernel memory +
kernel_string(address)
+ Obtain the string at address from kernel memory. +
kernel_string_n(address, n)
+ Obtain the string at address from the kernel memory and limits the string to n bytes. +
3.3.2.1.Â Pretty Printing Target Variables
+ SystemTap scripts are often used to observe what is happening within the code. In many cases just printing the values of the various context variables is sufficient. SystemTap makes a number operations available that can generate printable strings for target variables: +
$$vars
+ Expands to a character string that is equivalent to sprintf("parm1=%x ... parmN=%x var1=%x ... varN=%x", parm1, ..., parmN, var1, ..., varN) for each variable in scope at the probe point. Some values may be printed as â=?â if their run-time location cannot be found. +
$$locals
+ Expands to a subset of $$vars containing only the local variables. +
$$parms
+ Expands to a subset of $$vars containing only the function parameters. +
$$return
+ Is available in return probes only. It expands to a string that is equivalent to sprintf("return=%x", $return) if the probed function has a return value, or else an empty string. +
+ Below is a command-line script that prints the values of the parameters passed into the function vfs_read: +
stap -e 'probe kernel.function("vfs_read") {printf("%s\n", $$parms); exit(); }'
+ There are four parameters passed into vfs_read: file, buf, count, and pos. The $$parms generates a string for the parameters passed into the function. In this case all but the count parameter are pointers. The following is an example of the output from the previous command-line script: +
file=0xffff8800b40d4c80 buf=0x7fff634403e0 count=0x2004 pos=0xffff8800af96df48
+ Having the address a pointer points to may not be useful. You might be more interested in the fields of the data structure the pointer points to. You can use the the â$â suffix to pretty print the data structure. The following command-line example uses the pretty printing suffix to print more details about the data structures passed into the function vfs_read: +
stap -e 'probe kernel.function("vfs_read") {printf("%s\n", $$parms$); exit(); }'
+ The previous command line will generate something similar to the following with the fields of the data structure included in the output: +
file={.f_u={...}, .f_path={...}, .f_op=0xffffffffa06e1d80, .f_lock={...}, .f_count={...}, .f_flags=34818, .f_mode=31, .f_pos=0, .f_owner={...}, .f_cred=0xffff88013148fc80, .f_ra={...}, .f_version=0, .f_security=0xffff8800b8dce560, .private_data=0x0, .f_ep_links={...}, .f_mapping=0xffff880037f8fdf8} buf="" count=8196 pos=-131938753921208
+ With the â$â suffix fields that are composed of data structures are not expanded. The â$$â suffix will print the values contained within the nested data structures. Below is an example using the â$$â suffix: +
stap -e 'probe kernel.function("vfs_read") {printf("%s\n", $$parms$$); exit(); }'
+ The â$$â suffix, like all strings, is limited to the maximum string size. Below is a represenative output from the previous command-line script, which is truncated because of the string size limit: +
file={.f_u={.fu_list={.next=0xffff8801336ca0e8, .prev=0xffff88012ded0840}, .fu_rcuhead={.next=0xffff8801336ca0e8, .func=0xffff88012ded0840}}, .f_path={.mnt=0xffff880132fc97c0, .dentry=0xffff88001a889cc0}, .f_op=0xffffffffa06f64c0, .f_lock={.raw_lock={.slock=196611}}, .f_count={.counter=2}, .f_flags=34818, .f_mode=31, .f_pos=0, .f_owner={.lock={.raw_lock={.lock=16777216}}, .pid=0x0, .pid_type=0, .uid=0, .euid=0, .signum=0}, .f_cred=0xffff880130129a80, .f_ra={.start=0, .size=0, .async_size=0, .ra_pages=32, .
Prev3.3.Â Basic SystemTap Handler Constructs
Up
Home
Next3.3.2.2.Â Typecasting
diff --git a/SystemTap_Beginners_Guide/threadtimessect.html b/SystemTap_Beginners_Guide/threadtimessect.html index 3c26ad0d..6561d509 100644 --- a/SystemTap_Beginners_Guide/threadtimessect.html +++ b/SystemTap_Beginners_Guide/threadtimessect.html @@ -1,20 +1,21 @@ -4.3.3.Â Determining Time Spent in Kernel and User Space
Prev
Next
4.3.3.Â Determining Time Spent in Kernel and User Space
+5.3.3.Â Determining Time Spent in Kernel and User Space
Prev
Next
5.3.3.Â Determining Time Spent in Kernel and User Space
This section illustrates how to determine the amount of time any given thread is spending in either kernel or user-space. -
thread-times.stp
+
thread-times.stp
-
-#! /usr/bin/stap +
#! /usr/bin/env stap -probe timer.profile { - tid=tid() +probe perf.sw.cpu_clock!, timer.profile { + // NB: To avoid contention on SMP machines, no global scalars/arrays used, + // only contention-free statistics aggregates. + tid=tid(); e=execname() if (!user_mode()) - kticks[tid] <<< 1 + kticks[e,tid] <<< 1 else - uticks[tid] <<< 1 + uticks[e,tid] <<< 1 ticks <<< 1 - tids[tid] <<< 1 + tids[e,tid] <<< 1 } global uticks, kticks, ticks @@ -23,13 +24,13 @@ global tids probe timer.s(5), end { allticks = @count(ticks) - printf ("%5s %7s %7s (of %d ticks)\n", - "tid", "%user", "%kernel", allticks) - foreach (tid in tids- limit 20) { - uscaled = @count(uticks[tid])*10000/allticks - kscaled = @count(kticks[tid])*10000/allticks - printf ("%5d %3d.%02d%% %3d.%02d%%\n", - tid, uscaled/100, uscaled%100, kscaled/100, kscaled%100) + printf ("%16s %5s %7s %7s (of %d ticks)\n", + "comm", "tid", "%user", "%kernel", allticks) + foreach ([e,tid] in tids- limit 20) { + uscaled = @count(uticks[e,tid])*10000/allticks + kscaled = @count(kticks[e,tid])*10000/allticks + printf ("%16s %5d %3d.%02d%% %3d.%02d%%\n", + e, tid, uscaled/100, uscaled%100, kscaled/100, kscaled%100) } printf("\n") @@ -38,14 +39,12 @@ probe timer.s(5), end { delete ticks delete tids } -
-
- thread-times.stp lists the top 20 processes currently taking up CPU time within a 5-second sample, along with the total number of CPU ticks made during the sample. The output of this script also notes the percentage of CPU time each process used, as well as whether that time was spent in kernel space or user space. +
+ thread-times.stp lists the top 20 processes currently taking up CPU time within a 5-second sample, along with the total number of CPU ticks made during the sample. The output of this script also notes the percentage of CPU time each process used, as well as whether that time was spent in kernel space or user space.
- ExampleÂ 4.15, âthread-times.stp Sample Outputâ contains a 5-second sample of the output for thread-times.stp: -
- tid %user %kernel (of 20002 ticks) + ExampleÂ 5.15, âthread-times.stp Sample Outputâ contains a 5-second sample of the output for thread-times.stp: +
ExampleÂ 5.15.Â thread-times.stp Sample Output
tid %user %kernel (of 20002 ticks) 0 0.00% 87.88% 32169 5.24% 0.03% 9815 3.33% 0.36% @@ -65,5 +64,4 @@ probe timer.s(5), end { 3883 0.00% 0.00% 3775 0.00% 0.00% 3943 0.00% 0.00% - 3873 0.00% 0.00% -
ExampleÂ 4.15.Â thread-times.stp Sample Output

Prev4.3.2.Â Call Graph Tracing
Up
Home
Next4.3.4.Â Monitoring Polling Applications
+ 3873 0.00% 0.00%

Prev5.3.2.Â Call Graph Tracing
Up
Home
Next5.3.4.Â Monitoring Polling Applications
diff --git a/SystemTap_Beginners_Guide/timeoutssect.html b/SystemTap_Beginners_Guide/timeoutssect.html index 1f55f503..4639a702 100644 --- a/SystemTap_Beginners_Guide/timeoutssect.html +++ b/SystemTap_Beginners_Guide/timeoutssect.html @@ -1,11 +1,10 @@ -4.3.4.Â Monitoring Polling Applications
Prev
Next
4.3.4.Â Monitoring Polling Applications
- This section how to identify and monitor which applications are polling. Doing so allows you to track unnecessary or excessive polling, which can help you pinpoint areas for improvement in terms of CPU usage and power savings. -
timeout.stp
+5.3.4.Â Monitoring Polling Applications
Prev
Next
5.3.4.Â Monitoring Polling Applications
+ This section describes how to identify and monitor which applications are polling. Doing so allows you to track unnecessary or excessive polling, which can help you pinpoint areas for improvement in terms of CPU usage and power savings. +
timeout.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # Copyright (C) 2009 Red Hat, Inc. # Written by Ulrich Drepper <drepper@redhat.com> # Modified by William Cohen <wcohen@redhat.com> @@ -107,10 +106,9 @@ probe timer.s(1) { signal_timeout[p], process[p]) } } -

- timeout.stp tracks how many times each application used the following system calls over time: + timeout.stp tracks how many times each of the following system calls completed due to time expiring rather than due to an actual event occurring:
poll
@@ -125,10 +123,7 @@ probe timer.s(1) { nanosleep
signal -
- In some applications, these system calls are used excessively. As such, they are normally identified as "likely culprits" for polling applications. Note, however, that an application may be using a different system call to poll excessively; sometimes, it is useful to find out the top system calls used by the system (refer to SectionÂ 4.3.5, âTracking Most Frequently Used System Callsâ for instructions). Doing so can help you identify any additional suspects, which you can add to timeout.stp for tracking. -
- uid | poll select epoll itimer futex nanosle signal| process +
ExampleÂ 5.16.Â timeout.stp Sample Output
uid | poll select epoll itimer futex nanosle signal| process 28937 | 148793 0 0 4727 37288 0 0| firefox 22945 | 0 56949 0 1 0 0 0| scim-bridge 0 | 0 0 0 36414 0 0 0| swapper @@ -148,7 +143,6 @@ probe timer.s(1) { 4248 | 1591 0 0 0 0 0 0| nm-applet 3165 | 0 1441 0 0 0 0 0| xterm 29548 | 0 1440 0 0 0 0 0| httpd - 1862 | 0 0 0 0 0 1438 0| iscsid -
ExampleÂ 4.16.Â timeout.stp Sample Output

- You can increase the sample time by editing the timer in the second probe (timer.s()). The output of functioncallcount.stp contains the name and UID of the top 20 polling applications, along with how many times each application performed each polling system call (over time). ExampleÂ 4.16, âtimeout.stp Sample Outputâ contains an excerpt of the script: -
Prev4.3.3.Â Determining Time Spent in Kernel and User ...
Up
Home
Next4.3.5.Â Tracking Most Frequently Used System Calls
+ 1862 | 0 0 0 0 0 1438 0| iscsid

+ You can increase the sample time by editing the second probe (timer.s(1)). The output of timeout.stp contains the name and UID of the top 20 polling applications, along with how many times each application performed each polling system call (over time). ExampleÂ 5.16, âtimeout.stp Sample Outputâ contains an excerpt of the script. In this particular example firefox is doing an excessive amount of polling due to a plugin module. +
Prev5.3.3.Â Determining Time Spent in Kernel and User ...
Up
Home
Next5.3.5.Â Tracking Most Frequently Used System Calls
diff --git a/SystemTap_Beginners_Guide/topsyssect.html b/SystemTap_Beginners_Guide/topsyssect.html index 7496760a..b6356fe8 100644 --- a/SystemTap_Beginners_Guide/topsyssect.html +++ b/SystemTap_Beginners_Guide/topsyssect.html @@ -1,27 +1,10 @@ -4.3.5.Â Tracking Most Frequently Used System Calls
Prev
Next
4.3.5.Â Tracking Most Frequently Used System Calls
- timeout.stp from SectionÂ 4.3.4, âMonitoring Polling Applicationsâ helps you identify which applications are polling by pointing out which ones used the following system calls most frequently: -
- poll -
- select -
- epoll -
- itimer -
- futex -
- nanosleep -
- signal -
- However, in some systems, a different system call might be responsible for excessive polling. If you suspect that a polling application might is using a different system call to poll, you need to identify first the top system calls used by the system. To do this, use topsys.stp. -
topsys.stp
+5.3.5.Â Tracking Most Frequently Used System Calls
Prev
Next
5.3.5.Â Tracking Most Frequently Used System Calls
+ timeout.stp from SectionÂ 5.3.4, âMonitoring Polling Applicationsâ helps you identify which applications are polling by examining a small subset of system calls ( poll, select, epoll, itimer, futex, nanosleep, and signal). However, in some systems, an excessive number of system calls outside that small subset might be responsible for time spent in the kernel. If you suspect that an application is using system calls excessively, you need to identify the most frequently used system calls on the system. To do this, use topsys.stp. +
topsys.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # # This script continuously lists the top 20 systemcalls in the interval # 5 seconds @@ -45,12 +28,10 @@ probe timer.s(5) { print_systop () printf("--------------------------------------------------------------\n") } -

- topsys.stp lists the top 20 system calls used by the system per 5-second interval. It also lists how many times each system call was used during that period. Refer to ExampleÂ 4.17, âtopsys.stp Sample Outputâ for a sample output. -
--------------------------------------------------------------- + topsys.stp lists the top 20 system calls used by the system per 5-second interval. It also lists how many times each system call was used during that period. Refer to ExampleÂ 5.17, âtopsys.stp Sample Outputâ for a sample output. +
ExampleÂ 5.17.Â topsys.stp Sample Output
-------------------------------------------------------------- SYSCALL COUNT gettimeofday 1857 read 1821 @@ -72,5 +53,4 @@ probe timer.s(5) { time 81 sigreturn 72 fstat 66 --------------------------------------------------------------- -
ExampleÂ 4.17.Â topsys.stp Sample Output

Prev4.3.4.Â Monitoring Polling Applications
Up
Home
Next4.3.6.Â Tracking System Call Volume Per Process
+--------------------------------------------------------------

Prev5.3.4.Â Monitoring Polling Applications
Up
Home
Next5.3.6.Â Tracking System Call Volume Per Process
diff --git a/SystemTap_Beginners_Guide/traceio2sect.html b/SystemTap_Beginners_Guide/traceio2sect.html index b8629c14..c4a76e5e 100644 --- a/SystemTap_Beginners_Guide/traceio2sect.html +++ b/SystemTap_Beginners_Guide/traceio2sect.html @@ -1,11 +1,10 @@ -4.2.4.Â I/O Monitoring (By Device)
Prev
Next
4.2.4.Â I/O Monitoring (By Device)
+5.2.4.Â I/O Monitoring (By Device)
Prev
Next
5.2.4.Â I/O Monitoring (By Device)
This section describes how to monitor I/O activity on a specific device. -
traceio2.stp
+
traceio2.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap global device_of_interest @@ -24,22 +23,19 @@ probe vfs.write, vfs.read printf ("%s(%d) %s 0x%x\n", execname(), pid(), probefunc(), dev) } -
-
- traceio2.stp takes 1 argument: the whole device number. To get this number, use stat -c "0x%D" directory, where directory is located in the device you wish to monitor. -
+
+ traceio2.stp takes 1 argument: the whole device number. To get this number, use stat -c "0x%D" directory, where directory is located in the device you wish to monitor. +
The usrdev2kerndev() function converts the whole device number into the format understood by the kernel. The output produced by usrdev2kerndev() is used in conjunction with the MKDEV(), MINOR(), and MAJOR() functions to determine the major and minor numbers of a specific device.
- The output of traceio2.stp includes the name and ID of any process performing a read/write, the function it is performing (i.e. vfs_read or vfs_write), and the kernel device number. + The output of traceio2.stp includes the name and ID of any process performing a read/write, the function it is performing (i.e. vfs_read or vfs_write), and the kernel device number.
The following example is an excerpt from the full output of stap traceio2.stp 0x805, where 0x805 is the whole device number of /home. /home resides in /dev/sda5, which is the device we wish to monitor. -
-[...] +
ExampleÂ 5.9.Â traceio2.stp Sample Output
[...] synergyc(3722) vfs_read 0x800005 synergyc(3722) vfs_read 0x800005 cupsd(2889) vfs_write 0x800005 cupsd(2889) vfs_write 0x800005 cupsd(2889) vfs_write 0x800005 -[...] -
ExampleÂ 4.9.Â traceio2.stp Sample Output

Prev4.2.3.Â Track Cumulative IO
Up
Home
Next4.2.5.Â Monitoring Reads and Writes to a File
+[...]

Prev5.2.3.Â Track Cumulative IO
Up
Home
Next5.2.5.Â Monitoring Reads and Writes to a File
diff --git a/SystemTap_Beginners_Guide/traceiosect.html b/SystemTap_Beginners_Guide/traceiosect.html index ec098f20..81c89432 100644 --- a/SystemTap_Beginners_Guide/traceiosect.html +++ b/SystemTap_Beginners_Guide/traceiosect.html @@ -1,11 +1,10 @@ -4.2.3.Â Track Cumulative IO
Prev
Next
4.2.3.Â Track Cumulative IO
+5.2.3.Â Track Cumulative IO
Prev
Next
5.2.3.Â Track Cumulative IO
This section describes how to track the cumulative amount of I/O to the system. -
traceio.stp
+
traceio.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap # traceio.stp # Copyright (C) 2007 Red Hat, Inc., Eugene Teo <eteo@redhat.com> # Copyright (C) 2009 Kai Meyer <kai@unixlords.com> @@ -50,14 +49,12 @@ probe timer.s(1) { # Note we don't zero out reads, writes and total_io, # so the values are cumulative since the script started. } -

- traceio.stp prints the top ten executables generating I/O traffic over time. In addition, it also tracks the cumulative amount of I/O reads and writes done by those ten executables. This information is tracked and printed out in 1-second intervals, and in descending order. -
- Note that traceio.stp also uses the local variable $return, which is also used by disktop.stp from SectionÂ 4.2.1, âSummarizing Disk Read/Write Trafficâ. -
-[...] + traceio.stp prints the top ten executables generating I/O traffic over time. In addition, it also tracks the cumulative amount of I/O reads and writes done by those ten executables. This information is tracked and printed out in 1-second intervals, and in descending order. +
+ Note that traceio.stp also uses the local variable $return, which is also used by disktop.stp from SectionÂ 5.2.1, âSummarizing Disk Read/Write Trafficâ. +
ExampleÂ 5.8.Â traceio.stp Sample Output
[...] Xorg r: 583401 KiB w: 0 KiB floaters r: 96 KiB w: 7130 KiB multiload-apple r: 538 KiB w: 537 KiB @@ -78,5 +75,4 @@ pam_timestamp_c r: 138 KiB w: 0 KiB snmpd r: 46 KiB w: 0 KiB pcscd r: 28 KiB w: 0 KiB irqbalance r: 27 KiB w: 4 KiB - cupsd r: 4 KiB w: 18 KiB -
ExampleÂ 4.8.Â traceio.stp Sample Output

Prev4.2.2.Â Tracking I/O Time For Each File Read or Wr...
Up
Home
Next4.2.4.Â I/O Monitoring (By Device)
+ cupsd r: 4 KiB w: 18 KiB

Prev5.2.2.Â Tracking I/O Time For Each File Read or Wr...
Up
Home
Next5.2.4.Â I/O Monitoring (By Device)
diff --git a/SystemTap_Beginners_Guide/typecasting.html b/SystemTap_Beginners_Guide/typecasting.html new file mode 100644 index 00000000..5aa3a418 --- /dev/null +++ b/SystemTap_Beginners_Guide/typecasting.html @@ -0,0 +1,11 @@ + + +3.3.2.2.Â Typecasting
Prev
Next
3.3.2.2.Â Typecasting
+ In most cases SystemTap can determine a variable's type from the debug information. However, code may use void pointers for variables (for example memory allocation routines) and typing information is not available. Also the typing information available within a probe handler is not available within a function; SystemTap functions arguments use a long in place of a typed pointer. SystemTap's @cast operator (first available in SystemTap 0.9) can be used to indicate the correct type of the object. +
+ The ExampleÂ 3.9, âCasting Exampleâ is from the task.stp tapset. The function returns the value of the state field from a task_struct pointed to by the long task. The first argument of the @cast operator, task, is the pointer to the object. The second argument is the type to cast the object to, task_struct. The third arument lists what file that the type definition information comes from and is optional. With the @cast operator the various fields of this particular task_struct task can be accessed; in this example the state field is obtained. +
ExampleÂ 3.9.Â Casting Example
function task_state:long (task:long) +{ + return @cast(task, "task_struct", "kernel<linux/sched.h>")->state +} +

Prev3.3.2.Â Target Variables
Up
Home
Next3.3.2.3.Â Checking Target Variable Availablility
diff --git a/SystemTap_Beginners_Guide/understanding-how-systemtap-works.html b/SystemTap_Beginners_Guide/understanding-how-systemtap-works.html index 5d865ce4..e29d79f4 100644 --- a/SystemTap_Beginners_Guide/understanding-how-systemtap-works.html +++ b/SystemTap_Beginners_Guide/understanding-how-systemtap-works.html @@ -1,22 +1,22 @@ -ChapterÂ 3.Â Understanding How SystemTap Works
Prev
Next
ChapterÂ 3.Â Understanding How SystemTap Works
3.1. Architecture
3.2. SystemTap Scripts
3.2.1. Event
3.2.2. Systemtap Handler/Body
3.3. Basic SystemTap Handler Constructs
3.3.1. Variables
3.3.2. Conditional Statements
3.3.3. Command-Line Arguments
3.4. Associative Arrays
3.5. Array Operations in SystemTap
3.5.1. Assigning an Associated Value
3.5.2. Reading Values From Arrays
3.5.3. Incrementing Associated Values
3.5.4. Processing Multiple Elements in an Array
3.5.5. Clearing/Deleting Arrays and Array Elements
3.5.6. Using Arrays in Conditional Statements
3.5.7. Computing for Statistical Aggregates
3.6. Tapsets
+ChapterÂ 3.Â Understanding How SystemTap Works
Prev
Next
ChapterÂ 3.Â Understanding How SystemTap Works
3.1. Architecture
3.2. SystemTap Scripts
3.2.1. Event
3.2.2. Systemtap Handler/Body
3.3. Basic SystemTap Handler Constructs
3.3.1. Variables
3.3.2. Target Variables
3.3.3. Conditional Statements
3.3.4. Command-Line Arguments
3.4. Associative Arrays
3.5. Array Operations in SystemTap
3.5.1. Assigning an Associated Value
3.5.2. Reading Values From Arrays
3.5.3. Incrementing Associated Values
3.5.4. Processing Multiple Elements in an Array
3.5.5. Clearing/Deleting Arrays and Array Elements
3.5.6. Using Arrays in Conditional Statements
3.5.7. Computing for Statistical Aggregates
3.6. Tapsets
SystemTap allows users to write and reuse simple scripts to deeply examine the activities of a running Linux system. These scripts can be designed to extract data, filter it, and summarize it quickly (and safely), enabling the diagnosis of complex performance (or even functional) problems. -
+
The essential idea behind a SystemTap script is to name events, and to give them handlers. When SystemTap runs the script, SystemTap monitors for the event; once the event occurs, the Linux kernel then runs the handler as a quick sub-routine, then resumes. -
+
There are several kind of events; entering/exiting a function, timer expiration, session termination, etc. A handler is a series of script language statements that specify the work to be done whenever the event occurs. This work normally includes extracting data from the event context, storing them into internal variables, and printing results. -
3.1.Â Architecture
+
3.1.Â Architecture
A SystemTap session begins when you run a SystemTap script. This session occurs in the following fashion: -
ProcedureÂ 3.1.Â SystemTap Session
+
ProcedureÂ 3.1.Â SystemTap Session
First, SystemTap checks the script against the existing tapset library (normally in /usr/share/systemtap/tapset/ for any tapsets used. SystemTap will then substitute any located tapsets with their corresponding definitions in the tapset library. -
- SystemTap then translates the script to C, running the system C compiler to create a kernel module from it. The tools that perform this step are contained in the systemtap package (refer to SectionÂ 2.1.1, âInstalling SystemTapâ for more information). -
- SystemTap loads the module, then enables all the probes (events and handlers) in the script. The staprun in the systemtap-runtime package (refer to SectionÂ 2.1.1, âInstalling SystemTapâ for more information) provides this functionality. -
+
+ SystemTap then translates the script to C, running the system C compiler to create a kernel module from it. The tools that perform this step are contained in the systemtap package (refer to SectionÂ 2.1.1, âInstalling SystemTapâ for more information). +
+ SystemTap loads the module, then enables all the probes (events and handlers) in the script. The staprun in the systemtap-runtime package (refer to SectionÂ 2.1.1, âInstalling SystemTapâ for more information) provides this functionality. +
As the events occur, their corresponding handlers are executed. -
+
Once the SystemTap session is terminated, the probes are disabled, and the kernel module is unloaded.
This sequence is driven from a single command-line program: stap. This program is SystemTap's main front-end tool. For more information about stap, refer to man stap (once SystemTap is properly installed on your machine). diff --git a/SystemTap_Beginners_Guide/understanding-tapsets.html b/SystemTap_Beginners_Guide/understanding-tapsets.html index 543ed552..13e3d28a 100644 --- a/SystemTap_Beginners_Guide/understanding-tapsets.html +++ b/SystemTap_Beginners_Guide/understanding-tapsets.html @@ -1,11 +1,11 @@ -3.6.Â Tapsets
Prev
Next
3.6.Â Tapsets
- Tapsets are scripts that form a library of pre-written probes and functions to be used in SystemTap scripts. When a user runs a SystemTap script, SystemTap checks the script's probe events and handlers against the tapset library; SystemTap then loads the corresponding probes and functions before translating the script to C (refer to SectionÂ 3.1, âArchitectureâ for information on what transpires in a SystemTap session). +3.6.Â Tapsets
Prev
Next
3.6.Â Tapsets
+ Tapsets are scripts that form a library of pre-written probes and functions to be used in SystemTap scripts. When a user runs a SystemTap script, SystemTap checks the script's probe events and handlers against the tapset library; SystemTap then loads the corresponding probes and functions before translating the script to C (refer to SectionÂ 3.1, âArchitectureâ for information on what transpires in a SystemTap session).
- Like SystemTap scripts, tapsets use the filename extension .stp. The standard library of tapsets is located in /usr/share/systemtap/tapset/ by default. However, unlike SystemTap scripts, tapsets are not meant for direct execution; rather, they constitute the library from which other scripts can pull definitions. + Like SystemTap scripts, tapsets use the file name extension .stp. The standard library of tapsets is located in /usr/share/systemtap/tapset/ by default. However, unlike SystemTap scripts, tapsets are not meant for direct execution; rather, they constitute the library from which other scripts can pull definitions.
Simply put, the tapset library is an abstraction layer designed to make it easier for users to define events and functions. In a manner of speaking, tapsets provide useful aliases for functions that users may want to specify as an event; knowing the proper alias to use is, for the most part, easier than remembering specific kernel functions that might vary between kernel versions.
- Several handlers and functions in SectionÂ 3.2.1, âEventâ and SystemTap Functions are defined in tapsets. For example, thread_indent() is defined in indent.stp. -
Prev3.5.7.Â Computing for Statistical Aggregates
Up
Home
NextChapterÂ 4.Â Useful SystemTap Scripts
+ Several handlers and functions in SectionÂ 3.2.1, âEventâ and SystemTap Functions are defined in tapsets. For example, thread_indent() is defined in indent.stp. +
Prev3.5.7.Â Computing for Statistical Aggregates
Up
Home
NextChapterÂ 4.Â User-Space Probing
diff --git a/SystemTap_Beginners_Guide/useful-systemtap-scripts.html b/SystemTap_Beginners_Guide/useful-systemtap-scripts.html index c6e8d400..a940ca27 100644 --- a/SystemTap_Beginners_Guide/useful-systemtap-scripts.html +++ b/SystemTap_Beginners_Guide/useful-systemtap-scripts.html @@ -1,15 +1,14 @@ -ChapterÂ 4.Â Useful SystemTap Scripts
Prev
Next
ChapterÂ 4.Â Useful SystemTap Scripts
4.1. Network
4.1.1. Network Profiling
4.1.2. Tracing Functions Called in Network Socket Code
4.1.3. Monitoring Incoming TCP Connections
4.1.4. Monitoring TCP Packets
4.1.5. Monitoring Network Packets Drops in Kernel
4.2. Disk
4.2.1. Summarizing Disk Read/Write Traffic
4.2.2. Tracking I/O Time For Each File Read or Write
4.2.3. Track Cumulative IO
4.2.4. I/O Monitoring (By Device)
4.2.5. Monitoring Reads and Writes to a File
4.2.6. Monitoring Changes to File Attributes
4.2.7. Periodically Print I/O Block Time
4.3. Profiling
4.3.1. Counting Function Calls Made
4.3.2. Call Graph Tracing
4.3.3. Determining Time Spent in Kernel and User Space
4.3.4. Monitoring Polling Applications
4.3.5. Tracking Most Frequently Used System Calls
4.3.6. Tracking System Call Volume Per Process
4.4. Identifying Contended User-Space Locks
+ChapterÂ 5.Â Useful SystemTap Scripts
Prev
Next
ChapterÂ 5.Â Useful SystemTap Scripts
5.1. Network
5.1.1. Network Profiling
5.1.2. Tracing Functions Called in Network Socket Code
5.1.3. Monitoring Incoming TCP Connections
5.1.4. Monitoring TCP Packets
5.1.5. Monitoring Network Packets Drops in Kernel
5.2. Disk
5.2.1. Summarizing Disk Read/Write Traffic
5.2.2. Tracking I/O Time For Each File Read or Write
5.2.3. Track Cumulative IO
5.2.4. I/O Monitoring (By Device)
5.2.5. Monitoring Reads and Writes to a File
5.2.6. Monitoring Changes to File Attributes
5.2.7. Periodically Print I/O Block Time
5.3. Profiling
5.3.1. Counting Function Calls Made
5.3.2. Call Graph Tracing
5.3.3. Determining Time Spent in Kernel and User Space
5.3.4. Monitoring Polling Applications
5.3.5. Tracking Most Frequently Used System Calls
5.3.6. Tracking System Call Volume Per Process
5.4. Identifying Contended User-Space Locks
This chapter enumerates several SystemTap scripts you can use to monitor and investigate different subsystems. All of these scripts are available at /usr/share/systemtap/testsuite/systemtap.examples/ once you install the systemtap-testsuite RPM. -
4.1.Â Network
+
5.1.Â Network
The following sections showcase scripts that trace network-related functions and build a profile of network activity. -
4.1.1.Â Network Profiling
- This section describes how to profile network activity. nettop.stp provides a glimpse into how much network traffic each process is generating on a machine. -
nettop.stp
+
5.1.1.Â Network Profiling
+ This section describes how to profile network activity. nettop.stp provides a glimpse into how much network traffic each process is generating on a machine. +
nettop.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap global ifxmit, ifrecv global ifmerged @@ -57,22 +56,17 @@ probe timer.ms(5000), end, error { print_activity() } -
-
+
Note that function print_activity() uses the following expressions: -
-n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0 -n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0 -
+
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0 +n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0
These expressions are if/else conditionals. The first statement is simply a more concise way of writing the following psuedo code: -
-if n_recv != 0 then +
if n_recv != 0 then @sum(ifrecv[pid, dev, exec, uid])/1024 else - 0 -
- nettop.stp tracks which processes are generating network traffic on the system, and provides the following information about each process: + 0
+ nettop.stp tracks which processes are generating network traffic on the system, and provides the following information about each process:
PID â the ID of the listed process.
@@ -88,38 +82,35 @@ else
RECV_KB â amount of data received by the service, in kilobytes
- nettop.stp provides network profile sampling every 5 seconds. You can change this setting by editing probe timer.ms(5000) accordingly. ExampleÂ 4.1, ânettop.stp Sample Outputâ contains an excerpt of the output from nettop.stp over a 20-second period: -
-[...] - PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND - 0 0 eth0 0 5 0 0 swapper -11178 0 eth0 2 0 0 0 synergyc + nettop.stp provides network profile sampling every 5 seconds. You can change this setting by editing probe timer.ms(5000) accordingly. ExampleÂ 5.1, ânettop.stp Sample Outputâ contains an excerpt of the output from nettop.stp over a 20-second period: +
ExampleÂ 5.1.Â nettop.stp Sample Output
[...] + PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND + 0 0 eth0 0 5 0 0 swapper +11178 0 eth0 2 0 0 0 synergyc - PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND - 2886 4 eth0 79 0 5 0 cups-polld -11362 0 eth0 0 61 0 5 firefox - 0 0 eth0 3 32 0 3 swapper - 2886 4 lo 4 4 0 0 cups-polld -11178 0 eth0 3 0 0 0 synergyc + PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND + 2886 4 eth0 79 0 5 0 cups-polld +11362 0 eth0 0 61 0 5 firefox + 0 0 eth0 3 32 0 3 swapper + 2886 4 lo 4 4 0 0 cups-polld +11178 0 eth0 3 0 0 0 synergyc - PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND - 0 0 eth0 0 6 0 0 swapper - 2886 4 lo 2 2 0 0 cups-polld -11178 0 eth0 3 0 0 0 synergyc - 3611 0 eth0 0 1 0 0 Xorg + PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND + 0 0 eth0 0 6 0 0 swapper + 2886 4 lo 2 2 0 0 cups-polld +11178 0 eth0 3 0 0 0 synergyc + 3611 0 eth0 0 1 0 0 Xorg - PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND - 0 0 eth0 3 42 0 2 swapper -11178 0 eth0 43 1 3 0 synergyc -11362 0 eth0 0 7 0 0 firefox + PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND + 0 0 eth0 3 42 0 2 swapper +11178 0 eth0 43 1 3 0 synergyc +11362 0 eth0 0 7 0 0 firefox 3897 0 eth0 0 1 0 0 multiload-apple -[...] -
ExampleÂ 4.1.Â nettop.stp Sample Output

4.1.2.Â Tracing Functions Called in Network Socket Code
+[...]

5.1.2.Â Tracing Functions Called in Network Socket Code
This section describes how to trace functions called from the kernel's net/socket.c file. This task helps you identify, in finer detail, how each process interacts with the network at the kernel level. -
socket-trace.stp
+
socket-trace.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap probe kernel.function("*@net/socket.c").call { printf ("%s -> %s\n", thread_indent(1), probefunc()) @@ -127,12 +118,10 @@ probe kernel.function("*@net/socket.c").call { probe kernel.function("*@net/socket.c").return { printf ("%s <- %s\n", thread_indent(-1), probefunc()) } -

- socket-trace.stp is identical to ExampleÂ 3.6, âthread_indent.stpâ, which was earlier used in SystemTap Functions to illustrate how thread_indent() works. -
-[...] + socket-trace.stp is identical to ExampleÂ 3.6, âthread_indent.stpâ, which was earlier used in SystemTap Functions to illustrate how thread_indent() works. +
ExampleÂ 5.2.Â socket-trace.stp Sample Output
[...] 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 Xorg(3611): -> sock_poll @@ -151,15 +140,13 @@ probe kernel.function("*@net/socket.c").return { 28 scim-bridge(3883): <- sys_recvfrom 31 scim-bridge(3883): <- sys_recv 35 scim-bridge(3883): <- sys_socketcall -[...] -
ExampleÂ 4.2.Â socket-trace.stp Sample Output

- ExampleÂ 4.2, âsocket-trace.stp Sample Outputâ contains a 3-second excerpt of the output for socket-trace.stp. For more information about the output of this script as provided by thread_indent(), refer to SystemTap Functions ExampleÂ 3.6, âthread_indent.stpâ. -
4.1.3.Â Monitoring Incoming TCP Connections
+[...]

+ ExampleÂ 5.2, âsocket-trace.stp Sample Outputâ contains a 3-second excerpt of the output for socket-trace.stp. For more information about the output of this script as provided by thread_indent(), refer to SystemTap Functions ExampleÂ 3.6, âthread_indent.stpâ. +
5.1.3.Â Monitoring Incoming TCP Connections
This section illustrates how to monitor incoming TCP connections. This task is useful in identifying any unauthorized, suspicious, or otherwise unwanted network access requests in real time. -
tcp_connections.stp
+
tcp_connections.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap probe begin { printf("%6s %16s %6s %6s %16s\n", @@ -173,10 +160,9 @@ probe kernel.function("tcp_accept").return?, printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(), inet_get_local_port(sock), inet_get_ip_source(sock)) } -

- While tcp_connections.stp is running, it will print out the following information about any incoming TCP connections accepted by the system in real time: + While tcp_connections.stp is running, it will print out the following information about any incoming TCP connections accepted by the system in real time:
Current UID
@@ -187,16 +173,13 @@ probe kernel.function("tcp_accept").return?, Port used by the connection
IP address from which the TCP connection originated -
-UID CMD PID PORT IP_SOURCE +
ExampleÂ 5.3.Â tcp_connections.stp Sample Output
UID CMD PID PORT IP_SOURCE 0 sshd 3165 22 10.64.0.227 -0 sshd 3165 22 10.64.0.227 -
ExampleÂ 4.3.Â tcp_connections.stp Sample Output

4.1.4.Â Monitoring TCP Packets
+0 sshd 3165 22 10.64.0.227

5.1.4.Â Monitoring TCP Packets
This section illustrates how to monitor TCP packets received by the system. This is useful in analyzing network traffic generated by applications running on the system. -
tcpdumplike.stp
+
tcpdumplike.stp
-
- #! /usr/bin/env stap +
#! /usr/bin/env stap // A TCP dump like example @@ -210,10 +193,9 @@ probe tcp.receive { printf(" %15s %15s %5d %5d %d %d %d %d %d %d\n", saddr, daddr, sport, dport, urg, ack, psh, rst, syn, fin) } -

- While tcpdumplike.stp is running, it will print out the following information about any received TCP packets in real time: + While tcpdumplike.stp is running, it will print out the following information about any received TCP packets in real time:
Source and destination IP address (saddr, daddr, respectively)
@@ -221,7 +203,7 @@ probe tcp.receive {
Packet flags
- To determine the flags used by the packet, tcpdumplike.stp uses the following functions: + To determine the flags used by the packet, tcpdumplike.stp uses the following functions:
urg - urgent
@@ -236,8 +218,7 @@ probe tcp.receive { fin - finished
The aforementioned functions return 1 or 0 to specify whether the packet uses the corresponding flag. -
------------------------------------------------------------------ +
ExampleÂ 5.4.Â tcpdumplike.stp Sample Output
----------------------------------------------------------------- Source IP Dest IP SPort DPort U A P R S F ----------------------------------------------------------------- 209.85.229.147 10.0.2.15 80 20373 0 1 1 0 0 0 @@ -251,14 +232,12 @@ probe tcp.receive { 209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0 209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0 209.85.229.118 10.0.2.15 80 63433 0 1 1 0 0 0 -[...] -
ExampleÂ 4.4.Â tcpdumplike.stp Sample Output

4.1.5.Â Monitoring Network Packets Drops in Kernel
- - The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb"), which easily tracks where packets are discarded. dropwatch.stp uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes which locations discard packets every five-second interval. -
dropwatch.stp
+[...]

5.1.5.Â Monitoring Network Packets Drops in Kernel
+ + The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb"), which easily tracks where packets are discarded. dropwatch.stp uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes which locations discard packets every five-second interval. +
dropwatch.stp
-
-#! /usr/bin/env stap +
#! /usr/bin/env stap ############################################################ # Dropwatch.stp @@ -280,44 +259,40 @@ probe kernel.trace("kfree_skb") { locations[$location] <<< 1 } # Every 5 seconds report our drop locations probe timer.sec(5) { - printf("\n") - foreach (l in locations-) { - printf("%d packets dropped at location %p\n", - @count(locations[l]), l) - } - delete locations + printf("\n") + foreach (l in locations-) { + printf("%d packets dropped at %s\n", + @count(locations[l]), symname(l)) + } + delete locations } - -

- The kernel.trace("kfree_skb") traces which places in the kernel drop network packets. The kernel.trace("kfree_skb") has two arguments: a pointer to the buffer being freed ($skb) and the location in kernel code the buffer is being freed ($location). -
- Running the dropwatch.stp script 15 seconds would result in output similar in ExampleÂ 4.5, âdropwatch.stp Sample Outputâ. The output lists the number of misses for tracepoint address and the actual address. -
-Monitoring for dropped packets + The kernel.trace("kfree_skb") traces which places in the kernel drop network packets. The kernel.trace("kfree_skb") has two arguments: a pointer to the buffer being freed ($skb) and the location in kernel code the buffer is being freed ($location). The dropwatch.stp script provides the function containing $location where possible. The information to map $location back to the function is not in the instrumentation by default. On Systemtap 1.4 the --all-modules option will include the required mapping information and the following command can be used to run the script: +
stap --all-modules dropwatch.stp
+ On older versions of Systemtap you can use the following command to emulate the --all-modules option: +
stap -dkernel \ +`cat /proc/modules | awk 'BEGIN { ORS = " " } {print "-d"$1}'` \ +dropwatch.stp
+ Running the dropwatch.stp script 15 seconds would result in output similar in ExampleÂ 5.5, âdropwatch.stp Sample Outputâ. The output lists the number of misses for each tracepoint location with either the function name or the address. +
ExampleÂ 5.5.Â dropwatch.stp Sample Output
Monitoring for dropped packets -51 packets dropped at location 0xffffffff8024cd0f -2 packets dropped at location 0xffffffff8044b472 +1762 packets dropped at unix_stream_recvmsg +4 packets dropped at tun_do_read +2 packets dropped at nf_hook_slow -51 packets dropped at location 0xffffffff8024cd0f -1 packets dropped at location 0xffffffff8044b472 +467 packets dropped at unix_stream_recvmsg +20 packets dropped at nf_hook_slow +6 packets dropped at tun_do_read -97 packets dropped at location 0xffffffff8024cd0f -1 packets dropped at location 0xffffffff8044b472 +446 packets dropped at unix_stream_recvmsg +4 packets dropped at tun_do_read +4 packets dropped at nf_hook_slow Stopping dropped packet monitor -
ExampleÂ 4.5.Â dropwatch.stp Sample Output

- To make the location of packet drops more meaningful, refer to the /boot/System.map-`uname -r` file. This file lists the starting addresses for each function, allowing you to map the addresses in the output of ExampleÂ 4.5, âdropwatch.stp Sample Outputâ to a specific function name. Given the following snippet of the /boot/System.map-`uname -r` file, the address 0xffffffff8024cd0f maps to the function unix_stream_recvmsg and the address 0xffffffff8044b472 maps to the function arp_rcv: -
-[...] -ffffffff8024c5cd T unlock_new_inode -ffffffff8024c5da t unix_stream_sendmsg -ffffffff8024c920 t unix_stream_recvmsg -ffffffff8024cea1 t udp_v4_lookup_longway -[...] -ffffffff8044addc t arp_process -ffffffff8044b360 t arp_rcv -ffffffff8044b487 t parp_redo -ffffffff8044b48c t arp_solicit -[...] -
Prev3.6.Â Tapsets
Up
Home
Next4.2.Â Disk
+

+ When the script is being compiled on one machine and run on another the --all-modules and /proc/modules directory are not available; the symname function will just print out the raw address. To make the raw address of packet drops more meaningful, refer to the /boot/System.map-`uname -r` file. This file lists the starting addresses for each function, allowing you to map the addresses in the output of ExampleÂ 5.5, âdropwatch.stp Sample Outputâ to a specific function name. Given the following snippet of the /boot/System.map-`uname -r` file, the address 0xffffffff8149a8ed maps to the function unix_stream_recvmsg: +
[...] +ffffffff8149a420 t unix_dgram_poll +ffffffff8149a5e0 t unix_stream_recvmsg +ffffffff8149ad00 t unix_find_other +[...]
Prev4.3.Â User-Space Stack Backtraces
Up
Home
Next5.2.Â Disk
diff --git a/SystemTap_Beginners_Guide/userspace-probing.html b/SystemTap_Beginners_Guide/userspace-probing.html new file mode 100644 index 00000000..d82d2e93 --- /dev/null +++ b/SystemTap_Beginners_Guide/userspace-probing.html @@ -0,0 +1,35 @@ + + +ChapterÂ 4.Â User-Space Probing
Prev
Next
ChapterÂ 4.Â User-Space Probing
4.1. User-Space Events
4.2. Accessing User-Space Target Variables
4.3. User-Space Stack Backtraces
+ SystemTap initially focused on kernel-space probing. However, there are many instances where user-space probing can help diagnose a problem. SystemTap 0.6 added support to allow probing user-space processes. SystemTap includes support for probing the entry into and return from a function in user-space processes, probing predefined markers in user-space code, and monitoring user-process events. +
+ The SystemTap user-space probing requires the utrace kernel extensions which provide an API for tracking various user-space events. More details about the utrace infrastructure are available at http://sourceware.org/systemtap/wiki/utrace. The following command determines whether the currently running Linux kernel provides the needed utrace support: +
grep CONFIG_UTRACE /boot/config-`uname -r`
+ If the Linux kernel support user-space probing, the following output is printed: +
CONFIG_UTRACE=y
+ The SystemTap user-space probing also needs the uprobes kernel module. If the uprobes kernel module is not available, you will see an error message like the following when attempting to run a script that requires the uprobes kernel module: +
SystemTap's version of uprobes is out of date. +As root, or a member of the 'root' group, run +"make -C /usr/share/systemtap/runtime/uprobes". +Pass 4: compilation failed. Try again with another '--vp 0001' option.
+ If this occurs, you need to generate a uprobes.ko module for the kernel as directed. +
4.1.Â User-Space Events
+ All user-space event probes begin with process. The process events can be limited to a specific running process by specifying the process ID. The process events can also be limited to monitoring a particular executable by specifying the path to executable (PATH). SystemTap makes use of the PATH environment variable, so both the name used on the command-line to start the executable and the absolute path to the executable can be used. Several of user-space probe events limit their scope to a particular executable name (PATH) because SystemTap must use debug information to statically analyzed where to places the probes, but for many user-space probes events the process ID and executable name are optional. Any process event in the list below that include process ID or the path to the executable must include those arguments. The process ID and path to the executable are optional for the process events that do not list them: +
process("PATH").function("function")
+ The entry to the user-space function function for the executable PATH. This event is the user-space analogue of the kernel.function("function") event. It allows wildcards for the function function and .return suffix. +
process("PATH").statement("statement")
+ The earliest instruction in the code for statement. This is the user-space analogue of kernel.statement("statement"). +
process("PATH").mark("marker")
+ The static probe point marker defined in PATH. Wildcards can be used for marker to specify mutiple marks with a single probe. The static probe points may also have numbered arguments ($1, $2, etc.) available to the probe. A variety of user-space packages such as Java include these static probe points. Most packages that provide static probe points also provide aliases for the raw user-space mark events. Below is one such alias for the x86_64 Java hotspot JVM: +
probe hotspot.gc_begin = + process("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so").mark("gc__begin")
process.begin
+ User-space process is created. This can be limited to a a particular process ID or a full path to the executable. +
process.thread.begin
+ User-space thread is created. This can be limited to a a particular process ID or a full path to the executable. +
process.end
+ User-space process died. This can be limited to a a particular process ID or a full path to the executable. +
process.thread.end
+ User-space thread is destroyed. This can be limited to a a particular process ID or a full path to the executable. +
process.syscall
+ User-space process makes a system call. The system call number is available via $syscall context variable, and the fist six arguments are available via $arg1 through $arg6. The ".return" suffix will place the probe at the return from the system call. For the "syscall.return" the return value is available through the $return context variable. This can be limited to a a particular process ID or a full path to the executable. +
Prev3.6.Â Tapsets
Up
Home
Next4.2.Â Accessing User-Space Target Variables
diff --git a/SystemTap_Beginners_Guide/using-systemtap.html b/SystemTap_Beginners_Guide/using-systemtap.html index 31f91e19..b1cd191d 100644 --- a/SystemTap_Beginners_Guide/using-systemtap.html +++ b/SystemTap_Beginners_Guide/using-systemtap.html @@ -1,22 +1,22 @@ -ChapterÂ 2.Â Using SystemTap
Prev
Next
ChapterÂ 2.Â Using SystemTap
2.1. Installation and Setup
2.1.1. Installing SystemTap
2.1.2. Installing Required Kernel Information RPMs
2.1.3. Initial Testing
2.2. Generating Instrumentation for Other Computers
2.3. Running SystemTap Scripts
+ChapterÂ 2.Â Using SystemTap
Prev
Next
ChapterÂ 2.Â Using SystemTap
2.1. Installation and Setup
2.1.1. Installing SystemTap
2.1.2. Installing Required Kernel Information RPMs
2.1.3. Initial Testing
2.2. Generating Instrumentation for Other Computers
2.3. Running SystemTap Scripts
2.3.1. SystemTap Flight Recorder Mode
This chapter instructs users how to install SystemTap, and provides an introduction on how to run SystemTap scripts. -
2.1.Â Installation and Setup
+
2.1.Â Installation and Setup
To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of -devel, -debuginfo and -debuginfo-common packages for your kernel. If your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel kernel, you will need to install the -devel and -debuginfo packages for each of those kernel versions.
These procedures will be discussed in detail in the following sections. -
Important
+
Important
Many users confuse -debuginfo with -debug. Remember that the deployment of SystemTap requires the installation of the -debuginfo package of the kernel, not the -debug version of the kernel. -
2.1.1.Â Installing SystemTap
+
2.1.1.Â Installing SystemTap
To deploy Systemtap, you will need to to install the following RPMs: -
+
systemtap
systemtap-runtime
Assuming that yum is installed in the system, these two rpms can be installed with yum install systemtap systemtap-runtime. Note that before you can use SystemTap, you will still need to install the required kernel information RPMs. -
2.1.2.Â Installing Required Kernel Information RPMs
+
2.1.2.Â Installing Required Kernel Information RPMs
SystemTap needs information about the kernel in order to place instrumentation in it (i.e. probe it). This information also allows SystemTap to generate the code for the instrumentation. This information is contained in the matching -devel, -debuginfo, and -debuginfo-common packages for your kernel. The necessary -devel and -debuginfo packages for the ordinary "vanilla" kernel are as follows:
kernel-debuginfo @@ -26,11 +26,9 @@ kernel-devel
Likewise, the necessary packages for the PAE kernel would be kernel-PAE-debuginfo, kernel-PAE-debuginfo-common, and kernel-PAE-devel. -
+
To determine what kernel your system is currently using, use: -
-uname -r -
+
uname -r
For example, if you wish to use SystemTap on kernel version 2.6.18-53.el5 on an i686 machine, then you would need to download and install the following RPMs:
kernel-debuginfo-2.6.18-53.1.13.el5.i686.rpm @@ -38,20 +36,10 @@ uname -r kernel-debuginfo-common-2.6.18-53.1.13.el5.i686.rpm
kernel-devel-2.6.18-53.1.13.el5.i686.rpm -
Important
+
Important
The version, variant, and architecture of the -devel, -debuginfo and -debuginfo-common packages must match the kernel you wish to probe with SystemTap exactly. -
- The easiest way to install the required kernel information packages is through yum install and debuginfo-install. debuginfo-install is included with later versions of the yum-utils package (for example, version 1.1.10), and also requires an appropriate yum repository from which to download and install -debuginfo/-debuginfo-common packages. -
- Most of the required kernel packages can be found at http://download.fedoraproject.org/pub/fedora/linux/releases/. Configure yum accordingly by adding a new "debug" yum repository file under /etc/yum.repos.d containing the following lines: -
-[fedora-debuginfo] -name=Fedora $releasever - $basearch - Debug -failovermethod=priority -baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/ -enabled=1 -
- After configuring yum with the appropriate repository, you can now install the required -devel, -debuginfo, and -debuginfo-common packages for your kernel. To install the corresponding packages for a specific kernel, run the following commands: +
+ The easiest way to install the required kernel information packages is through yum install and debuginfo-install commands. debuginfo-install is included with later versions of the yum-utils package (for example, version 1.1.10), and also requires an appropriate yum repository from which to download and install -debuginfo/-debuginfo-common packages. You can install the required -devel, -debuginfo, and -debuginfo-common packages for your kernel. To install the corresponding packages for a specific kernel, run the following commands:
yum install kernelname-devel-version
@@ -64,10 +52,9 @@ enabled=1 debuginfo-install kernel-PAE-2.6.18-53.1.13.el5
If you do not have yum and yum-utils installed (and you are unable to install them), you will have to manually download and install the required kernel information packages. To generate the URL from which to download the required packages, use the following script: -
fedoradebugurl.sh
+
fedoradebugurl.sh
-
-#! /bin/bash +
#! /bin/bash echo -n "Enter nvr of kernel-debuginfo (e.g. 2.6.25-14.fc9.x86_64) " ; \ read NVR; \ BASE=`uname -m` ; \ @@ -76,22 +63,19 @@ VERSION=`echo $NVR | awk -F- '{print $1}'` ; \ RELEASE=`echo $NVR | awk -F- '{print $2}'` ; \ echo "http://kojipkgs.fedoraproject.org/\ packages/kernel/$VERSION/$RELEASE/$BASE/" -

Once you have manually downloaded the required packages to the machine, install the RPMs by running rpm --force -ivh package_names. -
2.1.3.Â Initial Testing
+
2.1.3.Â Initial Testing
If you are currently using the kernel you wish to probe with SystemTap, you can immediately test whether the deployment was successful. If not, you will need to reboot and load the appropriate kernel.
To start the test, run the command stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'. This command simply instructs SystemTap to print read performed then exit properly once a virtual file system read is detected. If the SystemTap deployment was successful, you should get output similar to the following: -
-Pass 1: parsed user script and 45 library script(s) in 340usr/0sys/358real ms. +
Pass 1: parsed user script and 45 library script(s) in 340usr/0sys/358real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 290usr/260sys/568real ms. Pass 3: translated to C into "/tmp/stapiArgLX/stap_e5886fa50499994e6a87aacdc43cd392_399.c" in 490usr/430sys/938real ms. Pass 4: compiled C into "stap_e5886fa50499994e6a87aacdc43cd392_399.ko" in 3310usr/430sys/3714real ms. Pass 5: starting run. read performed -Pass 5: run completed in 10usr/40sys/73real ms. -
- The last three lines of the output (i.e. beginning with Pass 5 indicate that SystemTap was able to successfully create the instrumentation to probe the kernel, run the instrumentation, detect the event being probed (in this case, a virtual file system read), and execute a valid handler (print text then close it with no errors). +Pass 5: run completed in 10usr/40sys/73real ms.
+ The last three lines of the output (i.e. beginning with Pass 5) indicate that SystemTap was able to successfully create the instrumentation to probe the kernel, run the instrumentation, detect the event being probed (in this case, a virtual file system read), and execute a valid handler (print text then close it with no errors).
Prev1.2.Â SystemTap Capabilities
Up
Home
Next2.2.Â Generating Instrumentation for Other Compute...
diff --git a/SystemTap_Beginners_Guide/using-usage.html b/SystemTap_Beginners_Guide/using-usage.html index 962a91e8..00b6d524 100644 --- a/SystemTap_Beginners_Guide/using-usage.html +++ b/SystemTap_Beginners_Guide/using-usage.html @@ -1,43 +1,64 @@ -2.3.Â Running SystemTap Scripts
Prev
Next
2.3.Â Running SystemTap Scripts
+2.3.Â Running SystemTap Scripts
Prev
Next
2.3.Â Running SystemTap Scripts
SystemTap scripts are run through the command stap. stap can run SystemTap scripts from standard input or from file. -
+
Running stap and staprun requires elevated privileges to the system. However, not all users can be granted root access just to run SystemTap. In some cases, for instance, you may want to allow a non-privileged user to run SystemTap instrumentation on his machine.
To allow ordinary users to run SystemTap without root access, add them to one of these user groups: -
stapdev
+
stapdev
Members of this group can use stap to run SystemTap scripts, or staprun to run SystemTap instrumentation modules.
Running stap involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This requires elevated privileges to the system, which are granted to stapdev members. Unfortunately, such privileges also grant effective root access to stapdev members. As such, you should only grant stapdev group membership to users whom you can trust root access. -
stapusr
+
stapusr
Members of this group can only run staprun to run SystemTap instrumentation modules. In addition, they can only run those modules from /lib/modules/kernel_version/systemtap/. Note that this directory must be owned only by the root user, and must only be writable by the root user.
Below is a list of commonly used stap options: -
-v
+
-v
Makes the output of the SystemTap session more verbose. You can repeat this option (for example, stap -vvv script.stp) to provide more details on the script's execution. This option is particularly useful if you encounter any errors in running the script.
- For more information about common SystemTap script errors, refer to ChapterÂ 5, Understanding SystemTap Errors. -
-o filename
+ For more information about common SystemTap script errors, refer to ChapterÂ 6, Understanding SystemTap Errors. +
-o filename
Sends the standard output to file (filename). -
-x process ID
- Sets the SystemTap handler function target() to the specified process ID. For more information about target(), refer to SystemTap Functions. -
-c command
- Sets the SystemTap handler function target() to the specified command. Note that you must use the full path to the specified command; for example, instead of specifying cp, use /bin/cp (as in stap script -c /bin/cp). For more information about target(), refer to SystemTap Functions. -
-e 'script'
+
-S size,count
+ Limit files to size megabytes and limit the the number of files kept around to count. The file names will have a sequence number suffix. This option implements logrotate operations for SystemTap. +
-x process ID
+ Sets the SystemTap handler function target() to the specified process ID. For more information about target(), refer to SystemTap Functions. +
-c 'command'
+ Sets the SystemTap handler function target() to the specified command and runs the SystemTap instrumentation for the duration of the specified command. For more information about target(), refer to SystemTap Functions. +
-e 'script'
Use script string rather than a file as input for systemtap translator. +
-F
+ Use SystemTap's Flight recorder mode and make the script a background process. For more information about flight recorder mode, refer to SectionÂ 2.3.1, âSystemTap Flight Recorder Modeâ.
You can also instruct stap to run scripts from standard input using the switch -. To illustrate: -
-echo "probe timer.s(1) {exit()}" | stap - -
ExampleÂ 2.1.Â Running Scripts From Standard Input

- ExampleÂ 2.1, âRunning Scripts From Standard Inputâ instructs stap to run the script passed by echo to standard input. Any stap options you wish to use should be inserted before the - switch; for instance, to make the example in ExampleÂ 2.1, âRunning Scripts From Standard Inputâ more verbose, the command would be: +
ExampleÂ 2.1.Â Running Scripts From Standard Input
echo "probe timer.s(1) {exit()}" | stap -

+ ExampleÂ 2.1, âRunning Scripts From Standard Inputâ instructs stap to run the script passed by echo to standard input. Any stap options you wish to use should be inserted before the - switch; for instance, to make the example in ExampleÂ 2.1, âRunning Scripts From Standard Inputâ more verbose, the command would be:
echo "probe timer.s(1) {exit()}" | stap -v -
For more information about stap, refer to man stap.
- To run SystemTap instrumentation (i.e. the kernel module built from SystemTap scripts during a cross-instrumentation), use staprun instead. For more information about staprun and cross-instrumentation, refer to SectionÂ 2.2, âGenerating Instrumentation for Other Computersâ. -
Note
+ To run SystemTap instrumentation (i.e. the kernel module built from SystemTap scripts during a cross-instrumentation), use staprun instead. For more information about staprun and cross-instrumentation, refer to SectionÂ 2.2, âGenerating Instrumentation for Other Computersâ. +
Note
The stap options -v and -o also work for staprun. For more information about staprun, refer to man staprun. -
Prev2.2.Â Generating Instrumentation for Other Compute...
Up
Home
NextChapterÂ 3.Â Understanding How SystemTap Works
+
2.3.1.Â SystemTap Flight Recorder Mode
+ SystemTap's flight recorder mode allows you to run a SystemTap script run for long periods and just focus on recent output. The flight recorder mode (the -F option) limits the amount of output generated. There are two variations of the flight recorder mode: in-memory and file mode. In both cases the SystemTap script runs as a background process. +
2.3.1.1.Â In-memory Flight Recorder
+ When flight recorder mode (the -F option) is used without a file name SystemTap uses a buffer in kernel memory to store the output of the script. The SystemTap instrumentation module will load and the probes start running, the instrumentation will then detach and be put in the background. When the interesting event occurs, you can reattach to the instrumentation and see the recent output in the memory buffer and any continuing output. The following command starts a script using the flight recorder in-memory mode: +
stap -F iotime.stp
+ Once the script starts, you will see a message like the following that provides the command to reconnect to the running script: +
Disconnecting from systemtap module. +To reconnect, type "staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556"
+ When the interesting event occurs, you reattach to the currently running script and output the recent data in the memory buffer and get continuing output with the following command: +
staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556
+ By default the kernel buffer is 1MB in size and it can be increased with the -s option specifying the size in megabytes (rounded up to the next power over 2) for the buffer. For example -s2 on the SystemTap command line would specify 2MB for the buffer. +
2.3.1.2.Â File Flight Recorder
+ The flight recorder mode can also store data to files. The number and size of the files kept is controlled by the -S option followed by two numerical arguments separated by a comma. The first argument is the maximum size in megabytes for the each output file. The second argument is the number of recent files to keep. The file name is specified by the -o option followed by the name. SystemTap will add a number suffix to the file name to indicate the order of the files. The following will start SystemTap in file flight recorder mode with the output going to files named /tmp/iotime.log.[0-9]+ and each file 1MB or smaller and keeping latest two files: +
stap -F -o /tmp/pfaults.log -S 1,2 pfaults.stp
+ The number printed by the command is the process ID. Sending a SIGTERM to the process will shutdown the SystemTap script and stop the data collection. For example if the previous command listed the 7590 as the process ID, the following command whould shutdown the systemtap script: +
kill -s SIGTERM 7590
+ Only the most recent two file generated by the script are kept and the older files are been removed. Thus, ls -sh /tmp/pfaults.log.* shows the only two files: +
1020K /tmp/pfaults.log.5 44K /tmp/pfaults.log.6
+ One can look at the highest number file for the latest data, in this case /tmp/pfaults.log.6. +
Prev2.2.Â Generating Instrumentation for Other Compute...
Up
Home
NextChapterÂ 3.Â Understanding How SystemTap Works
diff --git a/SystemTap_Beginners_Guide/ustack.html b/SystemTap_Beginners_Guide/ustack.html new file mode 100644 index 00000000..72b2a0cd --- /dev/null +++ b/SystemTap_Beginners_Guide/ustack.html @@ -0,0 +1,31 @@ + + +4.3.Â User-Space Stack Backtraces
Prev
Next
4.3.Â User-Space Stack Backtraces
+ The probe point (pp) function indicates which particular event triggered the the SystemTap event handler. A probe on the entry into a function would list the function name. However, in many cases the same probe point event may be triggered by many different modules in the program; this is particularly true for functions in shared libraries. A SystemTap backtrace of the user-space stack can provide additional context on how the probe point event is triggered. +
+ The user-space stack backtrace generation is complicated by the compiler producing code optimized to eliminate stack frame pointers. However, the compiler also includes information in the debug information section to allow debugging tools to produce stack backtraces. SystemTap user-space stack backtrace mechanism makes use of that debug information to walk the stack to generate stack traces for 32-bit and 64-bit x86 processors; other processor architectures do not yet support the use of debug information to unwind the user-space stack. You will need to use the -d executable for the application executable and -ldd for shared libraries to ensure that the needed debug information is used to produce the user-space stack backtraces. +
+ If you want to see how the function xmalloc function is being called by the command ls, you could use the user-space backtrack functions to provide that information. With the debuginfo for the ls command installed the following SystemTap command will provide a backtrace each time the xmalloc function is called: +
stap -d /bin/ls --ldd \ +-e 'probe process("ls").function("xmalloc") {print_ustack(ubacktrace())}' \ +-c "ls /"
+ When the SystemTap script runs will have output similar to the following: +
+bin dev lib media net proc sbin sys var +boot etc lib64 misc op_session profilerc selinux tmp +cgroup home lost+found mnt opt root srv usr + 0x4116c0 : xmalloc+0x0/0x20 [/bin/ls] + 0x4116fc : xmemdup+0x1c/0x40 [/bin/ls] + 0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls] + 0x4087e4 : main+0x3b4/0x1900 [/bin/ls] + 0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so] + 0x402799 : _start+0x29/0x2c [/bin/ls] + 0x4116c0 : xmalloc+0x0/0x20 [/bin/ls] + 0x4116fc : xmemdup+0x1c/0x40 [/bin/ls] + 0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls] + 0x40884a : main+0x41a/0x1900 [/bin/ls] + 0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so] + ... +
+ For more details on the functions available for user-space stack backtraces look at the ucontext-symbols.stp and ucontext-unwind.stp tapsets. The descriptions of the functions in those tapsets can also be found in the SystemTap Tapset Reference Manual. +
Prev4.2.Â Accessing User-Space Target Variables
Up
Home
NextChapterÂ 5.Â Useful SystemTap Scripts
diff --git a/SystemTap_Beginners_Guide/utargetvariable.html b/SystemTap_Beginners_Guide/utargetvariable.html new file mode 100644 index 00000000..b6049044 --- /dev/null +++ b/SystemTap_Beginners_Guide/utargetvariable.html @@ -0,0 +1,17 @@ + + +4.2.Â Accessing User-Space Target Variables
Prev
Next
4.2.Â Accessing User-Space Target Variables
+ User-space target variables can be accessed in the same manner as described in SectionÂ 3.3.2, âTarget Variablesâ. However, in Linux there are separate address spaces for the user and kernel code. When using "->" operator SystemTap will access the appropriate address space. For pointers to base types such as integers and strings there are a number of functions listed below to access user-space data. The first argument for each functions is the pointer to the data item. +
user_char(address)
+ Obtain the character at address for the current user process. +
user_short(address)
+ Obtain the short at address for the current user process. +
user_int(address)
+ Obtain the int at address for the current user process. +
user_long(address)
+ Obtain the long at address for the current user process. +
user_string(address)
+ Obtain the string at address for the current user process. +
user_string_n(address, n)
+ Obtain the string at address for the current user process and limits the string to n bytes. +
PrevChapterÂ 4.Â User-Space Probing
Up
Home
Next4.3.Â User-Space Stack Backtraces
-- 2.43.5