kernel_timing_systemtap_generator.awk is, almost obviously, an AWK script that
reads from its input lists of kernel functions and outputs a standardized
systemtap script that collects timings on the durations of the calls to those
functions. So you write in a file a set of kernel functions which you'd like
to "profile" and feed it to the AWK script, and in the output you get a
systemtap script that will time those calls.

The input to the AWK script can be in AWK's standard input, or can come from a
set of files you pass as paremeters to the AWK script's command line. The
output of the AWK script goes to AWK's standard output, so you'll want to
redirect that to a file, named something useful like "nfs_probes.stap".

The format of the input to the AWK script is one kernel function per line, or
one module name and module function per line. For example:

io_schedule
udp_queue_rcv_skb
nfs nfs3_proc_getattr
ext3 ext3_delete_inode

If the first character in a line fed to the script is a '#' it will ignore the
line. This way you can keep a large file containing kernel function names that
interest you and comment out some of them when needed.

The generated systemtap script will collect a microsecond timestamp upon entry
to each of the functions passed in, indexing the timestamps by function name
and process id, and upon function exit will calculate the time spent by the
process inside the function and feed that into a systemtap statistics
collector indexed by function names. When you stop the probe, it will iterate
(ascending asciibetic order) over the function names for which statistics were
collected, and print out each function name, how many calls it received,
minimum time spent on it, average time spent on it, and maximum time spent on
it, in a nicely formatted table.

This generator was made in hopes that it would be useful for quickly comparing
timing differences between kernel releases or different tuning parameters, so
that one can easily focus the debugging of performance regressions or simply
accurately determine the best set of system tunings.

The output from the generated systemtap script should look like this:

$ ./kernel_timing_systemtap_generator.awk nfs_v3_clientops_functions > \
	nfs_v3_clientops_test.stap

$ sudo stap nfs_v3_clientops_test.stap
... do some NFS testing and hit ^C
function                    count   min_us   avg_us   max_us
nfs3_decode_dirent            150        2        2        5
nfs3_forget_cached_acls        69        2        2        7
nfs3_proc_access                4   269623   302668   348510
nfs3_proc_fsinfo                1   274941   274941   274941
nfs3_proc_get_root              1   546148   546148   546148
nfs3_proc_getattr              92   259162   304874   800149
nfs3_proc_read_setup            1        4        4        4
nfs3_proc_readdir               4   292728   550859   790494
nfs3_proc_readlink              1   634002   634002   634002
nfs3_read_done                  1       11       11       11
nfs_open                        3        5        5        5
nfs_release                     3        5        5        6

Yes, that was NFS over a really slow link. :)