Script to measure resource usage based on process arguments

[Mike Mason]

I wrote the attached script in response to a customer question. They wanted to know if SystemTap could show them which set of java processes were consuming the most resources, based on the process arguments.  They may have several thousand java instances running at one time and those instances can have very long argument strings.  I'm not sure the script gives them exactly what they want, but I thought I'd send this out for comments anyway.

The script lets you to specify process names and strings to look for in their arguments.  It then aggregates the resources used by all processes with those names and  argument strings.  In this case, it tracks cpu time, but you could modify it to track any resource value that's available in task_struct.  It prints the results at the end of an interval.  For example, I ran the script as shown below during a kernel build. 

# ./argtap -i 30 -n gcc -n as -a fs/ -a mm/ -a arch/ -a security/ -a crypto/ -a drivers/ 
======================= Mon Jan  8 15:59:05 2007 ========================
TIME(ms) #PROC  PROCNAME         ARG SEGMENT
1472     33     as               drivers/
975      25     as               fs/
91       35     gcc              drivers/
69       27     gcc              fs/

======================= Mon Jan  8 15:59:35 2007 ========================
TIME(ms) #PROC  PROCNAME         ARG SEGMENT
2174     55     as               fs/
683      17     as               drivers/
155      56     gcc              fs/
44       17     gcc              drivers/

The results show how many gcc and as processes with the specified strings in their arguments ran during a 30 second interval and how much total cpu time they used.

While writing this script I had to figure out how get the arguments from existing processes. That took awhile.  Since the arguments are stored in the processes' address space, we can't just access them whenever we want to.  The usual routines used in /proc to grab another processes' arguments won't work in systemtap because irqs are always disabled in probes, even begin probes. We can't call functions that might sleep in systemtap.  I eventually realized I could access the arguments when the process is "current" in the scheduler.cpu_off probe.  This is okay because the script doesn't care about processes that don't get scheduled.

Note that there's a lot of embedded C in this script, which I don't really like, but I couldn't think of any other way to do what needed to be done.  Also note I had to use a big MAXACTION value to get it to work in some cases.  Given that limitation, I'm beginning to think systemtap scripts should be used more for extracting raw data and data processing should be left to user scripts and programs, especially where large amounts of data are involved.

Anyway, please look this over and give me your comments.  I'm especially interested in ways I could improve it.

Mike Mason

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: argtap
URL: <http://sourceware.org/pipermail/systemtap/attachments/20070109/48be6dc8/attachment.ksh>