comments from prospective VM tapset author
Jim Keniston
jkenisto@us.ibm.com
Wed May 4 23:16:00 GMT 2005
Here are notes from conversations with Darren Hart, who is working on VM
instrumentation that we hope will lead to a tapset. So far, he's happy
with kprobes, and will continue as best he can in C. As we get more
details about what a SystemTap tapset is supposed to look like, we'll
plan a conversion to a tapset.
Keep in mind that the goal of these conversations was to gather
information from Darren, not to sell him on the benefits of SystemTap.
Jim
--
Summary of conversations with Darren Hart Mon 4/25/05 and Wed 4/27/05
Darren works for IBM's Linux Technology Center. He has expertise in
the scheduler and, recently, in VM. He's interested in kprobes and
SystemTap for two main reasons:
1. He's chasing down a VM performance problem, and these tools sound
like they could help.
2. He has the expertise and the willingness to create (or help create)
SystemTap-based instrumentation (AKA a tapset) for the VM subsystem.
Vara and I met with Darren for about 90 minutes on 4/25/05. Darren and
I also talked for 10-15 minutes 1-2 days later.
Darren has been using kprobes and jprobes, and he really likes them.
So far, access to function args via jprobes has been sufficient for
him, but he foresees the need to access local variables in the middle
of a function. We talked about how with kprobes, you typically have to
use a disassembler to figure out where in a function to place a
probepoint, and to verify in what register or stack location the
variable resides.
We talked about ways in which SystemTap could provide access to such
variables and code locations more conveniently using debug info.
Darren didn't like the idea of having to switch to a new language in
order to do his instrumentation. His idea of what the instrumentation
should be includes the following points:
- coded in C
- SystemTap provides kernel functions and/or macros that provide
standardized, convenient access to local variables in a probed
function.
- Instrumentation is a library of functions similar to what Vara
envisioned, where each instrumentation function collects an advertised
set of values (typically only a few, maybe 10 max) and makes them
available through a standard mechanism (to be defined by us SystemTap
folks). A SystemTap-generated handler would call such a function to
obtain the advertised values.
We discussed the possibility of making the SystemTap runtime available
for use by the kprobes-based instrumentation that Darren is starting to
write. I was doubtful from what little I remembered about the runtime,
but I said I would investigate.
Darren provided the attached file, vm_instrumentation.txt, which
summarizes some thoughts on what instrumentation SystemTap should
provide in the VM area. For example, SystemTap should be able to
provide at least the VM info provided by /proc/meminfo, vmstat,
and atsar. Some of the stats he listed are probably of special
interest to the performance problem that he's analyzing. Darren
would like to be able to report the values of counters and levels,
function timings, return values, and function args. (I pointed out
that function timings would have to take into account the overhead
of the kprobes-related traps.) He would like to be able to record
stats on a per-process and/or per-CPU basis.
TAPSET TEMPLATE
I gave Darren a copy of Frank's "tapset/script information template"
from 4/15/05, and asked him to review it as someone who might be asked
to fill it out. His initial response was that the typical subsystem
expert would probably prefer to document their instrumentation after
the fact, touching on whatever points seem relevant to him/her.
He gave me his review comments 1-2 days later. His written comments
included the following:
- "Seems like a lot. Who fills it out? Who reads it?"
- Background should be combined with Motivation.
- Target software should be obvious from Motivation/Background.
- Interesting values, Data collection, and Data presentation should be
combined.
He said he could see the value of the items requested, but that not all
items would be relevant for all instrumentation sets.
-------------- next part --------------
VM Instrumentation
Currently Have
Polled info from meminfo,vmstat,atsar
most of these are running totals
linuxdocs has variables etc.
access these vars via systemtap
dvhart@kernel:~/work/linux/views$ cat /proc/meminfo
MemTotal: 16375212 kB
MemFree: 11519588 kB
Buffers: 831052 kB
Cached: 2991988 kB
SwapCached: 0 kB
Active: 1750788 kB
Inactive: 2110144 kB
HighTotal: 14548952 kB
HighFree: 11497984 kB
LowTotal: 1826260 kB
LowFree: 21604 kB
SwapTotal: 32611940 kB
SwapFree: 32611940 kB
Dirty: 84 kB
Writeback: 0 kB
Mapped: 48584 kB
Slab: 963620 kB
CommitLimit: 40799544 kB
Committed_AS: 287812 kB
PageTables: 2232 kB
VmallocTotal: 116728 kB
VmallocUsed: 16408 kB
VmallocChunk: 100104 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
dvhart@kernel:~/work/linux/views$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 11519588 831064 2992236 0 0 2 0 0 0 5 1 93 0
dvhart@kernel:~/work/linux/views$ atsar -rp 1 1
Linux kernel 2.6.11-22 #1 SMP Wed Mar 30 22:53:03 PST 2005 i686 04/25/2005
13:06:23 memtot memfree buffers cached slabmem swptot swpfree _mem_
13:06:24 15991M 11243M 812M 2924M 943M 31847M 31847M
13:06:23 pagein/s pageout/s swapin/s swapout/s fork/s _page_
13:06:24 0.00 0.00 0.00 0.00 5.00
Want
Mostly mem reclaim
How long shrink_caches etc. take
who called it
next 2: at which function?
all that receive a scan_control argument such as
shrink_zone
shrink_cache
shrink_caches
how many pages requested
how many freed to get them (ie cascading dirty inode problem)
per process stats
per processor stats
link start and end calls together
we considered __return_... something to use the stack pointer
balance of user pages vs cache pages (balance different types of memory)
kernel data, app data, fs data
DB2 wants this
latency instrumentation (swap in/out, something else? etc)
More information about the Systemtap
mailing list