systemtap memory usage

[Martin Hunt <hunt at redhat dot com>]

In the last month I have been looking at memory usage. Some recent
changes I checked in have provided some improvement and the rewrite of
the transport will help even more.  Here is a breakdown of the current
memory usage on x86.

For each probe:
staprun = 750K
module = 50K for simple probe, 1M for a very complex one tracing all
syscalls.
procfs = 2M
relayfs = 1M PER CPU (if used. in addition to 2M for procfs)
symbol and module database = 750-800K

So best case for a small probe using procfs is 3.5Mbytes. For a large
relayfs probe it would be 4.5M plus 1M per cpu.

But wait. We also have "stap" hanging around while the probe runs.
That's a huge 71Mbytes.

With the transport rewrite I am working on, we will eliminate the 2M
procfs and perhaps limit relayfs to a single shared 1M or 2M buffer
instead of percpu. (stap -b would still do per-cpu buffers for maximum
performance). staprun is also much smaller.

What next?  

Clearly we should rethink having "stap" hang around. 

The symbol and module database is a candidate to be eliminated if we can
get either a few small changes to the kernel or we spin the code out
into its own module. 

Hitachi is proposing having only one set of system-wide transport
buffers shared among all systemtap scripts.  This saves some memory at
the cost of performance. It would also require some radical changes to
our architecture. staprun becomes stpd again and it downloads all
systemtap data, and distributes it among multiple registered clients...
I have doubts the performance cost will be acceptable to everyone. 

Comments? Concerns?  Does anyone else care about keeping memory usage
down?