RE: pre-compiled modules

["Stone, Joshua I" <joshua dot i dot stone at intel dot com>]

On Tuesday, June 27, 2006 12:59 PM, David Smith wrote:
>>> Hmm, I hadn't considered that.  Will this code go in the module
>>> itself or go in the runtime?
>> 
>> Is there a difference?  The runtime is still compiled into the
>> module, right?
> 
> Sorry, I meant to say: "Will this code go in the module itself or the
> stpd daemon?"

Well, it's probably difficult for stpd to determine where the systemtap module thinks that all the kernel modules are loaded.  But I also don't know of any kernel API's for looking up modules addresses.  It may have to be some ugly cross where stpd looks in /proc/modules for current addresses, and the systemtap module keeps an array of where it thinks the module bases are.  Then pass addresses over the transport layer to compare (either direction would work).  We don't need to match all modules exactly, just the ones that we want to place probes in.  It would probably be best to compare the sizes too, as a crude version check.

Hien's suggestion about some sort of metadata file could work too, and then you can do all checks in stpd.  You then just have to make sure that the metadata and ko are paired correctly...
 
>> I would prefer to see an option to specify the directory, instead of
>> assuming $PWD.  You could always use '-S .', but some people might
>> want to drop it somewhere else.  Another possibility instead of -S
>> is to just extend the -m option to allow a path.
> 
> That's easy enough to add, but it seems like that 99% of the time I'd
> want it to go in the current directory.
> 
> Anyone else got any opinions?

I'll defer to popular opinion, of course.  I think the difference in opinion is really about what the usage model looks like.  Your example usage was to simply compile it, save to the local directory, and immediately turn around and run it.  But in this model, I don't see what value has been added over just running it in one shot.

I think the value of pre-compiled modules is that it enables you to keep a library of modules that users can fire up at will.  You would probably have a directory somewhere that is your systemtap module repository, and you might not even run the module right away.  One could imagine that stpd could be augmented to support a module search path, à la modprobe.  For these cases, I think the ability to specify the output directory would be helpful.

>> Have you thought about concurrent access to a precompiled module?  If
>> you have a systemtap module foo.ko on a multi-user system, you might
>> end up with a situation where multiple people want to run it at the
>> same time.  Of course you can only insmod one at a time, if nothing
>> else because of the naming issue.
> 
> Hmm, no I hadn't considered that.  We've also got a similar but
> different problem.  What happens when 2 users both compile two
> different scripts called 'foo.stp' into 'foo.ko', then try to run them
> concurrently?
> 
> I'm unsure of what we can do in either case.

Right now we hack around this by including the stap PID in the default name.  Perhaps the default name could instead just derive from the script name (foo.stp -> foo.ko), and then at load time the module could be copied to a unique name (foo_$PID.ko, stap_foo_$PID.ko, etc.).  Your -P option could manage this before it calls stpd.

I tried to do this manually, but it seems this won't work anyway.  As it turns out, the name of the module is compiled in, so renaming the ko later has no effect.  It looks like the name is specified in stap_###.mod.c.  I think the only way we'll be able to handle renaming is to save only the stap_###.o, and finish the compile into a ko with the desired name at load time.  Or we could try to patch the name in the ko, but that's a lot uglier.

This actually exposes a small bug in stpd too -- stpd looks in /proc/systemtap/<name>/ to communicate with the module, but it derives <name> from the filename, which won't match the module's real name if someone renamed it.


Josh