William Cohen writes:
> This afternoon I looked over the LTT-0.9.6-pre4 kernel patch
> from http://www.opersys.com/ltt/index.html. This was to see what event
> and which locations in the kernel are insrumented, what kind of
> functionality could be provided with the existing kprobe
> instrumentation, what things need additional help to implement, and
> what things are unlikely to be instrumentable. I will refer to things
> by the LTT event define in most cases.
>
> There are some LTT probe points that can be implemented with jprobes,
> probe at function entry and just using arguments passed into function:
>
> LTT_EV_PROCESS_WAIT
> LTT_EV_TIMER_EXPIRED
> LTT_EV_MEMORY_PAGE_FREE
> LTT_EV_SOCKET_SEND
> LTT_EV_SOCKET_RECEIVE
> do_page_fault() entry
>
> There are some that could implemented as regular function exit probes:
> do_page_fault() exit
>
> There are some that are basically exit probes but use an argument
> passed into the function:
> LTT_EV_SOCKET_CREATE (needs type arg)
> LTT_EV_MEMORY_PAGE_ALLOC (needs order arg)
>
>
> However, many LTT probes are the middle of functions, e.g. ones for
> scheduling and exit logging. Commonly, there is some checking that the
> operation can be done. After all the checks the operation is
> guaranteed to be sucessful. The probe is inserted after those checks
> to avoid writing a log entry for something that didn't actually
> happen. Quite often there is a function call do the operation after
> the checks, e.g. LTT_EV_MEMORY_SWAP_IN in
> mm/filemap.c:wait_on_page_bit() immediately before swapin_readahead().
> Maybe specify have kernel functions specify points where the error
> checking is all done in the function and a probe could/should go
> there, with a list of variables live there.
>
> Other alternatives spec of probe points:
> 1) have instrument callee functions and have predicates to specify
> caller
> 2) allow instrumentation to specify probe point before a call to
> a particular function in the function.
> Could do instruction after call, but need to be careful because
> the following instruction could be in a different basic block
>
>
> The do_traps logging in LTT will not work because kprobes uses traps
> and it would be recursive.
>
> LTT systemcall logging appears to collect do all or no
> syscalls. Similarly for traps, all or nothing. Given that traps and
> syscall do a wide variety of things, finer grain control of these
> would be desired.
>
> Need to get data transfer mechanism in place. printks are not going to
> work well for the volume of data that LTT produces. However, it would
> be hoped that some of the data analysis could be done in the probe and
> reduce the need to produce a trace that is analyzed offline.
>
> Wednesday I am going to see about implementing some of the LTT probe
> points with kprobes.
>
I believe the LKST project may have already done something along these
lines - I'm cc:ing Karim Yaghmour, the LTT maintainer, who would know
more. Following the recent lkml thread on the subject, I believe LKST
and LTT have also been combining efforts on a common framework for
high-speed tracing; it would seem to make sense to work with them and
get some synergy between the different tracing tools.
What level of integration with LTT were you planning on? Would an
'ltt provider' make sense? LTT provides a set of events and an API to
enable/disable them - the provider could just use these directly and
not worry about instrumenting anything i.e. I don't see why systemtap
would care whether they're instrumented statically or via kprobes -
that's already taken care of by LTT. As is the data transfer
mechanism - it uses relayfs for that; in the most recent versions of
relayfs, the file operations are exported so they can be used in other
filesystems - we should be able to put the ltt trace files into
'tapfs' directories if that's the way we want to go. (BTW, relayfs
has been pared down significantly in order to 'do one thing and do it
well', which is high-speed kernel->user data transfers and therefore
would't be useful as a general kernel<->user mechanism for systemtap
i.e. it's strictly one-way and doesn't support read(2) or anything
really besides mmap()). I believe new versions of LTT will also
support a generic filtering callback which can be used to filter the
event stream in the kernel if you want to.