Re: Interoperability of LTTng and LTTV with SystemTAP

[Tom Zanussi <zanussi at us dot ibm dot com>]

Tom Zanussi writes:
 > Mathieu Desnoyers writes:
 >  > Hi,
 >  > 
 >  > I just had a discussion with Frank Ch. Eigler from the SystemTAP team on the
 >  > topic "how can the tools cooperate well together".
 >  > 
 >  > SystemTAP is especially made for running user defined scripts at specific points
 >  > in the kernel. I has the ability to add/remove scripts and instrumentation
 >  > dynamically from a running kernel.
 >  > 
 >  > LTTng is a kernel tracer designed to have the minimum impact on the system while
 >  > having a design solid enough that it can trace NMI handlers (no locking). It
 >  > supports native C types and alignment. It can register new instrumentation sites
 >  > from dynamically loaded modules on-the-fly.
 >  > 
 >  > LTTV is a modular text/graphical trace analysis tool. You can think of it as
 >  > framework offered for plugins to interact together to show information. Simple
 >  > plugins can show generically some information (detailed event list, for
 >  > instance), while more sophisticated plugins can show a particular representation
 >  > of the system (each process state (user mode, system call, interrupted,
 >  > in trap, ...) evolving through time).
 >  > 
 >  > While the SystemTAP project focus on pre-processing of the information, LTT
 >  > focus on raw data copy (limiting the impact at probe site) at the cost of more
 >  > data to be transferred and analysed afterward.
 >  > 
 >  > I think that both approaches are complementary. I am trying to see where
 >  > integrated analysis of pre-processed and post-processed information could be
 >  > interesting, but I'm sure there are areas where it is.
 >  > 
 >  > If you have application ideas, feel free to express yourself.
 >  > 
 > 
 > One area that seems tailor-made for this type of interaction would be
 > to use systemtap probes to detect 'interesting' conditions in the
 > trace data, at which point the buffered event stream to that point
 > could be dumped to disk for further analysis, leaving a detailed trail
 > of information as to what exactly led to a condition i.e. an
 > intelligent flight recorder.  'Interesting' conditions could comprise
 > a single event occurrence, but since probes can maintain state over
 > multiple events, more complex combinations of events ('compound'
 > events) could be detected as well.
 > 

One other thought along these lines that I forgot to mention was that
it might also be useful for probes to be able to trigger tracing as
needed e.g. a probe notices some strange pattern in the data currently
being aggregated, and in response basically says, 'Hmm, this is
strange, let's turn on tracing and see if we can't get some more
detailed info about what's going on'.  The event mask could also be
dynamically changed based on the conditions detected by the probe,
etc...

Tom