[UPDATE] Linux Kernel Event Trace tool(LKET)

[William Cohen]

Li Guanglei wrote:
> Li Guanglei ??:
> 
>> Li Guanglei ??:
>>
>>> [...]
>>> So I'd like to break LKET's tapsets into the following two layers:
>>>
>>> <1> Generic Tapsets
>>> [...]
>>
>> I've checked them into tapset directory(together with some testcases),
>> They are:
>> ioscheduler.stp, memory.stp, networking.stp, scsi.stp,
>>
>> tskschedule.stp & process.stp from LKET were merged into scheduler.stp 
>> and process.stp respectively.
>>
>> In a summary, the following probe alias are defined:
>> [...]
>>
>>>
>>> <2> Tapsets Specific to Tracing
>>> [...]
>>> --Guanglei
>>>
>> All these traceing specific tapsets are checked into tapset/LKET 
>> direcotry
>>
>> I am writing the man pages for lket to make it easier to use.
>>
> 
> I've checked into cvs the draft man page for LKET. Now you can type "man 
> lket" to get an outline about LKET, including event hooks intro, 
> examples, trace data format etc.
> 
> But I think it will be better if I can add LKET's man page onto 
> SystemTap's website, i.e the "Man Pages" section in:
> http://sourceware.org/systemtap/documentation.html
> Because SystemTap's website is the first place people will go.
> 
> But I don't know how to add entries into the webpage. I am afraid I 
> don't have the permission. Could somebody help me on this?

I read through the LKET manpage this morning and tried out LKET with 
providing ASCII output on a 32-bit i686 FC5 machine and a 64-bit x86_64 
FC5 machine. The kernels are listed below.

Linux montague.devel.redhat.com 2.6.16-1.2122_FC5 #1 Sun May 21 15:01:01 
EDT 2006 i686 i686 i386 GNU/Linux

Linux dhcp59-221.rdu.redhat.com 2.6.16-1.2208_FC6 #1 SMP Sat May 20 
21:48:05 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux

It looks like there are some portability issues that need to be worked 
out in the code. Things seemed to work better on the 64-bit platform 
than on the 32-bit platform.



$ stap -D ASCII_TRACE -e "probe addevent.syscall.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

syscall seemed to generate reasonable data on 64-bit machine. However, 
on the 32-bit machine I got traces that looked like the following:

1|1|0|1148483176|803233|3311|3308|3311|<NULL>
1|2|0|1148483176|803245|3311|3308|3311|<NULL>
1|1|0|1148483176|803259|3311|3308|3311|<NULL>
1|2|0|1148483176|803303|3311|3308|3311|<NULL>


$ stap -D ASCII_TRACE -e "probe addevent.process.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

process trace seemed to generate reasonable information on both 32- and 
64-bit machines initially. Then later the 32-bit seemed to miss 
information like the following:

2|1|1148483294|472220|14862|14772|14862|0|13853|emacs-x
2|1|1148483294|472225|14862|14772|14862|0|14772|stap
2|3|0|1148483297|562834|1848|1|1848|0
2|3|0|1148483312|563887|1848|1|1848|0

$ stap -D ASCII_TRACE -e "probe addevent.ioscheduler.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

On 32-bit patform got bogus information:
3|1|0|1148483567|535253|0|0|0|<NULL>|-1070280432|22
3|3|0|1148483567|536381|3311|3308|3311|<NULL>|-1070280432|22
3|3|0|1148483567|537053|2091|2068|2091|<NULL>|-1070280432|22

The 64-bit information looked better however, sometimes saw traces like 
the following:

3|1|1148483662|28320|11716|11530|11716|0|cfq|-1|-1


stap -D ASCII_TRACE -e "probe addevent.tskdispatch.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

The tskdispatch didn't work on either machine. Couldn't find 
kernel.function("__switch_to") on 64-bit and couldn't find 
kernel.inline("idle_balance") on 32-bit


Neither machine has scsi interface. When trying out the flagged errors 
because no scsi modules.

$ stap -D ASCII_TRACE -e "probe addevent.pagefault {}" -I 
/usr/local/share/systemtap/tapset/LKET/


The data on the 32-bit machine looked bogus (see below). lots of 
negative references. Also would it make more sens to have the address 
printed in hexidecimal for the ascii trace?

6|1|0|1148483972|626999|1848|1|1848|35705849838043136|1
6|1|0|1148483972|627050|1848|1|1848|-4615772671219597312|1
6|1|0|1148483974|465231|1324|1|1324|-5191788353242005504|1


$ stap -D ASCII_TRACE  -e "probe addevent.netdev.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

The netdev worked on 64-bit, but it failed compilation on 32-bit:

cc1: warnings being treated as errors
/tmp/stapBs84Hc/stap_15261.c: In function 'function_log_netdev_extra':
/tmp/stapBs84Hc/stap_15261.c:434: warning: cast to pointer from integer 
of different size



-Will