Experiences with kprobes

[Baruch Even <baruch at ev-en dot org>]

Hello,

Just thought to share my current experience with kprobes, it might 
interest some of you.

I'm trying to improve the performance of the Linux TCP stack (as an 
end-host not a router), as such I need to measure the current 
performance in order to search for bottlenecks.

I had a first version where I simply wrapped the the calls I needed with 
rdtsc calls inline and added other measurements (number of packets acked 
 for each ACK packet and such). This worked beautifully, and I got some 
nice results and some pretty good improvements as well.

They say "if it ain't broken don't fix it", but if it's not broken it's 
no fun[0], so I tried to use kprobes as a way to get the measurement 
code out of my current code patches. The thinking was that it will be a 
lot easier to maintain the patches ready for LKML submission.

I also ported my code to 2.6.11 (since that's where kprobes is 
available, I was on 2.6.6 before and no kprobes there[1]), and got 
abysmal performance. After a bit of digging the overhead of the kprobes 
approach was the only possible problem, if with the old method I got a 
timing of about 3000 clocks on my machine[2], with the new one I got at 
least 10000 with about 3 kprobes and 3 jprobes.

I ported kprobes to 2.6.6 and the same performance patterns appeared on 
the formerly working code, with the only conclusion left that kprobes is 
not suitable for this kind of performance measurements under very high 
loads.

The specifics for me is that the tests are running using dummynet 
network to simulate a very high speed long distance network (about 300ms 
 rtt and 300Mbit/s bandwidth) so the packet rates are very high with 
BDP of about 8000 packets, i.e. lots of ack packets to process).

Baruch

[0] As a grad student, at least part of the idea is to have fun :-)

[1] If someone needs a back port of kprobes for 2.6.6 on i386 send me a 
note.

[2] 3GHz P-IV Xeon