Experiences with kprobes

[Baruch Even]

William Cohen wrote:
> I wrote some simple tests to check the overhead of kprobes and jprobes. 
> I have run them on an athlon and pentium III machine, but I haven't run 
> them on an pentium IV. It could be the costs are higher on Pentium IV. 
> Could you give these a try on your pentium IV machine? The following URL 
> has an attachment with software for measuring overhead:
> 
> http://sources.redhat.com/ml/systemtap/current/msg00093.html

I'll try that and report back.

> Also is an smp kernel or premption being used? The current locking 
> mechanism in kprobes serializes multiple kprobes. Is it being possible 
> that some of the overhead could be due to serialization of the probes?

Actually I'm using a UP kernel with no preemption due to limitations of 
oprofile, I need two profiling registers to check CPU utilization and 
memory accesses and so SMP is out (according to the OProfile developers).

>> The specifics for me is that the tests are running using dummynet 
>> network to simulate a very high speed long distance network (about 
>> 300ms  rtt and 300Mbit/s bandwidth) so the packet rates are very high 
>> with BDP of about 8000 packets, i.e. lots of ack packets to process).
> 
> What kind of rate are the probes firing at?  n*8000 probe firings per 
> second?  Could the delay introduced by the probes be affect behavior?

In the test with the probes I only get the cwnd up to about 3000, with a 
round trip of 300ms and delayed acking, it means 3000*1000/300/2 = 5000 
packets per second. But the real problem is that when we really get to 
this level and we loss a packet we start handling SACKs which are sent 
for each packet and not for every two packets, so we get double the rate 
of ACKs which sends us to 10000 pps, at which stage the probes will 
probably affect behaviour significantly enough.

At that point I usually see a complete failure to handle the stream of 
packets and network throttling kicks in killing the connection even further.

>> [0] As a grad student, at least part of the idea is to have fun :-)
> 
> Even if you are not a graduate student the previous line holds. :)

Still haven't found a work place that accepts that...

Baruch