Questions about kprobes implementation on SMP and preemptible kernels

[Quentin Barnes]

On Mon, Jan 29, 2007 at 09:57:55AM +0530, Ananth N Mavinakayanahalli wrote:
>On Sat, Jan 27, 2007 at 05:59:48PM -0600, Quentin Barnes wrote:
>> On Tue, Jan 16, 2007 at 11:09:55AM +0530, Ananth N Mavinakayanahalli wrote:
>> >On architectures which run kprobes with interrupts enabled, its not
>> >uncommon to have interrupts occur while we are processing a kprobe. In
>> >fact, the kprobe reentry handling code was a byproduct of such a
>> >scenario happening.
>> 
>> I'm not sure I follow this point clearly enough.
>> 
>> As far as I follow, recursion for kprobes is only handled to no more
>> than one level when re-entered and only from a kprobe encountered
>> in a pre- or post-handler.  In that case, kprobe_status is set to
>> KPROBE_REENTER and the pre- and post-handlers aren't run for that
>> one recursion level so we don't recurse yet again.
>> 
>> If we're processing a kprobe with interrupts enabled and an
>> interrupt fires and its handler has a kprobe, we would re-enter.  At
>> this point, this would "work" in the sense that kprobe's pre- and
>> post-handler would not run even though they may not contain any
>> kprobes themselves.  Is this considered an acceptable behavior?
>
>Yes. At this point, you just increment struct kprobe->nmissed to
>indicate that you missed running the handlers and move on.

I suppose.

>> If the same scenario happened above (kprobe in interrupt handler
>> and its interrupt fired) but this time we're already nested from
>> handling a kprobe and another kprobe in its pre- or post-handler,
>> save_previous_kprobe() would overwrite an already active previous
>> kprobe state.  This would seem to be a serious bug.
>
>Though such a scenario is eminently possible, we haven't encountered it
>in the real world (yet :).

It doesn't matter if it's actually been been encountered or not.  An
unexecuted bug in the code's logic is still a bug in the logic.

I would like to know for sure if it would happen or not though.

>AFAIR, Anil had a patch to handle this
>specific case. Not sure what the status of the patch is at the moment.
>The other question here is, how many levels of nesting would you allow?
>What'd you do beyond that nesting level?

The reason to allow interrupts in the first place is to reduce
interrupt latency to improve system responsiveness.  With interrupts
enabled, the number of kprobes you could encounter in interrupt
service routines is unbounded.  Even if at some deep nesting level
you chose to disable interrupts to stop the nesting, you've still
effectively created an elongated critical section at that time.
You've only delayed it, not avoided it, so it's still there.  So
you might as well have that elongated critical section initially up
front by never re-enabling interrupts in the first place and just
paying the price.  In other words, that's my very long explanation
for why there's no point in having more than one nesting level as we
have now.

>> Which architectures allow interrupts while handling a kprobe?
>
>Powerpc and AFAIR x86_64 and IA64 run with interrupts enabled
>throughout.

I looked at PowerPC in 2.6.20-rc4.  The function
program_check_exception() is calling notify_die() for a breakpoint
exeception before it is calling local_irq_enable().  I don't
see anywhere in the powerpc arch kprobes code interrupts being
re-enabled.  But I don't know the PowerPC code all that well.  Can
you point me me to where interrupts are getting re-enabled while a
kprobe is being processed?  I haven't written any PowerPC assembly
in over a decade, so I shied away from trying to parse head_8xx.S.

>Ananth

Quentin