[RFC] Multiple kprobes at an address redux (take3)

[Vara Prasad]

Suparna Bhattacharya wrote:

>On Mon, Apr 11, 2005 at 10:59:10PM -0700, Vara Prasad wrote:
>  
>
>>Maneesh Soni wrote:
>>
>>    
>>
>>>On Tue, Apr 12, 2005 at 09:26:50AM +0530, Suparna Bhattacharya wrote:
>>>
>>>
>>>      
>>>
>>>>It would be nice to summarize all the requirements upfront, before
>>>>moving on to the design.
>>>>
>>>>  
>>>>
>>>>        
>>>>
>>>I had the impression that systemTap needs that register_kprobe() should 
>>>never fail with -EEXIST. That means, if there is a kprobe inserted by some 
>>>other utility at the same location where systemTap wants to insert, the 
>>>kprobe
>>>mechanism should still go ahead and add handlers to the existing kprobe.
>>>
>>>
>>>      
>>>
>>That is right, we should allow multiple handlers at any given probe point.
>>
>>    
>>
>>>Thanks to Suparna, I see there could be a problem in this which might 
>>>occur due to this non-exclusiveness of kprobes. Consider the following 
>>>scenario,
>>>
>>>utility 1 inserts kprobe k1 with handler h1 at address A
>>>utility 2 wants to insert kprobe k2 with handler h2 at the same address A
>>>
>>>In this situation if we do _not_ fail user 2 and add handler h2 to the 
>>>existing
>>>kprobe, what might happen is that if at the hit of kprobe, the handler h1 
>>>can modify some data and things might not be as expected by h2. The result 
>>>could be the incorrect information being provided by utility 2.
>>>
>>>      
>>>
>>When we have multiple handlers we are not guaranteeing the order of 
>>execution of probes hence probe handlers should not assume any order. 
>>Handlers should  have the logic to verify the data of their interest 
>>before using,  if data is not appropriate they should handle that 
>>gracefully and exit.
>>
>>    
>>
>>>I understand that systemTap needs multiple handler probes, but still there 
>>>should be some exclusiveness among the kprobes inserted by different 
>>>utilities.
>>>One utility can synchronize its own multiple handlers for the same probe 
>>>but
>>>with other utilities.
>>>
>>>
>>>      
>>>
>>I think synchronization should be left to the users of the probes rather 
>>than trying to complicate the interface with exclusive probes and non 
>>exclusive probes. 
>>    
>>
>
>Indeed. That is exactly the point -- we can impose the requirement of
>verification on systemtap probes or multihandler probes in general,
>but changing all kprobes users to do this may not be appropriate.
>Simply keeping the same name of the interface and changing its semantics is
>potentially problematic.
>
>kprobes is a simple low level interface - the only expectation on
>kprobes handlers today is ability to execute in interrupt context
>and not recurse into itself. Other than that, a handler is a kernel
>function - which can do whatever it wants ... it would be rather hard
>if not impossible to verify everything.
>
>So, I can't see a way to make unrelated sets of users of probes
>to synchronize themselves :(
>
>  
>
Coming to changing the semantics by keeping the same name, was there any 
assurance that kprobes will not allow multiple probes, if not then this 
is not a change. If this breaks any previous assurances then we can not 
use same name. I agree with your point that we should not change the 
API's in a fashion that affects the end users but it may not be such a 
problem at this time as there are not many users.

What i envision most of the times probes will do is read data of 
interest to do some kind of analysis and decision making. Analysis and 
decision making can be done in the kernel or outside the kernel. For the 
most part using probes to change the data or system state is not safe 
and left to guru's to try it in their debugging environment where they 
can afford to take the system down.  Considering the most common usage 
scenarios for the most part multiple probes should not alter the state 
of the system.

If we want to provide multiple unrelated sets of probes to synchronize 
we have to either provide an api on top of  kernel synchronization 
primitives that can be used in the probes or we have to implement 
synchronization in the kprobes itself.   Another easy approach could be 
document this and let users follow normal synchronization primitives. 
But  if we don't implement synchronization in the kprobes itself you 
hope every one is using same protocol to synchronize otherwise there 
will be no consistency.

>This is why having the higher level kprobe call chain (for want of a better
>name), abstracted on top of kprobes made sense to me. 
>
>Could you explain the need for systemtap to takeover or insert additional
>handlers to existing kprobes in the system (potentially installed by
>other utilities or even by a kernel programmer for quick experimentation)
>a little more clearly ? Where did that requirement originate ? (I'm sorry
>I haven't really been following the design discussions on system tap)
>
>  
>
One of the reasons is return probes needs to place a probe at the entry, 
second is multiple users should be able to run systemtap scripts. One 
more reason is multiple probe packages of tapsets might be enabled with 
different probe handlers using wild card scheme like "*". An example 
might be user might want to try to enable all system calls using 
systemcall tap set and at the same time all the functions in the vm 
tapset which might have few system calls as well hence, intersection at 
some probes. Hope this gives you an idea of multiple probes usage that 
we are envisioning today.

>Regards
>Suparna
>
>  
>
>>>Probably now we need to modify the design having this problem in mind.
>>>
>>>Thanks
>>>Maneesh
>>>
>>>
>>>
>>>      
>>>
>>    
>>
>
>  
>