process().statement() doesn't seem to work

[Zoltan Kiss]

On 30/06/15 00:39, Josh Stone wrote:
> On 06/26/2015 11:58 AM, Zoltan Kiss wrote:
>> Hi,
>>
>> I want to use systemtap to figure out the local variables in a function
>> where something goes fishy. The function's code is here:
>>
>> http://dpdk.org/browse/dpdk/tree/lib/librte_pmd_ixgbe/ixgbe_rxtx.c?id=v1.7.1#n1313
>>
>> It's a receive function of DPDK's userspace poll mode driver, so it's
>> called in an infinite loop. When I try this, it appers to work:
>>
>> probe
>> process("/usr/sbin/ovs-vswitchd").function("ixgbe_recv_scattered_pkts").return
>> {
>>     log($$locals)
>>     exit();
>> }
>>
>> The output is:
>>
>> rxq=0x7fff852ee000 rx_ring=? rxdp=? sw_ring=? rxe=? first_seg=?
>> last_seg=? rxm=? nmb=? rxd={...} dma=? staterr=? hlen_type_rss=? rx_id=?
>> nb_rx=0x0 nb_hold=0x0 data_len=? pkt_flags=?
>>
>> But it doesn't show most of the variables I need. It returns nb_rx=0, so
>> I know where are the two points where things can go wrong, but I can't
>> see the output of those variables (staterr and nmb)
>
> The behavior of .return variables is often suprising to people.  By the
> time a .return probe is reached, the program actually will have already
> left that stack frame.  Technically, that means only the $return value
> is available to read.
>
> But we cheat a little and also save values from the *entry* of the
> function for you.  That's most useful for the $$parms string or any of
> the individual parameters.
>
> Sometimes you can also get a glimpse of $$locals, if the debuginfo tells
> us how, but again this is only from the very beginning of the function.
>   In your case, the three visible values all happen to have trivial
> initialization: nb_rx = 0; nb_hold = 0; rxq = rx_queue;
>
> So you can't trust that these were still the values after the function
> returned.  For that, you really need to probe the statement just before
> it actually returns.  Even on the return statement itself is ok.  Since
> your function has just one return, it looks like line 1574 is best.
> (but depending on optimizations, some locals might be missing there.)
>
>> When I try to define the probe as this:
>>
>> probe
>> process("/usr/sbin/ovs-vswitchd").statement("*@lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1359")
>>
>> Or with any line number inside that function, the probe is never
>> reached. I'm using gcc 4.8.4, DPDK is statically linked to my application.
>> Does anyone have an idea what might went wrong?
>
> It may be that you're really not reaching that line, e.g. if nb_pkts==0.
>
> You can see all the lines that stap can probe like this:
>
> stap -l 'process("/usr/sbin/ovs-vswitchd")
>           .statement("ixgbe_recv_scattered_pkts@*:*")'
>
> Or change -l to -L to also show all readable variables at each point.

Thanks, I've figured out what I want in other ways. I'm still facing 
problems with stap, but let's start a different thread for that

>
>
> You might like the varwatch example script:
>
> https://sourceware.org/systemtap/examples/#general/varwatch.stp
>
> It takes two arguments - first a probe pattern like your
> process.statement wildcard, then the variable to watch like $$parms,
> $$locals, or $$vars for both.  A single name like $nb_rx is fine too.
>