• TipContextVariables

Troublesome Context Variables

Due to limitations of compiler-generated debugging data, sometimes $variables that you know exist may not be accessible to systemtap probes. You may see error messages such as ...

• semantic error: not accessible at this address: identifier '$block' ... 

• semantic error: failed to retrieve location attribute for local: '$var' ... 

This is a known problem. The basic reason is that, until recently, gcc has not seriously attempted to produce accurate debugging data for highly-optimized programs. With http://gcc.gnu.org/wiki/Var_Tracking_Assignments merged in gcc 4.5, the situation is better, but still not perfect.

With an imperfect compiler, what can one do? There are several possibilities.

• If you're using statement probes, try specifying a line number nearby the original one.
• Try specifying a related variable from which the original variable may be computed.
• If you're probing the entry to a function that was inlined, try probing the call site within its caller (using a .statement probe). The variables may be accessible there. You can save/delete the parameters in some tid()-indexed lookup table in a .call/.return probe pair, and access that saved copy within the probe on the inlined function.

• If you are able to recompile the probed software, you could compile parts of it with less optimization. Something less drastic like -fno-inline-functions-called-once may help.

• If you are able to change the probed software, you could add dummy code that attempts to ensure that particular values stay "live" and debugger-visible at certain points. One way could be to strategically insert inline-assembly directives. This may preserve those variables, without materially altering the generated code (even if fully optimized):

  asm volatile (" /* %0 */ " :: "r" (variable));

• If your script might not need that particular context variable after all, version 1.2+ of systemtap supports a testing function to see if some $variable expression is resolvable:

  if (@defined($var->field)) { println($var->field) } else { /* oh well */ }

If you want to help debug why the location information is missing (for a nice and juicy gcc bug report for example) then try something like (lets assume we are looking for the task parameter from the nfs_read_done function in the nfs module):

• eu-readelf -N --debug-dump=info /usr/lib/debug/lib/modules/3.1.0-0.rc1.git1.1.fc17.x86_64/kernel/fs/nfs/nfs.ko.debug | less

• Look for the function:

   [10974b]    subprogram
               name                 (strp) "nfs_read_done"
               decl_file            (data1) 1
               decl_line            (data2) 631
               prototyped           (flag_present) Yes
               type                 (ref4) [ f3cc0]
               inline               (data1) inlined (1)
               sibling              (ref4) [109775]
   [10975c]      formal_parameter
                 name                 (strp) "task"
                 decl_file            (data1) 1
                 decl_line            (data2) 631
                 type                 (ref4) [ fd230]
   [109768]      formal_parameter
                 name                 (strp) "data"
                 decl_file            (data1) 1
                 decl_line            (data2) 631

• Notice the call is inlined. So we want to look for reference to either the subprogram [10974b] or the actual formal_parameter [10975c].

• We find the formal_parameter a little later:

     [10a1b1]      formal_parameter
                 abstract_origin      (ref4) [10975c]
                 location             (sec_offset) location list [ 23713]

  The location attribute is there, which is good (if it is missing, we already have our problem).

• Now look at the location list with eu-readelf --debug-dump=loc /usr/lib/debug/lib/modules/3.1.0-0.rc1.git1.1.fc17.x86_64/kernel/fs/nfs/nfs.ko.debug | less and search for offset 23713:

   [ 23713]  .text+0x00000000000002bd <nfs_init_server_rpcclient+0x35>...text+0x00000000000002f0 <nfs_init_server_rpcclient+0x68>
   [   0] reg5
             .text+0x00000000000002f0 <nfs_init_server_rpcclient+0x68>...text+0x0000000000000356 <nfs_init_server_rpcclient+0xce>
   [   0] reg12
             .text+0x0000000000000356 <nfs_init_server_rpcclient+0xce>...text+0x000000000000035d <nfs_init_server_rpcclient+0xd5>
   [   0] GNU_entry_value:
         [   0] reg5
         [   3] stack_value

  So this was inlined into the nfs_init_server_rpcclient function. Note there are 3 address ranges with 3 different location expressions for finding the task parameter.
• Now figure out whether these locations match your probe point address and/or the location expression makes sense. If anything looks odd or is missing file a gcc bug report. If everything looks like stap should be able to figure out, file a systemtap bug report. (Note that in this case it might be the last location expression, stap currently doesn't deal with DW_OP_GNU_entry_value - which here says that task can be found in the last code range by taking the value of reg5 upon entering this function.)

Tips