This is the mail archive of the mailing list for the systemtap project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Path resolution inconsistency between process("binary") and @var("var@cu","module")

Hi all

TL;DR: @var("...","module") doesn't resolve "module" against PATH like
process("execname") does; confusion ensures. SCCE included, fix
proposed, comments requested.


I've recently discovered how amazing and powerful SystemTap is, after
having it on my backburner to learn for years while struggling to
achieve anything with linux-perf.

I've been keeping notes on some suggestions to make for the
documentation which I'll send separately. But first I've found (and
worked around) an issue with user-space probing that I'd like to
patch, but I'm looking for confirmation and advice first.

The gist is that the PATH-resolution done by "process(...)" probes is
not consistent with how @var(...) and @cast(...) resolve modules
within stap functions called by probes. This causes stap to report
very confusing failures to resolve symbols that are obviously right
there as far as the user is concerned.

The working cases are, for some probe on a function/statement in a
userspace executable:

- @var("sym@CU") directly in a probe; doesn't require module argument at all
- @var("sym@CU", "/full/path/to/executable") in a function() called by
a probe: finds var

The failing cases are, all when in a function() called by a probe of a
userspace executable:

- @var("sym@CU") in the function: reports missing symbol, tries to
resolve in "kernel"
- @var("sym@CU","executable") in the function: also tries to resolve in kernel

The probe type doesn't seem to matter that much:
process("executable"), process("/path/to/executable"), process() and
target mode, process(pid), etc etc all behave the same. @cast has
similar issues to @var.

SCCE with cases shown:

program to probe:

     /* path_vs_atvar.c */
     #include <stdlib.h>
     static int global_variable = 0;
     int main(int argc, char *argv[])
         global_variable = argc > 1 ? atoi(argv[1]) : 0;
         return global_variable;
     /* end path_vs_atvar.c */


    gcc -ggdb3 -Wall path_vs_atvar.c -o path_vs_atvar


     function get_global()
         // Doesn't work, preprocesses to @var(...,"kernel") [1]
         //return @var("global_variable@path_vs_atvar.c");

         // Doesn't work, module argument treated as kernel [2]
         //return @var("global_variable@path_vs_atvar.c","path_vs_atvar");

         // Fully qualified path as literal string: works,
cumbersome/impractical [3]

         // Path as script-arg: works, though also cumbersome as user
must resolve
         // and specify path as absolute path which defeats the point
of path-expansion
         // in process probes, doesn't work well with -c, etc. [4]
         return @var("global_variable@path_vs_atvar.c",@1);

     probe process.function("main").return
         // Works: resolves variable in correct module for any
suitable probe type like:
         //     process("path_vs_atvar")
         //     process("/path/to/path_vs_atvar")
         //     process($1)  with path arg
         //     process() using -c mode
         // etc. [0]
         //printf("global is %d\n", @var("global_variable@path_vs_atvar.c"));

         // Wheras when we indirect via a function, results vary
         printf("global is %d\n", get_global());

To invoke, uncomment the [case] you want to try and run:

    sudo stap path_vs_atvar.stp -c ./path_vs_atvar

for case [1] the result is:

     probe main@/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar.c:3
reloc=.absolute pc=0x401126
     semantic error: unresolved @var() expression: operator '@var' at
        thrown from: elaborate.cxx:6621
             source:     return @var("global_variable@path_vs_atvar.c");

     Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
using 262184virt/143804res/13596shr/129868data kb, in
60usr/120sys/187real ms.
     Missing separate debuginfos, use: debuginfo-install

You'd expect case [1] above, with no module-spec, to resolve the @var
in the context the function is called in, but it doesn't. It's trying
to use the "kernel" module per -vvv:

    return @var("global_variable@probe_vs_atvar.c", "kernel")
    semantic error: unresolved @var() expression: operator '@var' at
       thrown from: elaborate.cxx:6621
            source:     return @var("global_variable@probe_vs_atvar.c");

so that's weird. The global is clearly there and in the specified CU:

    $ gdb -q -ex "info variables global_variable" -ex quit ./path_vs_atvar
    Reading symbols from ./path_vs_atvar...
    All variables matching regular expression "global_variable":

    File path_vs_atvar.c:
    2: static int global_variable;

But if you change it to explicitly reference the module
"probe_vs_atvar" per [2] the outcome is the same, even though we
*know* the module "path_vs_atvar" must exist and be accessible since
we're being invoked from a .process probe on it.

After all, it works when directly within the probe body [0].

If you use an absolute path to the module then the @var works within
the function [4], e.g.


but that's exceedingly impractical.

The most tolerable workaround I found so far was to use a commandline
argument to specify the full path to the executable. This makes
process PATH resolution completely useless, but at least it works,
like in case [5] in the SCCE.

As I have quite a large set of executables to probe I tried to
simplify use by using a macro to expand sub-paths from a basedir, but
that doesn't work either. Implicit string concatenation doesn't get
repeated after @MACRO expansion so @var("sym@CU", @MYBASEDIR
"bin/foo")  won't work. I didn't find any language support for
preprocessing-time explicit string concatenation.

So here I am.

I did some code digging. It's my first time looking at the sytemtap
code and I've only used it for a week, so please forgive
errors/oversights, but AFAICS the important code is
dwarf_atvar_expanding_visitor(...) and parse_atvar_op(...).

I haven't quite worked out why the case in [0] works but I'm assuming
it's something to do with focus_on_module().

For the function cases, I think this is partly due to the
is_user_module() call in setupdwfl.cxx which assumes that userspace
modules must always be absolute paths.

  is_user_module(const std::string &m)
    return m[0] == '/' && m.rfind(".ko", m.length() - 1) != m.length() - 3;

That strikes me as an awfully bogus assumption; it should be possible
to specify a userspace module by bare-name in @var, @cast, etc not
just a kernel module.

But taking a step back: systemtap appears to expand functions once per
invocation in a probe anyway, so why doesn't the function benefit from
the focus_on_module() of the probe and work like case [0] without any
specification of module name at all?

Phew! Ideas?

 Craig Ringer         
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]