Bug 17638 - Symbol resolution broken for PPC64 ABIv2
Summary: Symbol resolution broken for PPC64 ABIv2
Alias: None
Product: systemtap
Classification: Unclassified
Component: runtime (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Unassigned
Depends on:
Blocks: 18154
  Show dependency treegraph
Reported: 2014-11-24 12:06 UTC by Ananth Mavinakayanahalli
Modified: 2015-04-22 13:49 UTC (History)
7 users (show)

See Also:
Last reconfirmed:


Note You need to log in before you can comment on or make changes to this bug.
Description Ananth Mavinakayanahalli 2014-11-24 12:06:44 UTC
PPC64 ABIv2 has a facility where a function can have a Global Entry Point (GEP) and a Local Entry Point (LEP); the LEP is at an offset from the GEP and can be gotten to programmatically from the Elf64_Sym.st_other field. See the ABIv2 specification for more details. See arch/powerpc/kernel/module_64.c and ppc_function_entry@arch/powerpc/include/asm/code-patching.h for more details.

Kprobes at function entry need probes installed at the LEP. The symbol table and DWARF entry_pc/high_pc encode only the GEP instead. This needs fixing.

stap -p3 -ve 'probe kernel.function("do_fork") { printf("Hello\n"); exit() }'
... resolves do_fork as:

} stap_dwarf_probes[] = {
  { .address=(unsigned long)0x283950ULL, .module="kernel", .section="_stext", .probe=(&stap_probes[0]), },

nnr:~/kprobes # grep do_fork /boot/System.map
c0000000000a6920 T do_fork

nnr:~/kprobes # stap -g -ve 'probe kernel.statement(0xc0000000000a6928).absolute {printf("here\n") exit()}'
Pass 1: parsed user script and 102 library script(s) using 40256virt/33408res/5504shr/27072data kb, in 280usr/0sys/285real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 41216virt/35584res/6656shr/28032data kb, in 10usr/0sys/8real ms.
Pass 3: translated to C into "/tmp/stapVEC1GH/stap_9c9d5bee16716aeedfab6219de368bd7_1067_src.c" using 41216virt/36032res/7104shr/28032data kb, in 10usr/0sys/9real ms.
Pass 4: compiled C into "stap_9c9d5bee16716aeedfab6219de368bd7_1067.ko" in 2160usr/100sys/3254real ms.
Pass 5: starting run.
Pass 5: run completed in 50usr/90sys/28905real ms.

Now, the real question is, where should this be fixed:
Dwarf encoding of entry_pc/high_pc?
Or in the end consumer tools like stap/perf?
Comment 1 Mark Wielaard 2014-11-24 12:41:42 UTC
The following email on the GDB mailinglist explains the issue nicely:

"In ELFv2, functions may provide both a global and a local entry point.
The global entry point (where the function symbol points to) is intended
to be used for function-pointer or cross-module (PLT) calls, and requires
r12 to be set up to the entry point address itself.   The local entry
point (which is found at a fixed offset after the global entry point,
as defined by bits in the symbol table entries' st_other field), instead
expects r2 to be set up to the current TOC.

Now, when setting a breakpoint on a function by name, you really want
that breakpoint to trigger either way, no matter whether the function
is called via its local or global entry point.  Since the global entry
point will always fall through into the local entry point, the way to
achieve that is to simply set the breakpoint at the local entry point."

The above also explain the correct way this should work. See if the address is in the symbol table, if it is, check the st_other field and adjust the probe address accordingly.

There doesn't seem to be a public copy of the Power Architecture 64-Bit ELF V2 ABI specification online that documents this though. So you'll have to figure the correct rules out from the patches posted to gdb/binutils.

We could also do a hack like ftrace in kernel does (which is horrible, but probably works in practice because the compiler/linker will only ever use on of two pairs of instructions anyway):

commit c71b7eff426fa7d8fd33e0964a7f79a3b41faff9
Author: Anton Blanchard <anton@samba.org>
Date:   Tue Feb 4 16:09:15 2014 +1100

    powerpc: Add ABIv2 support to ppc_function_entry
    Skip over the well known global entry point code for ABIv2.
    Signed-off-by: Anton Blanchard <anton@samba.org>


We might also want to add some magic to elfutils dwfl_module_getsym_info () to provide the local entry point as address associated with the symbol if different from the global entry point address (st_value).

Or we might want to adjust the DWARF address as output by GCC as suggested in the Description of this bug (but that won't work when the address comes directly from the symbol table).
Comment 2 Mark Wielaard 2014-11-24 13:11:31 UTC
There seems to be a public copy of the "Power Architecture 64-Bit ELF V2 ABI Specification (also OpenPOWER ABI for Linux Supplement)" distributed under the GNU Free Documentation License at: http://openpowerfoundation.org/technical/technical-resources/technical-specifications/

The usual cypherpunks@cypherpunks.com with password cypherpunks seems to work to access it.
Comment 3 Ulrich Weigand 2014-11-24 14:30:27 UTC
The DWARF address ranges deliberately include the *whole* function,
both global and local entry points.  GDB relies on that behavior.  (For
one, if you look up the function belonging to the current PC, you should
find the correct one even if you're before the local entry point.)

If you want to set probes on a local entry point, you should look up the
symbol in the main symbol table (not DWARF), and check the st_other bits;
they will indicate whether the function has a local entry point, and what
its offset from the global entry point is.  Note that GDB does the same
when setting a breakpoint on a function entry.

You can use the PPC64_LOCAL_ENTRY_OFFSET macro from the binutils header
files (elf/ppc64.h) to decode the st_other bits.  See also the GDB routines
ppc_elfv2_elf_make_msymbol_special and ppc_elfv2_skip_entrypoint (in the
GDB patch linked to above).

Anton's kernel patch refered to above instead implements code parsing
in an attempt to detect the typical global entry point sequence.  This
should work OK for now, but might cause problems in the future if the
sequence generated by GCC (or another compiler) ever changes.  Thus,
using the ELF symbol st_other bits is certainly the prefered way.
Comment 4 hemkshaw@in.ibm.com 2015-02-06 08:22:57 UTC
Proposed patch to fix this bug :
Comment 5 Mark Wielaard 2015-04-22 13:49:34 UTC
commit b4c6a4b1cd00fd1681d76de6850a88aa516c4fef
Author: Hemant Kumar <hemant@linux.vnet.ibm.com>
Date:   Mon Apr 20 15:59:24 2015 +0530

    Prioritize symbol table lookup for ppc64le
    PPC64 ELF ABI v2 has a Global entry point and a local entry point
    for the functions. We need the Local entry point in order to probe
    these functions. However, the DIE for these functions in debuginfo
    return the function.entrypc which is same as the global entry point.
    The local entry point is not encoded in the debuginfo of the ELFs. The
    offset to local entry point is however encoded in the st_other field
    of these symbols in the symbol table.
    We need to use this field to adjust the sym.st_value to actually point
    to the local entry point instead of the global entry point.
    This patch is in relation to this bug :
    So, while adding symbols to the sym_table, we add an offset of
    PPC64_LOCAL_ENTRY_OFFSET(sym.st_other) to st_value.
    And when the function address is queried in query_dwarf_func(), we give
    priority to the cached sym_table, where we can retrieve the adjusted
    entry address of the function. If we don't get any address from the
    symbol table, then we proceed to get from the debuginfo.
    Macro definition PPC64_LOCAL_ENTRY_OFFSET has been picked up from glibc.
    It won't be defined if we are building systemtap on a machine having
    older elf.h and hence, won't recognize PPC64_LOCAL_ENTRY_OFFSET.
    Signed-off-by: Hemant Kumar <hemant@linux.vnet.ibm.com>