Unsuccessful attempts get aarch64 kernel backtraces support

[William Cohen <wcohen at redhat dot com>]

I found that the aarch64 kernel are always built with framepointers,
so unwinding the aarhc64 kernel stack should be trivial.  However, it
doesn't currenlty work in systemtap. The systemtap runtime appears to
be currently trying to use the dwarf unwinding and that isn't working
on aarch64.  The userspace dwarf-based backtrace DOES work:


[root@apm-mustang-ev3-01 ~]# more backtrace_test.c
#include <stdlib.h>

int c(){ return 0;}
int b(){return c();}
int a(){return b();}

int main(int argc, char* argv[])
{
  exit(a());
}
[root@apm-mustang-ev3-01 ~]# gcc -g -o backtrace_test backtrace_test.c 
[root@apm-mustang-ev3-01 ~]# ./systemtap_write/install/bin/stap -e 'probe process("./backtrace_test").function("c") {print_ubacktrace();}' -c ./backtrace_test
 0x4005c0 : c+0x0/0x8 [/root/backtrace_test]
 0x4005d4 : b+0xc/0x14 [/root/backtrace_test]
 0x4005e8 : a+0xc/0x14 [/root/backtrace_test]
 0x400604 : main+0x14/0x18 [/root/backtrace_test]
 0x3ffb037f68c [/usr/lib64/libc-2.22.so+0x1f68c/0x190000]
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/libc-2.22.so'


However, things do not work so well for the kernel backtraces on
aarch64 machine get the very limited:

[root@apm-mustang-ev3-01 ~]# ./systemtap_write/install/bin/stap -e 'probe kernel.function("vfs_write") {print_backtrace(); exit();}'
WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear search for /root/pratyush/linux/vmlinux
 0xfffffe00002397b8 : vfs_write+0x0/0x1c8 [kernel]
 0x0 (inexact)

Would expect something more like the following from x86-64:

$ stap -e 'probe kernel.function("vfs_write") {print_backtrace(); exit();}'

 0xffffffff81223e20 : vfs_write+0x0/0x1a0 [kernel]
 0xffffffff81224b85 : sys_write+0x55/0xc0 [kernel]
 0xffffffff8178162e : entry_SYSCALL_64_fastpath+0x12/0x71 [kernel]

Below are the sections for the locally built aarch64 kernel running on the system.

# eu-readelf -S vmlinux
There are 38 section headers, starting at offset 0x7ea3c28:

Section Headers:
[Nr] Name                 Type         Addr             Off      Size     ES Flags Lk Inf Al
[ 0]                      NULL         0000000000000000 00000000 00000000  0        0   0  0
[ 1] .head.text           PROGBITS     fffffe0000080000 00010000 00002000  0 AX     0   0 4096
[ 2] .text                PROGBITS     fffffe0000090000 00020000 00762770  0 AX     0   0 65536
[ 3] .text.init           PROGBITS     fffffe00007f2770 00782770 00000090  0 AX     0   0 16
[ 4] .init.init.text      PROGBITS     fffffe00007f2800 00782800 00000114  0 AX     0   0  4
[ 5] .rodata              PROGBITS     fffffe0000800000 00790000 002af540  0 A      0   0 4096
[ 6] .eh_frame            PROGBITS     fffffe0000aaf540 00a3f540 001a0fd8  0 A      0   0  8
[ 7] .init.rodata.str1.8  PROGBITS     fffffe0000c50518 00be0518 00000780  1 AMS    0   0  8
[ 8] .init.eh_frame       PROGBITS     fffffe0000c50c98 00be0c98 00001540  0 A      0   0  8
[ 9] __bug_table          PROGBITS     fffffe0000c521d8 00be21d8 000101dc  0 A      0   0  4
[10] .pci_fixup           PROGBITS     fffffe0000c623b8 00bf23b8 00002448  0 A      0   0  8
[11] __ksymtab            PROGBITS     fffffe0000c64800 00bf4800 0000f420  0 A      0   0  8
[12] __ksymtab_gpl        PROGBITS     fffffe0000c73c20 00c03c20 0000e920  0 A      0   0  8
[13] __ksymtab_strings    PROGBITS     fffffe0000c82540 00c12540 00023309  0 A      0   0  1
[14] __param              PROGBITS     fffffe0000ca5850 00c35850 00002490  0 A      0   0  8
[15] __modver             PROGBITS     fffffe0000ca7ce0 00c37ce0 00008320  0 A      0   0  8
[16] __ex_table           PROGBITS     fffffe0000cb0000 00c40000 00002bc0  0 A      0   0  8
[17] .notes               NOTE         fffffe0000cb2bc0 00c42bc0 00000024  0 A      0   0  4
[18] .init.text           PROGBITS     fffffe0000cc0000 00c50000 00046d98  0 AX     0   0 16
[19] .exit.text           PROGBITS     fffffe0000d06d98 00c96d98 000020e8  0 AX     0   0  4
[20] .init.data           PROGBITS     fffffe0000d08e80 00c98e80 0005a558  0 WA     0   0 32
[21] .data..percpu        PROGBITS     fffffe0000d70000 00d00000 00016380  0 WA     0   0 128
[22] .altinstructions     PROGBITS     fffffe0000d90000 00d20000 00006264  0 A      0   0  1
[23] .altinstr_replacement PROGBITS     fffffe0000d96264 00d26264 000020d4  0 AX     0   0  1
[24] .data                PROGBITS     fffffe0000da0000 00d30000 00132cf8  0 WA     0   0 65536
[25] .pecoff_edata_padding PROGBITS     fffffe0000ed2cf8 00e62cf8 00000108  0 WA     0   0  1
[26] .bss                 NOBITS       fffffe0000ee0000 00e62e00 001648f8  0 WA     0   0 65536
[27] .comment             PROGBITS     0000000000000000 00e62e00 0000002c  1 MS     0   0  1
[28] .debug_line          PROGBITS     0000000000000000 00e62e2c 0057068e  0        0   0  1
[29] .debug_info          PROGBITS     0000000000000000 013d34ba 0559c367  0        0   0  1
[30] .debug_abbrev        PROGBITS     0000000000000000 0696f821 0021c404  0        0   0  1
[31] .debug_aranges       PROGBITS     0000000000000000 06b8bc30 0001a100  0        0   0 16
[32] .debug_ranges        PROGBITS     0000000000000000 06ba5d30 00358250  0        0   0 16
[33] .debug_str           PROGBITS     0000000000000000 06efdf80 001faf8f  1 MS     0   0  1
[34] .debug_loc           PROGBITS     0000000000000000 070f8f0f 0090a008  0        0   0  1
[35] .shstrtab            STRTAB       0000000000000000 07a02f17 00000198  0        0   0  1
[36] .symtab              SYMTAB       0000000000000000 07a030b0 00327a50 24       37 113493  8
[37] .strtab              STRTAB       0000000000000000 07d2ab00 00179125  0        0   0  1

Turning on the unwind debug code it spews out a lot of messages, so used head to get the beginning:


[root@apm-mustang-ev3-01 systemtap]# ~/systemtap_write/install/bin/stap -DDEBUG_UNWIND=1 -e 'probe kernel.function("vfs_write") {print_backtrace(); exit();}' | head --lines=25
WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear search for /root/pratyush/linux/vmlinux
_stp_stack_unwind_one_kernel:265: STARTING kernel unwind
 0xfffffe00002397b8 : vfs_write+0x0/0x1c8 [kernel]
_stp_stack_unwind_one_kernel:300: CONTINUING kernel unwind to depth 1
unwind:1478: pc=fffffe00002397b8, fffffe00002397b8
unwind:1521: trying debug_frame
unwind_frame:1178: Module /root/pratyush/linux/vmlinux: no unwind frame data
unwind:1525: debug_frame failed: -5, trying eh_frame
unwind_frame:1194: /root/pratyush/linux/vmlinux: fde=0
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30

Suggestions on what to do or try next to get kernel backtraces to work on aarch64?

-Will