back trace of crack compiled with -fPIE -pie is broken see https://bugs.gentoo.org/show_bug.cgi?id=431678
Is it a regression, did gdb-7.4.1 work? I tried to build 'crack-language' but it failed on missing llvm despite configure succeeded, I did not investigate it more. For the comment there: # according to gdb it crashes at instruction "test %eax,%eax" # #10 0x0000038e3338d57f in operator< (other=..., this=<optimized out>) at # debug/DebugTools.cc:41 # # 0x0000038e3338d572 <+98>: 49 8b 74 24 20 mov 0x20(%r12),%rsi # 0x0000038e3338d577 <+103>: 48 89 ef mov %rbp,%rdi # 0x0000038e3338d57a <+106>: e8 f1 07 fe ff callq 0x38e3336dd70 <strcmp@plt> # 0x0000038e3338d57f <+111>: 85 c0 test %eax,%eax As this is frame #10 (and not frame #0) the address 0x0000038e3338d57f just points to the return address to the caller. That's correct, it probably crashed somewhere in 'strcmp' which will return here. # Is there any way how to unhide these functions? # #0 0x000003785c2cfff9 in ?? () # #1 0x000003785c2e68da in ?? () What is the output of 'info sharedlibrary' and what is really mapped to these addresses according to 'info proc mappings' (or 'cat /proc/PID/maps')?
It didn't work with 7.4.1 I will try to get better proc mappings. $ ./crack_dbg -- -l lib example/hello.crk GNU gdb (Gentoo 7.5 p1) 7.5 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". For bug reporting instructions, please see: <http://bugs.gentoo.org/>... Reading symbols from /mnt/md3/cache/inst/crack-language/.libs/crack...done. (gdb) r Starting program: /mnt/md3/cache/inst/crack-language/.libs/crack -l lib example/hello.crk [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x0000033b426b4ff9 in ?? () (gdb) bt #0 0x0000033b426b4ff9 in ?? () #1 0x0000033b426cb8da in ?? () #2 0x0000033b40184e60 in using_malloc_checking () from /lib64/libc.so.6 #3 0x000000000001ae50 in ?? () #4 0x00000012dc0be700 in ?? () #5 0x0000033b426be7d7 in ?? () #6 0x0000000000000020 in ?? () #7 0x0000033b3fe7a252 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:4065 #8 0x0000033b426c9830 in ?? () #9 0x000003b8b8095610 in ?? () #10 0x0000033b4177557f in operator< (other=..., this=<optimized out>) at debug/DebugTools.cc:41 #11 operator() (__y=..., __x=..., this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.4/include/g++-v4/bits/stl_function.h:236 #12 find (this=0x33b42536aa0 <(anonymous namespace)::internTable>, __k=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.4/include/g++-v4/bits/stl_tree.h:1539 #13 find (__x=..., this=0x33b42536aa0 <(anonymous namespace)::internTable>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.4/include/g++-v4/bits/stl_set.h:605 #14 (anonymous namespace)::lookUpString (key=...) at debug/DebugTools.cc:60 #15 0x0000033b41824710 in builder::mvll::LLVMJitBuilder::run (this=0x33b426c86a9) at builder/llvm/LLVMJitBuilder.cc:217 #16 0x0000033b41825257 in builder::mvll::LLVMJitBuilder::doRunOrDump (this=0x12dbfa20c0, context=...) at builder/llvm/LLVMJitBuilder.cc:375 #17 0x0000033b41826b73 in builder::mvll::LLVMJitBuilder::innerCloseModule (this=0x12dbfa20c0, context=..., moduleDef=0x12dc04cf90) at builder/llvm/LLVMJitBuilder.cc:360 #18 0x0000033b418273cd in recursiveClose (builder=<optimized out>, context=..., this=<optimized out>) at builder/llvm/BJitModuleDef.h:73 #19 closeOrDefer (builder=0x12dbfa20c0, context=..., this=0x12dc04cf90) at builder/llvm/BJitModuleDef.h:82 #20 builder::mvll::LLVMJitBuilder::closeModule (this=0x12dbfa20c0, context=..., moduleDef=<optimized out>) at builder/llvm/LLVMJitBuilder.cc:383 #21 0x0000033b417b3327 in model::ModuleDef::close (this=0x12dc04cf90, context=...) at model/ModuleDef.cc:48 #22 0x0000033b41784916 in model::Construct::parseModule (this=0x12dbbefdf0, context=..., module=0x12dc04cf90, path=..., src=...) at model/Construct.cc:397 #23 0x0000033b417872bc in model::Construct::loadModule (this=0x12dbbefdf0, moduleNameBegin="crack", moduleNameEnd=..., canonicalName=...) at model/Construct.cc:605 #24 0x0000033b417bf324 in parser::Parser::parseImportStmt (this=0x3b8b8096c30, ns=0x12dc04c178) at parser/Parser.cc:2679 #25 0x0000033b417cc39d in parser::Parser::parseStatement (this=0x3b8b8096c30, defsAllowed=true) at parser/Parser.cc:356 #26 0x0000033b417cc77b in parser::Parser::parseBlock (this=0x3b8b8096c30, nested=false, closeEvent=parser::Parser::noCallbacks) at parser/Parser.cc:485 #27 0x0000033b417cca36 in parser::Parser::parse (this=<optimized out>) at parser/Parser.cc:3396 #28 0x0000033b4178490b in model::Construct::parseModule (this=0x12dbbefdf0, context=..., module=0x12dc04c140, path=..., src=...) at model/Construct.cc:396 #29 0x0000033b4178559b in model::Construct::runScript (this=0x12dbbefdf0, src=..., name="example/hello.crk") at model/Construct.cc:782 #30 0x0000033b41872446 in Crack::runScript (this=0x3b8b8097240, src=..., name="example/hello.crk") at Crack.cc:108 #31 0x00000012db8260fa in main (argc=4, argv=0x3b8b8097688) at crack_main.cc:277 (gdb) info sharedlibrary From To Syms Read Shared Object Library 0x0000033b42540a80 0x0000033b42558d4a Yes /lib64/ld-linux-x86-64.so.2 0x0000033b4175db00 0x0000033b420f5618 Yes ./.libs/libCrackLang.so.3 0x0000033b412f97e0 0x0000033b412ff658 Yes /usr/lib64/libffi.so.6 0x0000033b41015720 0x0000033b410bcdd8 Yes (*) /usr/lib64/libasound.so.2 0x0000033b40de5df0 0x0000033b40de68c8 Yes /lib64/libdl.so.2 0x0000033b40bce6e0 0x0000033b40bda328 Yes /lib64/libpthread.so.0 0x0000033b4095bdd0 0x0000033b409b0fd8 Yes (*) /lib64/libpcre.so.1 0x0000033b406822e0 0x0000033b40715960 Yes /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6 0x0000033b403a4f00 0x0000033b403e2c48 Yes /lib64/libm.so.6 0x0000033b4018bea0 0x0000033b4019d1f8 Yes /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1 0x0000033b3fe1fce0 0x0000033b3ff3354c Yes /lib64/libc.so.6 0x0000033b3fbfa2f0 0x0000033b3fbfda78 Yes /lib64/librt.so.1 0x0000033b3f9d12f0 0x0000033b3f9efbe8 Yes (*) /usr/local/lib/crack-0.6/crack/runtime.so (*): Shared library is missing debugging information. (gdb) quit A debugging session is active. Inferior 1 [process 25491] will be killed. Quit anyway? (y or n) n Not confirmed. (gdb) info proc mappings process 25491 Mapped address spaces: Start Addr End Addr Size Offset objfile 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/crack 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/crack 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/crack 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 [heap] 0x0 0x0 0x0 0x0 /usr/local/lib64/crack-0.6/crack/runtime.so 0x0 0x0 0x0 0x0 /usr/local/lib64/crack-0.6/crack/runtime.so 0x0 0x0 0x0 0x0 /usr/local/lib64/crack-0.6/crack/runtime.so 0x0 0x0 0x0 0x0 /usr/local/lib64/crack-0.6/crack/runtime.so 0x0 0x0 0x0 0x0 /lib64/librt-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/librt-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/librt-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/librt-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libc-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libc-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libc-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libc-2.14.1.so 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1 0x0 0x0 0x0 0x0 /lib64/libm-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libm-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libm-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libm-2.14.1.so 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17 0x0 0x0 0x0 0x0 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 /lib64/libpcre.so.1.0.1 0x0 0x0 0x0 0x0 /lib64/libpcre.so.1.0.1 0x0 0x0 0x0 0x0 /lib64/libpcre.so.1.0.1 0x0 0x0 0x0 0x0 /lib64/libpcre.so.1.0.1 0x0 0x0 0x0 0x0 /lib64/libpthread-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libpthread-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libpthread-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libpthread-2.14.1.so 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 /lib64/libdl-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libdl-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libdl-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/libdl-2.14.1.so 0x0 0x0 0x0 0x0 /usr/lib64/libasound.so.2.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libasound.so.2.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libasound.so.2.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libasound.so.2.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libffi.so.6.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libffi.so.6.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libffi.so.6.0.0 0x0 0x0 0x0 0x0 /usr/lib64/libffi.so.6.0.0 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/libCrackLang.so.3.0.1 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/libCrackLang.so.3.0.1 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/libCrackLang.so.3.0.1 0x0 0x0 0x0 0x0 /mnt/md3/cache/inst/crack-language/.libs/libCrackLang.so.3.0.1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 /lib64/ld-2.14.1.so 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 [vdso] 0x0 0x0 0x0 0x0 /lib64/ld-2.14.1.so 0x0 0x0 0x0 0x0 /lib64/ld-2.14.1.so 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 [stack] 0xffffffffff600000 0xffffffffff601000 0x1000 0x0 [vsyscall]
'info proc mappings' output is really broken, please attach /proc/PID/maps as exact as possible (as Bugzilla attachment, not inlined into a Bugzilla comment like above). But this is unrelated bug. From the output above the address 0x0000033b426b4ff9 I guess is not mapped to any file in /proc/PID/maps. Highest address in 'info sharedlibrary' is: From To Shared Object Library 0x0000033b42540a80 0x0000033b42558d4a /lib64/ld-linux-x86-64.so.2 crash is: 0x0000033b426b4ff9 in ?? () I guess it is in some code runtime generated by the 'crack-language' project (or it could be [vdso]) so I ask also for (1) /proc/PID/maps for this case and then possible also (2) the same outputs for gdb-7.4.1 how it could find the symbols.
Created attachment 6591 [details] fixed info proc mappings
Created attachment 6592 [details] gdb 7.4.1
So gdb-7.4.1 also cannot find symbol for frame #0, it is not a regression. Matching 'mappings' entry is: 0x3fff7f4e000 0x3fff7fce000 0x80000 0x0 This confirms my suspection it is a JIT-generated code. 'crack-language' should provide a GDB JIT module to use via: (gdb) jit-reader-load some-future-jit-module-from-crack-language.so 'crack-language' is aware it does not support debugging: # Crack has only minimal support for debugging in 0.6. If your program # seg-faults or aborts, you can at least get a sparse stack-trace by running it # in JIT mode under a fairly recent version of GDB (7.0 or later). OK to make it RESOLVED-INVALID? This is 'crack-language' Bug, not GDB's. (BTW I would be interested why happened the 0x0 entries in 'info proc mappings' as this code has changes 7.4->7.5.)
Does strcmp called JIT generated core or it is just broken? 0x0 entries were caused by kernel.
When you have a frame with failed unwind (like the frame #0, as JIT code unwinding is not supported by 'crack-language') then anything can happen. Frames from #1 upwards are only a wild guess. Nobody had to call strcmp in this case, it could be leftover return address from some previous execution (preserved due to a stack buffer) etc.
OK