Created attachment 14115 [details] pstack output captured when gdb was stuck We recently migrated our projects to gcc 11.2 and it has dwarf 5 as default debugging format (unsure if it's consequential). And I notice that gdb is not liking it, debugging 11.2 compiled binary with gdb 11.2 is pathetically slow and sometimes even backtrace takes 3 mins and memory consumption of gdb process also goes up significantly. I then tried old gdb 8.3 and I still have same problem I dumped pstack on gdb process (with gdb 8.3) when it was churning wheels doing backtrace and breakpoint completion and that's the only information I can share right now. Debugging in general is painstakingly slow and almost impractical. I did try to compile with dwarf4 format but it doesn't seem to help with the gdb performance. Are there any suggestions/workarounds to deal with this issue? Let me know if any other information is needed. Binary was compiled with -g3 option.
By the looks of you backtraces (GDB stuck in processing macro information), I am guessing that it's the same as: https://sourceware.org/bugzilla/show_bug.cgi?id=27754 If you run "readelf --debug-dump=macro" on your binary, do you get a ton of DW_MACRO_import - offset : 0x0 ? If so it's a problem in the compiler or linker. It has apparently been fixed (you can follow the paper trail on that bug), but I don't really understand the details on that side.
Thanks for your suggestion but no there isn't even a single occurrence of it. % readelf --debug-dump=macro <mybinary> %
(In reply to gusain from comment #2) > Thanks for your suggestion but no there isn't even a single occurrence of it. > > % readelf --debug-dump=macro <mybinary> > % This is surprising given that you said you compiled with -g3. And your backtraces show GDB reading some macro information. It could also be in a shared library, though.
I did check shared libraries and I couldn't find it there either. Is there any other way to confirm it's the same issue? If there is more information needed to narrow it further, can you let me know ? Also, I checked other ticket and I have two comments 1. LTO is not in picture here 2. I have tried lld and gold as well but none seems to fix the issue.
Well, your backtraces do show GDB reading macro information, so there's gotta be some somewhere. Actually, you can turn on: (gdb) set debug dwarf-read 1 prior to running your experiment. You should get some messages like this, that will help you know which CU in which file GDB is slow at reading: [dwarf-read] process_queue: Expanding one or more symtabs of objfile /home/simark/build/binutils-gdb-one-target/gdb/gdb ... [dwarf-read] process_queue: Expanding symtab of CU at offset 0x1054
I tend to think this is a dup as well. The way gcc emits the macro info also triggered a problem with mold. Maybe gdb should just ignore imports of 0x0 as a workaround.
(In reply to Tom Tromey from comment #6) > I tend to think this is a dup as well. > > The way gcc emits the macro info also triggered a problem with mold. > Maybe gdb should just ignore imports of 0x0 as a workaround. I'm just worried that there could be legitimate cases of "import 0".
(In reply to Simon Marchi from comment #5) > Well, your backtraces do show GDB reading macro information, so there's > gotta be some somewhere. > > Actually, you can turn on: > > (gdb) set debug dwarf-read 1 > > prior to running your experiment. You should get some messages like this, > that will help you know which CU in which file GDB is slow at reading: > > [dwarf-read] process_queue: Expanding one or more symtabs of objfile > /home/simark/build/binutils-gdb-one-target/gdb/gdb ... > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x1054 Thanks, and yes now it prints and appears to be stuck when I try breakpoint completion by hitting tab after name hint [dwarf-read] process_queue: Expanding symtab of CU at offset 0x5d2cd20 [dwarf-read] process_queue: Done expanding CU at offset 0x5d2cd20 [dwarf-read] process_queue: Done expanding symtabs of <binary>. [dwarf-read] process_queue: Expanding one or more symtabs of objfile <binary> ... [dwarf-read] process_queue: Expanding symtab of CU at offset 0x3644296 [dwarf-read] process_queue: Done expanding CU at offset 0x3644296 [dwarf-read] process_queue: Done expanding symtabs of <binary>. [dwarf-read] process_queue: Expanding one or more symtabs of objfile <binary> ... [dwarf-read] process_queue: Expanding symtab of CU at offset 0x2cc08bb [dwarf-read] process_queue: Done expanding CU at offset 0x2cc08bb [dwarf-read] process_queue: Done expanding symtabs of <binary>. [dwarf-read] process_queue: Expanding one or more symtabs of objfile <binary> ... [dwarf-read] process_queue: Expanding symtab of CU at offset 0x18f13e [dwarf-read] process_queue: Done expanding CU at offset 0x18f13e [dwarf-read] process_queue: Done expanding symtabs of <binary>.
(In reply to gusain from comment #8) > (In reply to Simon Marchi from comment #5) > > Well, your backtraces do show GDB reading macro information, so there's > > gotta be some somewhere. > > > > Actually, you can turn on: > > > > (gdb) set debug dwarf-read 1 > > > > prior to running your experiment. You should get some messages like this, > > that will help you know which CU in which file GDB is slow at reading: > > > > [dwarf-read] process_queue: Expanding one or more symtabs of objfile > > /home/simark/build/binutils-gdb-one-target/gdb/gdb ... > > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x1054 > > > Thanks, and yes now it prints and appears to be stuck when I try breakpoint > completion by hitting tab after name hint > > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x5d2cd20 > [dwarf-read] process_queue: Done expanding CU at offset 0x5d2cd20 > [dwarf-read] process_queue: Done expanding symtabs of <binary>. > [dwarf-read] process_queue: Expanding one or more symtabs of objfile > <binary> ... > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x3644296 > [dwarf-read] process_queue: Done expanding CU at offset 0x3644296 > [dwarf-read] process_queue: Done expanding symtabs of <binary>. > [dwarf-read] process_queue: Expanding one or more symtabs of objfile > <binary> ... > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x2cc08bb > [dwarf-read] process_queue: Done expanding CU at offset 0x2cc08bb > [dwarf-read] process_queue: Done expanding symtabs of <binary>. > [dwarf-read] process_queue: Expanding one or more symtabs of objfile > <binary> ... > [dwarf-read] process_queue: Expanding symtab of CU at offset 0x18f13e > [dwarf-read] process_queue: Done expanding CU at offset 0x18f13e > [dwarf-read] process_queue: Done expanding symtabs of <binary>. Is there any other information I can get to confirm it's same as other issue you mentioned ?
(In reply to gusain from comment #9) > Is there any other information I can get to confirm it's same as other issue > you mentioned ? Try something like: readelf --debug-dump=macro | grep 'import.* 0x0$' See bug #27754
Thanks for your help. It does seem like it's similar to 27754 but let me share few observations % readelf --debug-dump=macro ld-g | grep 'import.*0x0$' | wc -l 277652 % readelf --debug-dump=macro lld-g | grep 'import.*0x0$' | wc -l 277655 % readelf --debug-dump=macro gold-g | grep 'import.*0x0$' | wc -l 0 You had a comment on other ticket that it doesn't seem to affect lld but my binary linked with ldd doesn't suggest it. With gold, I have 0 occurrences but binary is no good because when I bring it up in gdb, source information seems to be incorrect. When I put a breakpoint in my code, gdb tells me that it's in some libstdc++ file. Do you still think it's not gdb bug ? If not, whose bug is it ? Are you aware of any workaround (gold link doesn't work for me and I have other issues with it as I mentioned before) ? Do you think it's gold linker bug and I should file a ticket for it ? Is there information I can get which can shed more light on this gold-linked binary issue ? (In reply to Tom Tromey from comment #10) > (In reply to gusain from comment #9) > > > Is there any other information I can get to confirm it's same as other issue > > you mentioned ? > > Try something like: > > readelf --debug-dump=macro | grep 'import.* 0x0$' > > See bug #27754
(In reply to gusain from comment #11) > With gold, I have 0 occurrences but binary is no good because when I bring > it up in gdb, source information seems to be incorrect. When I put a > breakpoint in my code, gdb tells me that it's in some libstdc++ file. This sounds like a different problem, which we'd have to investigate separately. > Do you still think it's not gdb bug ? If not, whose bug is it ? I think the mold author put in a workaround for this issue. While pedantically it may be a gcc bug, in practice the fixes are in the linker. > Are you aware of any workaround (gold link doesn't work for me and I have > other issues with it as I mentioned before) ? GNU ld should work, or an updated mold. I'm unsure about lld. > Do you think it's gold linker bug and I should file a ticket for it ? Is > there information I can get which can shed more light on this gold-linked > binary issue ? I'm afraid I don't know the answer here. Since I think this particular bug is another instance of the macro-import problem, I'm closing it as a dup. *** This bug has been marked as a duplicate of bug 27754 ***