Bug 21023 - The abidw tool does not appear to read dwarf from .dwp files associated with executables
Summary: The abidw tool does not appear to read dwarf from .dwp files associated with ...
Status: UNCONFIRMED
Alias: None
Product: libabigail
Classification: Unclassified
Component: default (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: dodji
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-01-04 18:15 UTC by andrew.c.morrow
Modified: 2018-12-17 01:39 UTC (History)
3 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description andrew.c.morrow 2017-01-04 18:15:26 UTC
$ abidw --version
1.0.rc7
I'm not actually sure if this is a bug in libabigail, or one of the supporting libraries like libelf or libdw, but I'm starting here, since the end state I want is that abidw works binaries built using DebugFission.

I would expect that abidw would search for DWARF info in the associated .dwp file, however, it does not appear to do so.

On my Ubuntu 16.04 machine:

$ g++ --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

$ dwp --version
GNU dwp (GNU Binutils for Ubuntu) 2.26.1

$ cat hello_world.cpp
#include <cstdlib>
#include <iostream>

int main(int argc, char* argv[]) {
    std::cout << "Hello, World!\n" << argc;
    return EXIT_SUCCESS;
}

$ g++ -g ./hello_world.cpp -o hello_world
$ abidw ./hello_world | cksum
1150852500 53174

$ g++ -gsplit-dwarf -g ./hello_world.cpp -o hello_world
$ dwp -e ./hello_world
$ abidw ./hello_world | cksum
2156308181 1319

As you can see from the sizes, running abidw against the DebugFission version results in severely truncated output. Running under strace appears to confirm that the dwp file was never stat'ed or read:

$ strace -f -o trace abidw ./hello_world | cksum
2156308181 1319
$ grep -c dwp trace
0
Comment 1 andrew.c.morrow 2017-02-01 18:32:57 UTC
A gentle ping on this issue?
Comment 2 Mark Wielaard 2017-02-01 19:16:56 UTC
This probably is something that elfutils should transparently support for libabigail. But currently elfutils libdw doesn't support GNU DebugFission dwp/split-dwarf. Parts of it are in the process of being standardized now for DWARF5. But the standard hasn't been finalized and code for elfutils is not yet there (and after DWARF5 public beta was released some details were changed).

For now I would recommend not using -gsplit-dwarf and dwp. If you are looking for smaller or "split" DWARF then yse using eu-strip -o and dwz to create debug multi-files. That is supported.
Comment 3 andrew.c.morrow 2017-02-04 14:54:59 UTC
Hi Mark - 

Thank you for clarifying. It sounds from your description like this will eventually work automatically, which is great.

My actual goal for using DebugFission was more for the build time improvements, rather than debug symbol packaging. My use of libabigail/abidw is somewhat unusual, and also focused on build time improvements:

https://sourceware.org/bugzilla/show_bug.cgi?id=18838#c4
http://scons-users.scons.narkive.com/E9RiycaY/replacement-for-sharedlibrarysignatureoverride

I had hoped to be able to use both DebugFission and abidw driven re-link pruning simultaneously, but it appears that this will not work, for the moment.

I have experimented with dwz as well, with the aim of reducing the size of our debug symbol packages, but so far without success: https://bugzilla.redhat.com/show_bug.cgi?id=1399866. If you have any additional thoughts on that ticket it would be greatly appreciated. I still don't have an explanation for why dwz cannot process the output of ld.gold.

I'm also unclear on the intended relationship between dwp and dwz. When I tried running dwz on dwp files, my recollection is that it didn't work, at all. I'd presume that the process would be to collect the .dwo files associated with each binary into a .dwp file for that binary with the dwp tool, and then run dwz across all the dwp files to extract common symbols into a shared file, along with any other optimization/compression. I would then hope that abidw run on the binary would be able to present the full ABI, extracting information from both the associated .dwp file, and any common underling file produced by dwz.

Or, perhaps the functionality of dwz should be folded directly into dwp? The documentation on these tools and the intended direction for development is somewhat unclear to me.
Comment 4 Mark Wielaard 2017-02-08 14:16:53 UTC
(In reply to andrew.c.morrow from comment #3)
> Thank you for clarifying. It sounds from your description like this will
> eventually work automatically, which is great.

It will however take some work, it is not really trivial to support. Also while GNU DebugFission is being standardized in the upcoming DWARFv5 there have been various changes. So it isn't clear yet which tools will support which version.

> I have experimented with dwz as well, with the aim of reducing the size of
> our debug symbol packages, but so far without success:
> https://bugzilla.redhat.com/show_bug.cgi?id=1399866. If you have any
> additional thoughts on that ticket it would be greatly appreciated. I still
> don't have an explanation for why dwz cannot process the output of ld.gold.

I commented on that bug report. It looks like the linker is creating a bogus .bss section, but there is not really enough information in the bug report to know for sure.

> I'm also unclear on the intended relationship between dwp and dwz. When I
> tried running dwz on dwp files, my recollection is that it didn't work, at
> all. I'd presume that the process would be to collect the .dwo files
> associated with each binary into a .dwp file for that binary with the dwp
> tool, and then run dwz across all the dwp files to extract common symbols
> into a shared file, along with any other optimization/compression. I would
> then hope that abidw run on the binary would be able to present the full
> ABI, extracting information from both the associated .dwp file, and any
> common underling file produced by dwz.
> 
> Or, perhaps the functionality of dwz should be folded directly into dwp? The
> documentation on these tools and the intended direction for development is
> somewhat unclear to me.

The have been developed completely independently and I don't think anybody has really thought about how to combine them. They also serve somewhat different purposes. And not many tools support both DWARF debuginfo data/file variants.
Comment 5 andrew.c.morrow 2017-10-13 15:33:47 UTC
This recently ended up on my radar again. Is there a ticket tracker for elfutils where I could file an issue, or an already filed issue I should watch? My understanding is that DWARF5 is now an issued standard, and it would be great to see tools like abidw which are built above elfutils be able to reach out to .dwo or .dwp files when -gsplit-dwarf (or its DWARF5 equivalent) is in play.
Comment 6 Mark Wielaard 2018-06-05 13:44:05 UTC
elfutils 0.171 was recently released with support for DWARF5 and split-dwarf (including GNU DebugFission as extension to older DWARF).

https://sourceware.org/ml/elfutils-devel/2018-q2/msg00141.html

For now .dwo files are supported, but not yet .dwp files.
(Also the binutils dwp tools doesn't work with DWARF5 split dwarf.)
It will take a bit more programming to also add .dwp support, but hopefully the structure of the elfutils code is now so that this isn't major surgery.

Although most things should work as is, libabigail will need to use a few new interfaces to properly handle split dwarf, iterate through CUs using the new dwarf_get_units, and change from using offsets and dwarf_dieoffset/dwarf_offdie to using the new dwarf_die_addr_die interface (because DIEs can now come from multiple files, so using offsets alone isn't enough to identify a DIE).

While we update libabigail for this could you try out the new elfutils-0.171 tools to see if they work correctly in your setup?

And could you indicate how important .dwp support is? Would a libabigail that just works with .dwo files be enough for now?

The interfaces should be generic enough that when .dwp support is added to elfutils libdw, then they should work as is and libabigail shouldn't really care whether the split dwarf units come from the .dwo files or the .dwp package file.
Comment 7 andrew.c.morrow 2018-06-16 16:02:41 UTC
(In reply to Mark Wielaard from comment #6)
> elfutils 0.171 was recently released with support for DWARF5 and split-dwarf
> (including GNU DebugFission as extension to older DWARF).
> 
> https://sourceware.org/ml/elfutils-devel/2018-q2/msg00141.html

That is great news, thanks.

> 
> For now .dwo files are supported, but not yet .dwp files.
> (Also the binutils dwp tools doesn't work with DWARF5 split dwarf.)
> It will take a bit more programming to also add .dwp support, but hopefully
> the structure of the elfutils code is now so that this isn't major surgery.
>

Support for .dwo files is currently more important for me than .dwp.

 
> Although most things should work as is, libabigail will need to use a few
> new interfaces to properly handle split dwarf, iterate through CUs using the
> new dwarf_get_units, and change from using offsets and
> dwarf_dieoffset/dwarf_offdie to using the new dwarf_die_addr_die interface
> (because DIEs can now come from multiple files, so using offsets alone isn't
> enough to identify a DIE).
> 
> While we update libabigail for this could you try out the new elfutils-0.171
> tools to see if they work correctly in your setup?

A quick and dirty test suggests that it is working:

$ g++ -g -gsplit-dwarf -c -o hello_world.o ./hello_world.cpp -std=c++14
$ g++ -o hello_world hello_world.o
$ strace -o trace ~/opt/bin/eu-readelf --debug-dump=info+ --all ./hello_world
$ grep dwo trace
write(1, "           GNU_dwo_name         "..., 57) = 57
write(1, "           GNU_dwo_id           "..., 59) = 59
openat(AT_FDCWD, "/home/andrew/hello_world.dwo", O_RDONLY) = 4
readlink("/proc/27201/fd/4", "/home/andrew/hello_world.dwo", 4095) = 28
lstat("/home/andrew/hello_world.dwo", {st_mode=S_IFREG|0664, st_size=21696, ...}) = 0
write(1, "           GNU_dwo_id           "..., 59) = 59 

More complete testing will need to wait on the libabigail support.

> 
> And could you indicate how important .dwp support is? Would a libabigail
> that just works with .dwo files be enough for now?

I can definitely make use of a libabigail that only supports .dwo, perhaps indefinitely. Thanks!