Re: Re: gdb and fission scheme (gcc: -gsplit-dwarf, gnu ld gold :--gdb-index )

[Doug Evans <dje at google dot com>]

On Sat, Sep 21, 2013 at 7:29 PM, ISHIKAWA,chiaki <ishikawa@yk.rim.or.jp> wrote:
> I have investigated a little further.
>
> Now it has dawned on me that using "-gsplit-dwarf" binaries for
> debugigng is not a simple plug-and-play in contrast to ordinary dwarf
> binaries where debug information for a source module is in the single
> .o file and linker consolidates the information into a single final
> binary.

Hi.
You surmise correctly. :)

> After looking at this error message line below, I have come to the
> conclusion that gdb is complaining that
>  - it has found a CU dwarf section (or whatever),
>  - this CU section refers to a DWO section (or file) with ID
>    77955146488e868b,
>  - but it has no knowledge of such DWO section and its whereabout,
>    thus "unknown DWO".

Essentially that's correct.

DWOs are referred to by ID, and this ID is contained in the binary.
Accompanying the ID is the path to find the DWO file with this ID.

You can see the details if you do "readelf -wi binary".
["binary" here can also be "shared library"]

E.g.

bash$ readelf -wi hello
[...]
 <0><10e>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <10f>   DW_AT_low_pc      : 0x400710
    <117>   DW_AT_high_pc     : 0x15
    <11f>   DW_AT_stmt_list   : 0xa2
    <123>   DW_AT_GNU_dwo_name: (indirect string, offset: 0xee): hello.dwo
    <127>   DW_AT_comp_dir    : (indirect string, offset: 0xca):
/home/dje/src/play
    <12b>   DW_AT_GNU_pubnames: 1
    <12b>   DW_AT_GNU_addr_base: 0x0
    <12f>   DW_AT_GNU_dwo_id  : 0x92ea2b0b62f0cbd3

The DWO ID is in the DW_AT_GNU_dwo_id attribute.
The DWO name is in the DW_AT_GNU_dwo_name attribute.

Also note DW_AT_comp_dir.  That is the compilation directory, and is
where gdb will look for the DWO file by default (but one can provide
gdb other places to look for as well).

>
> --- quote ---
>
> Reading symbols from
> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...Dwarf Error: CU at
> offset 0x0 references
> unknown DWO with ID 77955146488e868b [in module
> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
> (no debugging symbols found)...done.
> --- end quote ---
>
> So I think I need to tell that relevant .dwo files are in the object
> directory of thunderbird.
>
> Now that debug information is split, I have to do this. OK, fair enough.
>
> But how can I tell gdb that all the .dwo files and dynamically linked
> .so files are under certain directories?  Do I have to tell gdb.
> Also, do I have to tell gdb in advance, all the *.dwo files that are
> referenced by thunderbird, and
> how?

You don't have to tell gdb what all the dwo files are, but if they're
not findable from DW_AT_comp_dir, then you need to tell gdb where to
look.
This is done by adding the path to the "debug-file-directory".
E.g.
(gdb) set debug-file-directory /path/to/my/dwos

Depending on how the program was built, this may involve adding one
path to debug-file-directory or several.
[The goal here is to provide an updated value of DW_AT_comp_dir:
DW_AT_GNU_dwo_name contains the path name from there.]

btw, one can "package" all the DWO files into one file, a DWP file.
That is done with the dwp program that comes with binutils.
[dwp is like a linker for dwo files]
Use this program like you would the linker:
bash$ dwp -o my-prog.dwp dwo1.o dwo2.o ...
Note that here you *do* need to know about all the dwo files, but it
really is just akin to linking.


HTH.
[And let me know if it doesn't.]

>
> After reading through gdb documentation and not finding much about
> split dwarf feature itself , I now realize that it looks to me that I have
> to
>
> - create file location dwarf information for gdb to locate
>   information in scattered *.dwo files,
>
>  B ut again, how?
>
> - One possibility: this is done by using objcopy by extracting certain debug
>   information from ALL *.dwo files and merging into a certain dwarf
>   data structure (list of known DWO sections with its unique IDs and
>   the file pathname?),
>
> - AND maybe physically attaching this merged dwarf information to
>   |thunderbird| and |thunderbird-bin| binary.  Basically extracting all
>   the sections of |thunderbird| and merge them with the list of known
>   DWO sections with the IDs and file pathnames.) and creating a new
>   binary.
>
> - This recreated binary will be used so that gdb will notice all the
>   indirect references to the debug information in all *.dwo files when
>   |thunderbird| and |thunderbird-bin| is loaded?
>
> Is this what one is supposed to do when -gsplit-dwarf is used
> and gdb debugging is attempted?
>
> Oh, I now realize another possibility of creating consolidated
> symbolic information for gdb debugging.
> maybe I can read all the *.dwo files one by one from within gdb
> (of course, I will use gdb script to read them in one sweep),
> and write out the symbolic information for next time by |save gdb-index|
> This may be the way to go, but not sure
>
> I appreciate any helpful tips and guidance.
>
> TIA
>
>
> cf. *.dwo files and *.so files created and used
> by mozilla thunderbird
>
> I have re-compiled the tree under /REF-COMM-CENTRAL/comm-central
> using the ordinary dwarf debug information (without -gsplit-dwarf and
> -Wl,--gdb-index)
> so I am referring to a different tree which was compiled using
> -gsplit-dwarf.
> The top-most directory of the source file hierarchy is
> /COMM-CENTRAL/comm-central, and
> the top-most directory of the object file hierarchy is
> /TEST-MAIL-OBJ/objdir-tb3.
>
> The final thunderbird binary is in the following directory.
> /TEST-MAIL-DIR/objdir-tb3 is the top of the object directory.
>
> There are many *.dwo files scattered in the object tree.
> (The structure of the object tree reflects the original source tree more or
> less.)
> The accompanying *.o files are under the same tree.
>
> Also, there are *.so files scattered in the object tree. They are
> consolidated into several directories and the number of directories were
> smaller than in the case of
> *.dwo files.
>
>
> ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ ls -l
> mozilla/dist/bin/thunderbird*
> -rwxr-xr-x 1 ishikawa ishikawa 267277 Sep 20 12:19
> mozilla/dist/bin/thunderbird*
> -rwxr-xr-x 1 ishikawa ishikawa 267277 Sep 20 08:42
> mozilla/dist/bin/thunderbird-bin*
>
> [*.dwo files]
>
> ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ find . -name "*.dwo"
> -print
> ./ldap/xpcom/src/nsLDAPProtocolModule.dwo
>         [..omission..]
> ./ldap/xpcom/src/nsLDAPServer.dwo
> ./db/mork/src/morkStore.dwo
> ./db/mork/src/morkPortTableCursor.dwo
>         [...omission...]
> ./db/mork/src/morkWriter.dwo
> ./db/mork/build/nsMorkFactory.dwo
> ./mozilla/parser/htmlparser/src/nsHTMLEntities.dwo
> ./mozilla/parser/htmlparser/src/nsHTMLTags.dwo
>         [...omission...]
> ./mozilla/parser/htmlparser/src/nsExpatDriver.dwo
> ./mozilla/parser/xml/src/nsSAXXMLReader.dwo
> ./mozilla/parser/xml/src/nsSAXLocator.dwo
> ./mozilla/parser/xml/src/nsSAXAttributes.dwo
> ./mozilla/parser/expat/lib/xmlparse.dwo
> ./mozilla/parser/expat/lib/xmltok.dwo
> ./mozilla/parser/expat/lib/xmlrole.dwo
> ./mozilla/parser/html/nsHtml5TreeBuilder.dwo
>         [...omission...]
> ./mozilla/parser/html/nsHtml5Parser.dwo
> ./mozilla/xpcom/reflect/xptcall/src/xptcall.dwo
>         [...omission...]
> ./mozilla/xpcom/reflect/xptinfo/src/xptiWorkingSet.dwo
> ./mozilla/xpcom/io/Base64.dwo
>         [...omission...]
> ./mozilla/xpcom/io/nsAppFileLocationProvider.dwo
>
>         [... lots of lines omitted.]
> ./mail/components/shell/DirectoryProvider.dwo
>
> [*.so files]
>
> ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ find . -name "*.so"
> -print
> ./ldap/sdks/c-sdk/ldap/libraries/libldif/libldif60.so
> ./ldap/sdks/c-sdk/ldap/libraries/libprldap/libprldap60.so
> ./ldap/sdks/c-sdk/ldap/libraries/libldap/libldap60.so
> ./mozilla/xpcom/tests/component_no_aslr/libtestcompnoaslr.so
> ./mozilla/xpcom/tests/component/libtestcomponent.so
> ./mozilla/xpcom/tests/bug656331_component/libtest656331.so
> ./mozilla/dist/plugins/libnptest.so
> ./mozilla/dist/plugins/libnpsecondtest.so
> ./mozilla/dist/bin/libmozalloc.so       <--- this is the directory where
> ./mozilla/dist/bin/libnspr4.so          <--- the main binary is stored.
> ./mozilla/dist/bin/libfreebl3.so        <--- *.so files are symlinks to
> other files under the
> ./mozilla/dist/bin/libssl3.so           <--- top-most objdir-tb3
> ./mozilla/dist/bin/libnssdbm3.so
> ./mozilla/dist/bin/libplds4.so
> ./mozilla/dist/bin/libsmime3.so
> ./mozilla/dist/bin/libmozsqlite3.so
> ./mozilla/dist/bin/libnssutil3.so
> ./mozilla/dist/bin/libxul.so            <--- a symlink libxul.so ->
> ../../toolkit/library/libxul.so
> ./mozilla/dist/bin/libsoftokn3.so
> ./mozilla/dist/bin/libprldap60.so
> ./mozilla/dist/bin/libldap60.so
> ./mozilla/dist/bin/libnssckbi.so
> ./mozilla/dist/bin/libldif60.so
> ./mozilla/dist/bin/libnss3.so
> ./mozilla/dist/bin/libplc4.so
> ./mozilla/dist/bin/libnpsecondtest.so
> ./mozilla/dist/bin/components/libdbusservice.so
> ./mozilla/dist/bin/components/libmozgnome.so
> ./mozilla/dist/lib/libmozalloc.so
> ./mozilla/dist/lib/libnspr4.so
> ./mozilla/dist/lib/libfreebl3.so
> ./mozilla/dist/lib/libssl3.so
> ./mozilla/dist/lib/libnssdbm3.so
> ./mozilla/dist/lib/libplds4.so
> ./mozilla/dist/lib/libsmime3.so
> ./mozilla/dist/lib/libmozsqlite3.so
> ./mozilla/dist/lib/libnssutil3.so
> ./mozilla/dist/lib/libxul.so            <---  symlink to  libxul.so ->
> ../../toolkit/library/libxul.so
> ./mozilla/dist/lib/libsoftokn3.so
> ./mozilla/dist/lib/libprldap60.so
> ./mozilla/dist/lib/libldap60.so
> ./mozilla/dist/lib/libnssckbi.so
> ./mozilla/dist/lib/libldif60.so
> ./mozilla/dist/lib/libnss3.so
> ./mozilla/dist/lib/libplc4.so
> ./mozilla/dist/lib/libnpsecondtest.so
> ./mozilla/dist/sdk/lib/libmozalloc.so
> ./mozilla/dist/sdk/lib/libnspr4.so
> ./mozilla/dist/sdk/lib/libssl3.so
> ./mozilla/dist/sdk/lib/libplds4.so
> ./mozilla/dist/sdk/lib/libsmime3.so
> ./mozilla/dist/sdk/lib/libnssutil3.so
> ./mozilla/dist/sdk/lib/libxul.so        <--- libxul.so ->
> ../../../toolkit/library/libxul.so
> ./mozilla/dist/sdk/lib/libnss3.so
> ./mozilla/dist/sdk/lib/libplc4.so
> ./mozilla/dom/plugins/test/testplugin/secondplugin/libnpsecondtest.so
> ./mozilla/dom/plugins/test/testplugin/libnptest.so
> ./mozilla/security/nss/lib/util/libnssutil3.so
> ./mozilla/security/nss/lib/ckfw/builtins/libnssckbi.so
> ./mozilla/security/nss/lib/smime/libsmime3.so
> ./mozilla/security/nss/lib/softoken/legacydb/libnssdbm3.so
> ./mozilla/security/nss/lib/softoken/libsoftokn3.so
> ./mozilla/security/nss/lib/freebl/libfreebl3.so
> ./mozilla/security/nss/lib/nss/libnss3.so
> ./mozilla/security/nss/lib/ssl/libssl3.so
> ./mozilla/build/unix/elfhack/test-array.so
> ./mozilla/build/unix/elfhack/test-ctors.so
> ./mozilla/nsprpub/pr/src/libnspr4.so
> ./mozilla/nsprpub/lib/libc/src/libplc4.so
> ./mozilla/nsprpub/lib/ds/libplds4.so
> ./mozilla/memory/mozalloc/libmozalloc.so
> ./mozilla/toolkit/system/dbus/libdbusservice.so
> ./mozilla/toolkit/system/gnome/libmozgnome.so
> ./mozilla/toolkit/crashreporter/test/libtestcrasher.so
> ./mozilla/toolkit/library/libxul.so    <--- libary file : size is 283MiB
>
>        This is acccompanied by libxul.so-gdb.py in the same directory
>         """ GDB Python customization auto-loader for libxul """
>
>         import os.path
>         sys.path[0:0] = [os.path.join('/COMM-CENTRAL/comm-central/mozilla',
> 'js', 'src', 'gdb')]
>
>         import mozilla.autoload
>         mozilla.autoload.register(gdb.current_objfile())
>
>
>
>
> ./mozilla/toolkit/components/ctypes/tests/libjsctypes-test.so
> ./mozilla/db/sqlite3/src/libmozsqlite3.so
> ./mozilla/js/xpconnect/tests/components/native/libxpctest.so
> ./mozilla/_tests/testing/mochitest/chrome/libraries/libjsctypes-test.so
> ./mozilla/_tests/testing/mochitest/chrome/toolkit/components/toolkit/components/ctypes/tests/libjsctypes-test.so
> ./mozilla/_tests/mozmill-virtualenv/lib/python2.7/site-packages/simplejson/_speedups.so
> ./mozilla/_tests/mozmill/mozmillprofile/plugins/libnptest.so
> ./mozilla/_tests/mozmill/mozmillprofile/plugins/libnpsecondtest.so
> ./mozilla/_tests/xpcshell/xpcom/tests/unit/libtestcompnoaslr.so
> ./mozilla/_tests/xpcshell/xpcom/tests/unit/libtestcomponent.so
> ./mozilla/_tests/xpcshell/xpcom/tests/unit/libtest656331.so
> ./mozilla/_tests/xpcshell/toolkit/crashreporter/test/unit/libtestcrasher.so
> ./mozilla/_tests/xpcshell/toolkit/crashreporter/test/unit_ipc/libtestcrasher.so
> ./mozilla/_tests/xpcshell/toolkit/components/ctypes/tests/unit/libjsctypes-test.so
> ./mozilla/_tests/xpcshell/js/xpconnect/tests/components/native/libxpctest.so
> ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$
>
>
>
> (2013/09/21 6:43), ISHIKAWA,chiaki wrote:
>>
>> I think the following problem is related to the reading of dynamically
>> linked libxul.so library file.
>> The problem affects the behavior of gdb in that it fails to access the
>> source files.
>> So it can not list the source lines.
>>
>> I checked the operation of gdb against statically linked small program and
>> all is well.
>> It can list the source files, and this is the first thing I immediately
>> noticed.
>>
>> With the setup described in the previous post of mine,
>> gdb can print the stack trace (sans the argument values), but
>> it can't print source files. Since the argument values are not shown in
>> the
>> stack trace. Its value is also diminished.
>>
>> Help is appreciated.
>> Yes, libxul.so is rather large. Thunderbird is a large program and so
>> we may hit into an unknown bug here.
>>
>>
>>
>> (2013/09/20 14:24), ishikawa wrote:
>>>
>>> Hi,
>>>
>>> I recently learned of "-gsplit-dwarf" of GCC 4.8 and -Wl,--gdb-index
>>> that is passed to GNU ld.gold linker, and have tried to use the split
>>> debug information scheme for local compilation and link of mozilla
>>> thunderbird mail client.
>>> ( http://gcc.gnu.org/wiki/DebugFission )
>>>
>>> The compiling and linking produced working thunderbird binary.  The
>>> time required for whole build process, especially the time necessary
>>> for creating a large libxul.so file diminished dramatically.  I would
>>> like to thank all the developers who have made this possible.
>>>
>>> There is a fly in the ointment, though.  Unfortunately, somehow gdb is
>>> not reading the symbol information from a created shared library,
>>> libxul.so, correctly.  Below is the beginning of a gdb session of
>>> trying to debug a crash of thunderbird: I attach gdb to a dying
>>> thunderbird process.  Note the error message marked with **** in the
>>> excerpted lines.
>>>
>>> This bug is discussed in the following bug entry and
>>> the gdb session log is also uploaded at the following URLs.
>>>
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=918234
>>> https://bug918234.bugzilla.mozilla.org/attachment.cgi?id=807287
>>>
>>>
>>> Except of gdb session
>>>
>>>
>>> $ gdb ../../.././mozilla/dist/bin/thunderbird 29680
>>> GNU gdb (GDB) 7.6 (Debian 7.6-5)
>>> Copyright (C) 2013 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>>> copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>...
>>>
>>> *****
>>> ***** The next line states that such file or directory does not exist
>>> ***** (in Japanese).
>>> ***** Strange: the command line is suggested by mozilla's
>>> ***** |make mozmill| test suite. I have never figured out what
>>> ***** the relative path is meant to be.
>>> ***** This may play an important role when -gsplit-dwarf is used.
>>> ***** Anyway as you can see from the full session log in the above URL,
>>> ***** libraries and such are correctly accessed and symbols are
>>> ***** deciphered except for libxul.so and a few others.
>>>
>>> ../../.././mozilla/dist/bin/thunderbird: そのようなファイルやディレクトリは
>>> ありません.
>>> Attaching to process 29680
>>>
>>> **** PROBLEM! A fly in the ointment!
>>> **** Next line indicates that there is some mismatch of the data
>>> **** layout and gdb's read routine.
>>>
>>> Reading symbols from
>>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...Dwarf Error: CU at
>>> offset 0x0 references
>>> unknown DWO with ID 77955146488e868b [in module
>>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
>>> (no debugging symbols found)...done.
>>>
>>>       **** I am afraid Debian GNU/Linux does not ship symbol information
>>> for
>>>       **** ld*.so type files: these files seem to be shipped in stripped
>>>       **** form.
>>> warning: Could not load shared library symbols for linux-vdso.so.1.
>>> Do you need "set solib-search-path" or "set sysroot"?
>>> Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading
>>> symbols
>>> from
>>> /usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.17.so...done.
>>> done.
>>> [New LWP 29712]
>>> [New LWP 29708]
>>> [New LWP 29693]
>>> [New LWP 29692]
>>> [New LWP 29687]
>>> [New LWP 29686]
>>>
>>>        ... the rest omitted ...
>>>
>>>
>>> I wonder if anyone can shed light on the error:
>>> Dwarf Error: CU at offset 0x0 references unknown DWO with ID
>>> 77955146488e868b [in module
>>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
>>> (no debugging symbols found)...done.
>>>
>>> I am using
>>> gcc 4.8
>>>       gcc-4.8 --version
>>>       gcc-4.8 (Debian 4.8.1-2) 4.8.1
>>>       Copyright (C) 2013 Free Software Foundation, Inc.
>>>       This is free software; see the source for copying conditions.
>>> There is NO
>>>       warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>>> PURPOSE.
>>>
>>> ld.gold
>>>       ld    --version
>>>       GNU gold (GNU Binutils for Debian 2.22) 1.11
>>>       Copyright 2011 Free Software Foundation, Inc.
>>>       This program is free software; you may redistribute it under the
>>> terms of
>>>       the GNU General Public License version 3 or (at your option) a
>>> later
>>> version.
>>>       This program has absolutely no warranty.
>>>       ishikawa@vm-debian-amd64:/COMM-CENTRAL/comm-central$
>>>
>>> gdb
>>>       gdb --version
>>>       GNU gdb (GDB) 7.6 (Debian 7.6-5)
>>>       Copyright (C) 2013 Free Software Foundation, Inc.
>>>       License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>>       This is free software: you are free to change and redistribute it.
>>>       There is NO WARRANTY, to the extent permitted by law.  Type "show
>>> copying"
>>>       and "show warranty" for details.
>>>       This GDB was configured as "x86_64-linux-gnu".
>>>       For bug reporting instructions, please see:
>>>       <http://www.gnu.org/software/gdb/bugs/>.
>>>
>>> (I have upgraded objcopy, etc. to support the split-dwarf scheme, also.
>>> Otherwise, the compile and liking with the options mentioned above would
>>> fail.)
>>>
>>> If this reading failure on startup of gdb
>>> is a known error (in which case, I would like it to be fixed
>>> soon), or if there is a workaround, or if I am using gdb incorrectly,
>>> I would like to hear about it.
>>>
>>> TIA
>>>
>>>
>>
>>
>>
>
>
>