gdb and fission scheme (gcc: -gsplit-dwarf, gnu ld gold :--gdb-index )
ISHIKAWA,chiaki
ishikawa@yk.rim.or.jp
Sun Sep 22 02:29:00 GMT 2013
I have investigated a little further.
Now it has dawned on me that using "-gsplit-dwarf" binaries for
debugigng is not a simple plug-and-play in contrast to ordinary dwarf
binaries where debug information for a source module is in the single
.o file and linker consolidates the information into a single final
binary.
After looking at this error message line below, I have come to the
conclusion that gdb is complaining that
- it has found a CU dwarf section (or whatever),
- this CU section refers to a DWO section (or file) with ID
77955146488e868b,
- but it has no knowledge of such DWO section and its whereabout,
thus "unknown DWO".
--- quote ---
Reading symbols from
/REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...Dwarf Error: CU at
offset 0x0 references
unknown DWO with ID 77955146488e868b [in module
/REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
(no debugging symbols found)...done.
--- end quote ---
So I think I need to tell that relevant .dwo files are in the object directory of thunderbird.
Now that debug information is split, I have to do this. OK, fair enough.
But how can I tell gdb that all the .dwo files and dynamically linked
.so files are under certain directories? Do I have to tell gdb.
Also, do I have to tell gdb in advance, all the *.dwo files that are referenced by thunderbird, and
how?
After reading through gdb documentation and not finding much about
split dwarf feature itself , I now realize that it looks to me that I have to
- create file location dwarf information for gdb to locate
information in scattered *.dwo files,
B ut again, how?
- One possibility: this is done by using objcopy by extracting certain debug
information from ALL *.dwo files and merging into a certain dwarf
data structure (list of known DWO sections with its unique IDs and
the file pathname?),
- AND maybe physically attaching this merged dwarf information to
|thunderbird| and |thunderbird-bin| binary. Basically extracting all
the sections of |thunderbird| and merge them with the list of known
DWO sections with the IDs and file pathnames.) and creating a new
binary.
- This recreated binary will be used so that gdb will notice all the
indirect references to the debug information in all *.dwo files when
|thunderbird| and |thunderbird-bin| is loaded?
Is this what one is supposed to do when -gsplit-dwarf is used
and gdb debugging is attempted?
Oh, I now realize another possibility of creating consolidated
symbolic information for gdb debugging.
maybe I can read all the *.dwo files one by one from within gdb
(of course, I will use gdb script to read them in one sweep),
and write out the symbolic information for next time by |save gdb-index|
This may be the way to go, but not sure
I appreciate any helpful tips and guidance.
TIA
cf. *.dwo files and *.so files created and used
by mozilla thunderbird
I have re-compiled the tree under /REF-COMM-CENTRAL/comm-central
using the ordinary dwarf debug information (without -gsplit-dwarf and -Wl,--gdb-index)
so I am referring to a different tree which was compiled using -gsplit-dwarf.
The top-most directory of the source file hierarchy is /COMM-CENTRAL/comm-central, and
the top-most directory of the object file hierarchy is /TEST-MAIL-OBJ/objdir-tb3.
The final thunderbird binary is in the following directory.
/TEST-MAIL-DIR/objdir-tb3 is the top of the object directory.
There are many *.dwo files scattered in the object tree.
(The structure of the object tree reflects the original source tree more or less.)
The accompanying *.o files are under the same tree.
Also, there are *.so files scattered in the object tree. They are
consolidated into several directories and the number of directories were smaller than in the case of
*.dwo files.
ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ ls -l mozilla/dist/bin/thunderbird*
-rwxr-xr-x 1 ishikawa ishikawa 267277 Sep 20 12:19 mozilla/dist/bin/thunderbird*
-rwxr-xr-x 1 ishikawa ishikawa 267277 Sep 20 08:42 mozilla/dist/bin/thunderbird-bin*
[*.dwo files]
ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ find . -name "*.dwo" -print
./ldap/xpcom/src/nsLDAPProtocolModule.dwo
[..omission..]
./ldap/xpcom/src/nsLDAPServer.dwo
./db/mork/src/morkStore.dwo
./db/mork/src/morkPortTableCursor.dwo
[...omission...]
./db/mork/src/morkWriter.dwo
./db/mork/build/nsMorkFactory.dwo
./mozilla/parser/htmlparser/src/nsHTMLEntities.dwo
./mozilla/parser/htmlparser/src/nsHTMLTags.dwo
[...omission...]
./mozilla/parser/htmlparser/src/nsExpatDriver.dwo
./mozilla/parser/xml/src/nsSAXXMLReader.dwo
./mozilla/parser/xml/src/nsSAXLocator.dwo
./mozilla/parser/xml/src/nsSAXAttributes.dwo
./mozilla/parser/expat/lib/xmlparse.dwo
./mozilla/parser/expat/lib/xmltok.dwo
./mozilla/parser/expat/lib/xmlrole.dwo
./mozilla/parser/html/nsHtml5TreeBuilder.dwo
[...omission...]
./mozilla/parser/html/nsHtml5Parser.dwo
./mozilla/xpcom/reflect/xptcall/src/xptcall.dwo
[...omission...]
./mozilla/xpcom/reflect/xptinfo/src/xptiWorkingSet.dwo
./mozilla/xpcom/io/Base64.dwo
[...omission...]
./mozilla/xpcom/io/nsAppFileLocationProvider.dwo
[... lots of lines omitted.]
./mail/components/shell/DirectoryProvider.dwo
[*.so files]
ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ find . -name "*.so" -print
./ldap/sdks/c-sdk/ldap/libraries/libldif/libldif60.so
./ldap/sdks/c-sdk/ldap/libraries/libprldap/libprldap60.so
./ldap/sdks/c-sdk/ldap/libraries/libldap/libldap60.so
./mozilla/xpcom/tests/component_no_aslr/libtestcompnoaslr.so
./mozilla/xpcom/tests/component/libtestcomponent.so
./mozilla/xpcom/tests/bug656331_component/libtest656331.so
./mozilla/dist/plugins/libnptest.so
./mozilla/dist/plugins/libnpsecondtest.so
./mozilla/dist/bin/libmozalloc.so <--- this is the directory where
./mozilla/dist/bin/libnspr4.so <--- the main binary is stored.
./mozilla/dist/bin/libfreebl3.so <--- *.so files are symlinks to other files under the
./mozilla/dist/bin/libssl3.so <--- top-most objdir-tb3
./mozilla/dist/bin/libnssdbm3.so
./mozilla/dist/bin/libplds4.so
./mozilla/dist/bin/libsmime3.so
./mozilla/dist/bin/libmozsqlite3.so
./mozilla/dist/bin/libnssutil3.so
./mozilla/dist/bin/libxul.so <--- a symlink libxul.so -> ../../toolkit/library/libxul.so
./mozilla/dist/bin/libsoftokn3.so
./mozilla/dist/bin/libprldap60.so
./mozilla/dist/bin/libldap60.so
./mozilla/dist/bin/libnssckbi.so
./mozilla/dist/bin/libldif60.so
./mozilla/dist/bin/libnss3.so
./mozilla/dist/bin/libplc4.so
./mozilla/dist/bin/libnpsecondtest.so
./mozilla/dist/bin/components/libdbusservice.so
./mozilla/dist/bin/components/libmozgnome.so
./mozilla/dist/lib/libmozalloc.so
./mozilla/dist/lib/libnspr4.so
./mozilla/dist/lib/libfreebl3.so
./mozilla/dist/lib/libssl3.so
./mozilla/dist/lib/libnssdbm3.so
./mozilla/dist/lib/libplds4.so
./mozilla/dist/lib/libsmime3.so
./mozilla/dist/lib/libmozsqlite3.so
./mozilla/dist/lib/libnssutil3.so
./mozilla/dist/lib/libxul.so <--- symlink to libxul.so -> ../../toolkit/library/libxul.so
./mozilla/dist/lib/libsoftokn3.so
./mozilla/dist/lib/libprldap60.so
./mozilla/dist/lib/libldap60.so
./mozilla/dist/lib/libnssckbi.so
./mozilla/dist/lib/libldif60.so
./mozilla/dist/lib/libnss3.so
./mozilla/dist/lib/libplc4.so
./mozilla/dist/lib/libnpsecondtest.so
./mozilla/dist/sdk/lib/libmozalloc.so
./mozilla/dist/sdk/lib/libnspr4.so
./mozilla/dist/sdk/lib/libssl3.so
./mozilla/dist/sdk/lib/libplds4.so
./mozilla/dist/sdk/lib/libsmime3.so
./mozilla/dist/sdk/lib/libnssutil3.so
./mozilla/dist/sdk/lib/libxul.so <--- libxul.so -> ../../../toolkit/library/libxul.so
./mozilla/dist/sdk/lib/libnss3.so
./mozilla/dist/sdk/lib/libplc4.so
./mozilla/dom/plugins/test/testplugin/secondplugin/libnpsecondtest.so
./mozilla/dom/plugins/test/testplugin/libnptest.so
./mozilla/security/nss/lib/util/libnssutil3.so
./mozilla/security/nss/lib/ckfw/builtins/libnssckbi.so
./mozilla/security/nss/lib/smime/libsmime3.so
./mozilla/security/nss/lib/softoken/legacydb/libnssdbm3.so
./mozilla/security/nss/lib/softoken/libsoftokn3.so
./mozilla/security/nss/lib/freebl/libfreebl3.so
./mozilla/security/nss/lib/nss/libnss3.so
./mozilla/security/nss/lib/ssl/libssl3.so
./mozilla/build/unix/elfhack/test-array.so
./mozilla/build/unix/elfhack/test-ctors.so
./mozilla/nsprpub/pr/src/libnspr4.so
./mozilla/nsprpub/lib/libc/src/libplc4.so
./mozilla/nsprpub/lib/ds/libplds4.so
./mozilla/memory/mozalloc/libmozalloc.so
./mozilla/toolkit/system/dbus/libdbusservice.so
./mozilla/toolkit/system/gnome/libmozgnome.so
./mozilla/toolkit/crashreporter/test/libtestcrasher.so
./mozilla/toolkit/library/libxul.so <--- libary file : size is 283MiB
This is acccompanied by libxul.so-gdb.py in the same directory
""" GDB Python customization auto-loader for libxul """
import os.path
sys.path[0:0] = [os.path.join('/COMM-CENTRAL/comm-central/mozilla', 'js', 'src', 'gdb')]
import mozilla.autoload
mozilla.autoload.register(gdb.current_objfile())
./mozilla/toolkit/components/ctypes/tests/libjsctypes-test.so
./mozilla/db/sqlite3/src/libmozsqlite3.so
./mozilla/js/xpconnect/tests/components/native/libxpctest.so
./mozilla/_tests/testing/mochitest/chrome/libraries/libjsctypes-test.so
./mozilla/_tests/testing/mochitest/chrome/toolkit/components/toolkit/components/ctypes/tests/libjsctypes-test.so
./mozilla/_tests/mozmill-virtualenv/lib/python2.7/site-packages/simplejson/_speedups.so
./mozilla/_tests/mozmill/mozmillprofile/plugins/libnptest.so
./mozilla/_tests/mozmill/mozmillprofile/plugins/libnpsecondtest.so
./mozilla/_tests/xpcshell/xpcom/tests/unit/libtestcompnoaslr.so
./mozilla/_tests/xpcshell/xpcom/tests/unit/libtestcomponent.so
./mozilla/_tests/xpcshell/xpcom/tests/unit/libtest656331.so
./mozilla/_tests/xpcshell/toolkit/crashreporter/test/unit/libtestcrasher.so
./mozilla/_tests/xpcshell/toolkit/crashreporter/test/unit_ipc/libtestcrasher.so
./mozilla/_tests/xpcshell/toolkit/components/ctypes/tests/unit/libjsctypes-test.so
./mozilla/_tests/xpcshell/js/xpconnect/tests/components/native/libxpctest.so
ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$
(2013/09/21 6:43), ISHIKAWA,chiaki wrote:
> I think the following problem is related to the reading of dynamically linked libxul.so library file.
> The problem affects the behavior of gdb in that it fails to access the source files.
> So it can not list the source lines.
>
> I checked the operation of gdb against statically linked small program and
> all is well.
> It can list the source files, and this is the first thing I immediately noticed.
>
> With the setup described in the previous post of mine,
> gdb can print the stack trace (sans the argument values), but
> it can't print source files. Since the argument values are not shown in the
> stack trace. Its value is also diminished.
>
> Help is appreciated.
> Yes, libxul.so is rather large. Thunderbird is a large program and so
> we may hit into an unknown bug here.
>
>
>
> (2013/09/20 14:24), ishikawa wrote:
>> Hi,
>>
>> I recently learned of "-gsplit-dwarf" of GCC 4.8 and -Wl,--gdb-index
>> that is passed to GNU ld.gold linker, and have tried to use the split
>> debug information scheme for local compilation and link of mozilla
>> thunderbird mail client.
>> ( http://gcc.gnu.org/wiki/DebugFission )
>>
>> The compiling and linking produced working thunderbird binary. The
>> time required for whole build process, especially the time necessary
>> for creating a large libxul.so file diminished dramatically. I would
>> like to thank all the developers who have made this possible.
>>
>> There is a fly in the ointment, though. Unfortunately, somehow gdb is
>> not reading the symbol information from a created shared library,
>> libxul.so, correctly. Below is the beginning of a gdb session of
>> trying to debug a crash of thunderbird: I attach gdb to a dying
>> thunderbird process. Note the error message marked with **** in the
>> excerpted lines.
>>
>> This bug is discussed in the following bug entry and
>> the gdb session log is also uploaded at the following URLs.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=918234
>> https://bug918234.bugzilla.mozilla.org/attachment.cgi?id=807287
>>
>>
>> Except of gdb session
>>
>>
>> $ gdb ../../.././mozilla/dist/bin/thunderbird 29680
>> GNU gdb (GDB) 7.6 (Debian 7.6-5)
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>>
>> *****
>> ***** The next line states that such file or directory does not exist
>> ***** (in Japanese).
>> ***** Strange: the command line is suggested by mozilla's
>> ***** |make mozmill| test suite. I have never figured out what
>> ***** the relative path is meant to be.
>> ***** This may play an important role when -gsplit-dwarf is used.
>> ***** Anyway as you can see from the full session log in the above URL,
>> ***** libraries and such are correctly accessed and symbols are
>> ***** deciphered except for libxul.so and a few others.
>>
>> ../../.././mozilla/dist/bin/thunderbird: ãã®ã‚ˆã†ãªãƒ•ã‚¡ã‚¤ãƒ«ã‚„ディレクトリã¯
>> ã‚ã‚Šã¾ã›ã‚“.
>> Attaching to process 29680
>>
>> **** PROBLEM! A fly in the ointment!
>> **** Next line indicates that there is some mismatch of the data
>> **** layout and gdb's read routine.
>>
>> Reading symbols from
>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...Dwarf Error: CU at
>> offset 0x0 references
>> unknown DWO with ID 77955146488e868b [in module
>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
>> (no debugging symbols found)...done.
>>
>> **** I am afraid Debian GNU/Linux does not ship symbol information for
>> **** ld*.so type files: these files seem to be shipped in stripped
>> **** form.
>> warning: Could not load shared library symbols for linux-vdso.so.1.
>> Do you need "set solib-search-path" or "set sysroot"?
>> Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols
>> from
>> /usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.17.so...done.
>> done.
>> [New LWP 29712]
>> [New LWP 29708]
>> [New LWP 29693]
>> [New LWP 29692]
>> [New LWP 29687]
>> [New LWP 29686]
>>
>> ... the rest omitted ...
>>
>>
>> I wonder if anyone can shed light on the error:
>> Dwarf Error: CU at offset 0x0 references unknown DWO with ID
>> 77955146488e868b [in module
>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
>> (no debugging symbols found)...done.
>>
>> I am using
>> gcc 4.8
>> gcc-4.8 --version
>> gcc-4.8 (Debian 4.8.1-2) 4.8.1
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions. There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> ld.gold
>> ld --version
>> GNU gold (GNU Binutils for Debian 2.22) 1.11
>> Copyright 2011 Free Software Foundation, Inc.
>> This program is free software; you may redistribute it under the terms of
>> the GNU General Public License version 3 or (at your option) a later
>> version.
>> This program has absolutely no warranty.
>> ishikawa@vm-debian-amd64:/COMM-CENTRAL/comm-central$
>>
>> gdb
>> gdb --version
>> GNU gdb (GDB) 7.6 (Debian 7.6-5)
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>>
>> (I have upgraded objcopy, etc. to support the split-dwarf scheme, also.
>> Otherwise, the compile and liking with the options mentioned above would fail.)
>>
>> If this reading failure on startup of gdb
>> is a known error (in which case, I would like it to be fixed
>> soon), or if there is a workaround, or if I am using gdb incorrectly,
>> I would like to hear about it.
>>
>> TIA
>>
>>
>
>
>
More information about the Gdb
mailing list