Bug 20981

Summary: Dynamic library debugging on MacOS X 10.12 (Sierra) and dyld 15 is broken
Product: gdb Reporter: Stefan Budeanu <stefan.mb>
Component: gdbAssignee: Not yet assigned to anyone <unassigned>
Status: RESOLVED FIXED    
Severity: normal CC: benconvey, chrisfriedt, evan.torrie, jothiram.s, mephi42, pedro, question123446, rostyslav.skrypnyk, shahzeb001, simon, stefan.mb, tromey, wsimmons
Priority: P2    
Version: HEAD   
Target Milestone: 8.3   
Host: Target: *-*-darwin*
Build: Last reconfirmed:

Description Stefan Budeanu 2016-12-18 05:23:56 UTC
I am running Mac OS X 10.12.2 (Sierra) and I've noticed gdb's support for debugging dynamic libraries is broken. The following message is printed on startup:

"warning: unhandled dyld version (15)"

It appears the latest dyld on MacOS Sierra has undergone major changes and the version was bumped from 14 to 15. (https://opensource.apple.com/source/dyld/dyld-421.2/) 

Last time a version bump happened the fix was to simply modify the DYLD_VERSION_MAX define (see https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=d4ccb5e05c99c4006fe43ab08ebe13b7a74fc111).

I tried doing this and was met with a new error:

"Cannot insert breakpoint -1
Cannot access memory at address 0xe85d"

The address 0xe85d is the offset of the "gdb_image_notifier" function in the /usr/lib/dyld executable (see https://opensource.apple.com/source/dyld/dyld-421.2/src/dyld_gdb.cpp.auto.html).

It appears Apple has changed something fundamental about how dyld is built and/or loaded (possibly related to PIE/ASLR), here are before and after examples:

Mac OS X 10.11.6:

otool -l /usr/lib/dyld 
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           7    14       1696 0x00000085
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 552
  segname __TEXT
   vmaddr 0x00007fff5fc00000
   vmsize 0x0000000000038000
  fileoff 0
 filesize 229376
  maxprot 0x00000007
 initprot 0x00000005
   nsects 6
    flags 0x0

Mac OS X 10.12.2:

otool -l /usr/lib/dyld
/usr/lib/dyld:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           7    14       1696 0x00000085
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 552
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x000000000003e000
  fileoff 0
 filesize 253952
  maxprot 0x00000007
 initprot 0x00000005
   nsects 6
    flags 0x0

Notice that vmaddr is now 0x0. I wrote a hack to add the base address of the TEXT segment of dyld to the notifier address and this gets gdb to start successfully: https://github.com/stefanmb/binutils-gdb/commit/ee5bfeff7174f6e0b515f87d8ba2214ce45a02aa

With the patch "info shared" appears to work correctly.

I am now encountering further issues related to shared libraries and forking (children are seeing SIGSEGV after my patch), but I have not had a chance to investigate further. I will update this issue if I find more relevant info.

This behaviour reproduces on the latest gdb (7.12.50.20161217-git).

Does anyone know what has to be done to fix this issue properly?

I am available to test potential fixes, but I require some support to resolve this problem properly.

Thank you very much for your help!
Comment 1 Stefan Budeanu 2016-12-21 20:42:15 UTC
It turns out the fork crashing issues may be due to the children having software breakpoints left over after fork. I get a similar problem with lldb, and disassembling the core dump shows `INT 3` instructions being inserted in gdb_image_notifier. I think that is a separate problem (though also annoying).

The hack I linked to got gdb starting but I've run into a new problem setting breakpoints in my library because the location of my functions seem to also be read as offsets instead of the relocated addresses. This issue only seems to happen for certain object files and the raw offsets are being read directly from the DWARF2 info as far as I can tell.
Comment 2 Joe 2017-05-06 15:51:23 UTC
In my case, the gdb is starting the original program successfully, but 

gdb) info shared

does not show any information. Also, the break points that I had set in the shared library are not detected and the execution does not stop there.
Comment 3 Shahzeb 2017-05-15 03:09:24 UTC
Any update on this?

I've uninstalled and then installed gdb twice (via brew). I've also codesigned it. Still get "No shared libraries loaded at this time.". Really frustrating.

macOS: 10.12.4
Comment 4 Chris Friedt 2017-09-26 23:05:37 UTC
I'm also encountering this on 10.12.6 ... sigh..
Comment 5 EvanT 2017-10-25 02:32:22 UTC
This also occurs on Mac OS X 10.13 (High Sierra)
Comment 6 Shahzeb 2017-10-25 04:30:59 UTC
This problem still isn't resolved, but I came up with a solution: Simply run a docker container I've built which has GDB within it. Instructions on my github: https://github.com/shahzeb1/cpp-gdb-docker-image
Comment 7 Andy Law 2017-10-27 12:27:14 UTC
Gdb 8.0.1 doesn't have this problem as I see. However, the dynamic library still can't debug at all. It can't collect debug symbols...
Comment 8 Chris Friedt 2017-12-05 00:47:10 UTC
#sadface :(
Comment 9 Pedro Alves 2017-12-05 11:55:45 UTC
The simple hard fact is that unfortunately the macOS/Darwin port isn't really getting much attention.  Most gdb devs tend to run/focus on GNU/Linux.
We need someone motivated to actually dig in to the sources, study the issue in detail, and fix the issue.  Who knows, maybe end up becoming port maintainer.  :-)

This could be you!
Comment 10 ben convey 2018-03-17 12:02:11 UTC
same issue on high sierra ( 10.13.2 ) :(
Comment 11 Chris Friedt 2018-07-09 21:38:23 UTC
These days, things are also just being run as Linux binaries inside of a Docker container on Mac OS X.. except those rare occasions when it *really* needs to be MachO.

Maybe one of these days I'll get around to it.
Comment 12 Sourceware Commits 2018-09-19 21:24:32 UTC
The master branch has been updated by Tom Tromey <tromey@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=79b32f4a3a109321cb9a99014293b0ade57535ab

commit 79b32f4a3a109321cb9a99014293b0ade57535ab
Author: Xavier Roirand <roirand@adacore.com>
Date:   Wed Aug 22 12:11:14 2018 +0200

    Darwin: Handle unrelocated dyld.
    
    On Darwin, debugging an helloworld program with GDB does
    not work and ends with:
    
      (gdb) set startup-with-shell off
      (gdb) start
      Temporary breakpoint 1 at 0x100000fb4: file /tmp/helloworld.c, line 1.
      Starting program: /private/tmp/helloworld
      [New Thread 0x2703 of process 18906]
      [New Thread 0x2603 of process 18906]
    
      [1]+  Stopped                 ./gdb/gdb /tmp/helloworld
    
    When debugging with lldb, instead of having the STOP signal, we can
    see that a breakpoint is not set to a proper location:
    
      Warning:
      Cannot insert breakpoint -1.
      Cannot access memory at address 0xf726
    
      Command aborted.
    
    The inserted breakpoint is the one used when GDB has to stop the target
    when a shared library is loaded or unloaded. The notifier address used
    for adding the breakpoint is wrong thus the above failure.
    This notifier address is an offset relative to dyld base address, so
    the value calculation has to be updated to reflect this.
    
    This was tested on High Sierra by trying to run a simple "hello world"
    program.
    
    gdb/ChangeLog:
    
    	PR gdb/20981:
            * solib-darwin.c (darwin_get_dyld_bfd): New function.
            (darwin_solib_get_all_image_info_addr_at_init): Update call.
            (darwin_handle_solib_event): New function.
            (darwin_solib_create_inferior_hook): Handle unrelocated dyld.
    
    Change-Id: I7dde5008c9158f17b78dc89bd7f4bd8a12d4a6e1
Comment 13 Tom Tromey 2018-09-19 21:28:19 UTC
I've pushed a variant of Xavier's patch which fixes the problem
for me on High Sierra.  I think this is fixed.

However, I could only test High Sierra.  So, if you can check
out git master gdb and try it on other versions, that would be nice.

I am going to close this bug; but feel free to report back here
(I'll still be listening) and/or reopen if there are similar problems.
Comment 14 Rostyslav Skrypnyk 2020-04-13 21:04:25 UTC
I also encounter "warning: unhandled dyld version (16)" (now the version has incremented) on Mac OS Catalina 10.15.4. 

Also, every other time I experience the issue of hanging described here: https://sourceware.org/bugzilla/show_bug.cgi?id=24069

I tried both installing via Homebrew and building from source.