Re: dlmopen and core dumps

["Carlos O'Donell" <carlos at systemhalted dot org>]

Gary,

Very interesting problem. Thanks for the post.

CC'ing Roland who our expert on old things.

On Mon, Nov 5, 2012 at 8:21 AM, Gary Benson <gbenson@redhat.com> wrote:
> I've been looking into ways of allowing debuggers to see libraries
> loaded into namespaces other than the initial namespace with dlmopen.
> The SystemTap probes added to the runtime linker recently make such
> libraries visible during live debugging, but there is currently no way
> to access them from core dumps.  I'd like to discuss some ideas to
> make this possible with the view of submitting a patch.

Is that because r_debug->r_map won't contain anything not in the
initial namespace?

What do core dumps have to do with this? Isn't it the same problem in
a live process?

> Currently, only the r_debug structure of the initial namespace is
> visible to the debugger, which reads its location from the DT_DEBUG
> element of the executable's dynamic section.  The simplest way to make
> the other namespaces available (as I see it) would be to also expose
> the r_debug structures for the other namespaces somehow.

Agreed.

> There are two questions really, what format should the data be in, and
> how should the debugger discover it.

Actually the first question I have is: Is there prior art for this?
Have you done
a review of the existing solutions from other unices to see if any of them
have implemented what we think we need?

> The obvious way to expose the structures would be in a linked list.
> It *could* be done with a fixed-length array--namespaces are not
> currently allocated dynamically--but this would cause problems for any
> future hacker trying to remove this restriction, so I'm not seriously
> proposing that.

A fixed-length array is not a good future-proof solution.

> If it is to be a linked list, there are two possibilities for what it
> should contain.  It could contain only active namespaces, which would
> require some maintainence as namespaces were created and destroyed, or
> it could be allowed to contain empty namespaces, in which case the
> list could be set up by the linker at startup and then forgotten
> about.  I prefer the former, but I'm happy with the latter too.

The linker has to do namespace management anyway, so I'm not sure
how much it buys you to avoid the maintenance.

> If it is to be a linked list, a data structure for the list would be
> needed somewhere.  Would link.h be a good place for it?

Yes, link.h.

> Concerning how the debugger would discover the new structure, I can
> think of two ways.  One would be to add another element to the dynamic
> section, DT_DEBUG_EXTENDED or something similar.  I don't know the
> process for doing this (is there some group that controls the ELF
> format?)  The other way would be to have some specifically named
> symbol that debuggers could locate, perhaps restricting their search
> to the object containing the symbol referenced in DT_DEBUG.  I'm
> happy with either of these solutions, but I don't know enough to say
> if one or the other has problems so I'd appreciate feedback.

(1) New DT_* entry.

We could create a new entry for this purpose.

We would have to talk it over in the "Generic System V Application
Binary Interface" google group. That is where you can make these
kinds of requests, and then try to get it added to the official documentation.

DT_DEBUG_EXTENDED is a mouthful, but the name is really just
a bike-shed, we can call it anything.

(2) Bump r_version.

I also think we could bump the r_version in _r_debug to
indicate the protocol has changed and simply extend the
structure to whatever we need.

A new debugger would notice r_version = 2, and use the
newer *larger* struct layout with new members off the end.

The problem is if current gdb's abort if they see r_version != 1.
Lets hope not.

> The final consideration is how to version the new interface.  I'm
> presuming that other interfaces glibc implements have some kind of
> versioning information, but I'm not familiar enough with everything to
> pick some specific example.  My instict says to go for a pair of
> integers, a major and a minor version, such that nonbreaking changes
> increment the minor version and breaking changes increment the major
> and reset the minor.  If so, is it best to put two ints in the exposed
> structure, or maybe just one with the major version encoded in the
> upper bits?  If there is a standard way interfaces are versioned (or
> an example of a good interface) then I'm happy to follow that.

r_debug is already versioned with r_version, see suggestion (2) above.

Careful with the word interfaces. In a shared object you have symbols.
The symbols can point to many different things, like data, or functions.
An interface is a broader concept than the concept to which versioning
is applied in glibc. Versioning applies to symbols. While a symbol and
the implementation of the interface may not change (no new version),
the protocol used in the exchange of data might (different higher-level
interface!).

We can version symbols. We could version the _r_debug symbol,
but then we'd have two different _r_debug's e.g. _r_debug@@GLIBC_2.18
(new default symbol) and _r_debug@GLIBC_2.2 (old), which would the
debugger pick?

The dynamic loader would then have to maintain both copies, one for
old debuggers with r_version=1, and a new one with r_version=2
for newer debuggers.

That's very costly backwards compatibility.

Versioning the symbol should be avoided here since it's overkill.

We usually just use a single integer to indicate the version of
a particular data structure, anything more is overkill. Just bump
the version by one for each change. All changes are backwards
compatible. Don't every break the debugger.

In summary:
- Suggest you bump r_version to 2.
- Extend existing r_debug struct.
- Make sure gdb doesn't fail when it sees r_version==2.
- Extend gdb to notice r_version==2 as an ABI compatible extension.
- When gdb notices r_version==2 have it use the newer struct layout.
- WARNING: Special consideration should be given to static appliations
  that use dlmopen. I would have to review what we do in that case.

Comments?

Cheers,
Carlos.

Note:
* Further reading on the topic of versioned structures:
http://sourceware.org/glibc/wiki/Development/Versioning_A_Structure