Re: Should strip discard the .ctf section ?

[Nick Alcock <nick dot alcock at oracle dot com>]

On 7 Oct 2019, Rich Felker stated:

> On Mon, Oct 07, 2019 at 04:22:37PM +0100, Nick Alcock wrote:
>> Sure, except that these probably have the same content, or at least I
>> can't see how they would differ. I hope the current .ctf section is
>> useful for both use cases (and if it's not, I want to know, so the next
>> format revision, format v4, can fix it so it is!).
>
> Likewise the content of .eh_frame and .debug_frame does not differ. If
> you build with unwind information, the content ends up in .eh_frame
> which is part of the loadable program segments. If you build without
> it, but with -g, the content ends up in .debug_frame. If you build
> with neither, it's absent entirely. Debuggers know to use either
> .eh_frame or .debug_frame. My view is that CTF should match.

Well, OK. This is in a non-loaded segment always, though, so I guess
nothing needs to be done (the loadability of the segment never varies).

>> >                                Forcing the latter to include it in
>> > load segments, making it unstrippable, will make many of us unhappy to
>> > have it at all.
>> 
>> I think we already do what you want here. .ctf is already a non-loaded
>> section,
>
> If it's a non-loaded section, programs cannot use it to introspect. If
> you want to have that functionality, you need a variant where it's
> loaded.

Err... 'cannot'? They *do*. This is what libctf is *for*. Do you mean
'should not'? Why not? What goes wrong if they do?

>> so there is more-or-less zero memory hit from having .ctf
>> present unless libctf loads the section in at runtime using bfd. This is
>> *why* libctf can load in .ctf sections using bfd, and why the default
>> ctf_open() does just that :)
>
> This is not following ELF semantics correctly. Program behavior
> absolutely should not depend, ever, on details of the file outside of
> the loadable segments.

Really?! That's the first I've heard of that. If it's such an important
principle, it should probably be documented somewhere, in bigger letters
than it already is. I know the presence or absence of debuginfo shold
not affect the behaviour of a program that isn't a debugger, that's
obvious, but that's because it's *debugging* info, not because it's not
loaded. There is an obvious user QoI reason to not modify program
behaviour when you start to debug something, and it's not because the
section is non-loaded. It would apply just as much if debuginfo were
loaded.


What I'm doing here is not radical, but is exactly what Solaris has done
for nearly twenty years: also FreeBSD, etc, all of which build CTF data
into non-loadable sections of many programs and then provide a library
*any* interested programs can use to access it for whatever purpose they
wish: perhaps debugging of other programs, perhaps introspection of
themselves: both work, and with the same code. And obviously if you're
using the content of a section in any way whatsoever you're going to use
it to modify program behaviour.

(This also suggests that program behaviour should never depend on the
content of debug information, which suggests that it is invalid for a
debugger to debug itself. GDB tests exactly that in its own testsuite.
Should debuginfo be kept in a loaded section just in case a debugger
does this? If not, why does the same argument not apply to CTF?)

Your assertion is tantamount to saying "non-loaded sections should never
exist", since they amount to a prohibition on ever using them for any
reason. You'll pardon me if I find this argument quite unconvincing.

>                        If you're going to keep the CTF data anyway,
> there is no cost (except some virtual address space, which I guess
> could matter for huge programs on 32-bit archs, but if the CTF is that
> huge then disk space matters too) to having it be loadable. But it
> should be a choice.

There was a demand *in this thread* that the CTF must be non-loaded.
I've now had demands that it be non-loaded *and* demands that it be
loaded. Having it being loaded requires a major linker rearchitecture,
and I cannot see any advantage to doing so, so for now it is non-loaded.

The user-visible behaviour is identical to what it would be if it were
loaded, except I don't need to massively change the linker, and libctf
is simpler because it only needs one code path ("load the section"
rather than having to have extra code paths and probably extra API for
introspecting yourself and already-loaded shared objects as opposed to
things that happen not to be loaded into the current address space yet,
since one of them uses a filename and... honestly, I have no idea how
you'd even reference CTF in different shared objects if it was a loaded
section, other than by loading it in yourself as if it were non-loaded:
another strike against it).

>> I have had people asking me to make .ctf a loaded segment, but in
>> addition to having the same qualms as you about loading this section in
>> and eating memory before it is in widespread use, I can see no way to do
>
> The idea that it "eats memory" is incorrect.

Eats address space, then. It might or might not get paged in. People on
this thread alone definitely dislike the idea for something that is not
yet widely used.

>> such a thing without a radical linker rearchitecture. The section is
>> usually compressed, so its size depends on its content: and its content
>> depends on the content of the ELF string and symbol tables, which GNU ld
>> lays out only after the size of all loaded sections is known. So we
>> cannot make .ctf a loaded section. At least not yet.
>
> I think this still needs major work then.

I don't really think rewriting GNU ld for the sake of CTF is remotely
proportionate. (And I'm sure that a change so radical for the sake of
something like this would be rejected. At least I'd hope so. I know I'd
reject it if I were looking at a change like that.)


And it would be a rewrite. I frankly wouldn't even know where to
*start*, given the number of barely-documented invariants scattered
through elflink.c and surrounding code that depend on addresses all
being assigned before the symtab and strtab are laid out: I'd need to
change it to some sort of multi-pass system in which sections could
depend on each other, recursively, and get laid out and addresses
assigned in dependency order (with the CTF section depending on .strtab
and .symtab)... which is a massive rewrite and would be sure to break
all sorts of other things horribly. And what would all this work be for,
when every single library that has ever existed that uses CTF data --
libctf in binutils is not the first -- expects to load it in itself
without relying on the ELF loader to do so, so all that work wouldn't be
useful without further changes -- and even if those changes are done,
there are no functional effects or benefits to users whatsoever that I
can see.

What is the actual *benefit* of doing this? Ontological cleanliness?
Desirable, I suppose, but a long way below things with actual user-
visible effect on my priority list.

>> .... I really must submit the spec, since it might make much of this
>> stuff clearer. It's not quite complete (no documentation of older file
>> formats or the libctf API yet): is that a problem, or can I add that
>> incrementally?
>
> Tooling should not be committing to new features for which
> compatibility will need to be kept basically-forever until there is a
> clear spec

It's written, I just need to submit it. :)

>            and until issues like the above are worked out.

In the absence of a time machine, that's a bit difficult. Nobody ever
suggested the necessity of anything like this before. Good thing too,
because it is frankly utterly impractical. Maybe Alan can wrangle the
linker enough to make changes of that magnitude. I know I can't.