[PATCH v3 00/33] CTF linking support

[Nick Alcock <nick dot alcock at oracle dot com>]

This is v3 of the CTF linking patch series, with many bugfixes since the
last submission, including unbreaking the assembler, fixing linking on
non-ELF platforms, speeding the CTF linker up massively and fixing a
significant number of bugs in libctf exposed by the linking code, some
of which are design bugs fifteen years old.  No regressions in make
check on any platform I have access to (the usual mix of
x86_64-pc-linux-gnu, FreeBSD 12, Solaris, SPARCLinux, and mingw); no
regressions in make check with -gt forced on except for ones that are
necessarily there because of the introduction of a new section in the
output without regenerating the expected output.

As another change in this series, we make libctf into an installable
(.a) library, so that external programs can benefit from the new libctf
linking API, to link CTF like ld does.  Since libctf uses libiberty, and
many plausible uses link libctf.a into a shared library, this also
requires us to install the PIC version of libiberty: most distros are
doing this already, so this is probably worth doing regardless.  (This
change let us test using the libctf linking machinery with a custom CTF
merging tool to merge all the types in the entire Linux kernel, as an
extreme load test due to the lack of a type deduplicator at this stage:
doing things at that sort of scale also found a great many bugs.)

I'll probably try some full-distro builds next, both with -gt off and
with it on. :)

Some patches below are marked REVIEW and need close looking at: they're
mostly the parts which tie the libctf linking machinery into ld itself.
It's quite possible that you'd rather I do what I'm doing there some
other way.  Extensive questions for reviewers are in the commit log for
that patch.

But despite the caveats in that patch, in conjunction with a CTF-capable
GCC, it does work, without warnings or leaks, and merges the type
sections down quite satisfactorily, though they are definitely afflicted
by the lack of type deduplication as yet:

oranix@loom 551 % size -A /tmp/gcc/bin/ld
/tmp/gcc/bin/ld  :
section                  size      addr
.interp                    28   4195040
.note.gnu.build-id         36   4195068
.note.ABI-tag              32   4195104
.gnu.hash                 172   4195136
.dynsym                  3288   4195312
.dynstr                  1106   4198600
.gnu.version              274   4199706
.gnu.version_r            112   4199984
.rela.dyn                 192   4200096
.rela.plt                2856   4200288
.init                      26   4206592
.plt                     1920   4206624
.plt.got                    8   4208544
.text                  960162   4208560
.fini                       9   5168724
.rodata               1444864   5169152
.eh_frame_hdr           19596   6614016
.eh_frame              124928   6633616
.tbss                       8   6766088
.init_array                 8   6766088
.fini_array                 8   6766096
.dynamic                  480   6766104
.got                        8   6766584
.got.plt                  976   6766592
.data                   25328   6767584
.bss                    22976   6792928
.comment                   91         0
.debug_aranges           6320         0
.debug_info           4405042         0
.debug_abbrev          146018         0
.debug_line            765662         0
.debug_str             233645         0
.debug_loc            2125078         0
.debug_ranges          186368         0
.ctf                   388104   6824096
Total                10865729

oranix@loom 552 % PATH=/tmp/gcc/bin:$PATH objdump --ctf=.ctf /tmp/gcc/bin/ld

/tmp/gcc/bin/ld:     file format elf64-x86-64

Contents of CTF section .ctf:

  Header:
    Magic number: dff2
    Version: 4 (CTF_VERSION_3)
    Flags: 0x1 (CTF_F_COMPRESS)
    Variable section:   0x0 -- 0x1a17 (0x1a18 bytes)
    Type section:       0x1a18 -- 0x1ba623 (0x1b8c0c bytes)
    String section:     0x1ba624 -- 0x1d9940 (0x1f31d bytes)

  Labels:

  Data objects:

  Function objects:

  Variables:
    VerilogDataWidth ->  10: unsigned int (size 0x4)
    _CTF_NULLSTR ->  4b5c: const char [0] (size 0x0)
    _CTF_SECTION ->  4b5c: const char [0] (size 0x0)
    _IO_2_1_stderr_ ->  68: struct _IO_FILE_plus (size 0x6)
    _IO_2_1_stdin_ ->  68: struct _IO_FILE_plus (size 0x6)
    _IO_2_1_stdout_ ->  68: struct _IO_FILE_plus (size 0x6)
    __after_morecore_hook ->  1df1: void (*volatile) () (size 0x8) -> 1df0: void (*)() (size 0x8) -> 1def: void () (size 0x0)
    __daylight ->  5: int (size 0x4)
    __environ ->  17f: char ** (size 0x8) -> 17e: char * (size 0x8) -> 4b: char (size 0x1)
    __free_hook ->  1df7: void (*volatile) () (size 0x8) -> 1df6: void (*)() (size 0x8) -> 1df5: void () (size 0x0)
[...]

     1: long int (size 0x8)
        [0x0] (ID 0x1) (kind 1) long int  (aligned at 0x8, format 0x1, offset:bits 0x0:0x40)
     2: ptrdiff_t (size 0x8) -> 1: long int (size 0x8)
        [0x0] (ID 0x2) (kind 10) ptrdiff_t  (aligned at 0x8)
     3: long unsigned int (size 0x8)
        [0x0] (ID 0x3) (kind 1) long unsigned int  (aligned at 0x8, format 0x0, offset:bits 0x0:0x40)
     4: size_t (size 0x8) -> 3: long unsigned int (size 0x8)
        [0x0] (ID 0x4) (kind 10) size_t  (aligned at 0x8)
     5: int (size 0x4)
        [0x0] (ID 0x5) (kind 1) int  (aligned at 0x4, format 0x1, offset:bits 0x0:0x20)
     6: wchar_t (size 0x4) -> 5: int (size 0x4)
        [0x0] (ID 0x6) (kind 10) wchar_t  (aligned at 0x4)
     7: struct  (size 0x20)
        [0x0] (ID 0x7) (kind 6) struct   (aligned at 0x8)
            [0x0] (ID 0x8) (kind 1) long long int __max_align_ll (aligned at 0x8, format 0x1, offset:bits 0x0:0x40)
            [0x80] (ID 0x9) (kind 2) long double __max_align_ld (aligned at 0x10, format 0x6, offset:bits 0x0:0x80)
     8: long long int (size 0x8)
        [0x0] (ID 0x8) (kind 1) long long int  (aligned at 0x8, format 0x1, offset:bits 0x0:0x40)
     9: long double (size 0x10)
        [0x0] (ID 0x9) (kind 2) long double  (aligned at 0x10, format 0x6, offset:bits 0x0:0x80)
     a: struct  (size 0x20)
        [0x0] (ID 0xa) (kind 6) struct   (aligned at 0x8)
            [0x0] (ID 0x8) (kind 1) long long int __max_align_ll (aligned at 0x8, format 0x1, offset:bits 0x0:0x40)
            [0x80] (ID 0x9) (kind 2) long double __max_align_ld (aligned at 0x10, format 0x6, offset:bits 0x0:0x80)
     b: max_align_t (size 0x20) -> a: struct  (size 0x20)
        [0x0] (ID 0xb) (kind 10) max_align_t  (aligned at 0x8)
            [0x0] (ID 0x8) (kind 1) long long int __max_align_ll (aligned at 0x8, format 0x1, offset:bits 0x0:0x40)
            [0x80] (ID 0x9) (kind 2) long double __max_align_ld (aligned at 0x10, format 0x6, offset:bits 0x0:0x80)
[...]
     dbf5: struct bfd_section * (size 0x8) -> dbf4: struct bfd_section (size 0x6)
        [0x0] (ID 0xdbf5) (kind 3) struct bfd_section *  (aligned at 0x8)
     dbf6: struct bfd_section *() (size 0x0)
        [0x0] (ID 0xdbf6) (kind 5) struct bfd_section *()  (aligned at 0x8)
     dbf7: int () (size 0x0)
        [0x0] (ID 0xdbf7) (kind 5) int ()  (aligned at 0x8)
     dbf8: void () (size 0x0)
        [0x0] (ID 0xdbf8) (kind 5) void ()  (aligned at 0x8)
     dbf9: int () (size 0x0)
        [0x0] (ID 0xdbf9) (kind 5) int ()  (aligned at 0x8)
     dbfa: int () (size 0x0)
        [0x0] (ID 0xdbfa) (kind 5) int ()  (aligned at 0x8)

  Strings:
    0: 
    1: A
    3: AARCH64_ELF_DATA
    14: ALPHA_ELF_DATA
    23: AOUTHDR
    2b: AOUTHDR64
    35: ARC_ELF_DATA
    42: ARM_ELF_DATA
    4f: AUXU
    54: AVR_ELF_DATA
[...]

The 400KiB of CTF (2MiB uncompressed) is basically undeduplicated, hence
the huge number of types and the excessive size before compression.
Once the deduplicator is written (underway now), I expect this to shrink
a *lot*, probably to only a few dozen kilobytes.  (And there is more
shrinkage planned after that, from compression improvements through
to improvements of format compactness.)