Re: [PATCH 00/19] libctf, and CTF support for objdump and readelf

[Nick Alcock <nick dot alcock at oracle dot com>]

On 3 May 2019, Nick Clifton verbalised:

> Hi Nick,
>
>   OK, so I am going to stop my review of this patch series until we have 
>   some answers to a few high level questions/requirements.  Specifically:
>
>   * Hosting
>
>     Is the binutils project the right place for this code ?  As Joseph 
>     has already mentioned libctf appears to use code from the elfutils 
>     project, so maybe that would be a better home for libctf ?

It uses no code from elfutils: It uses one datatype from elfutils, and
shouldn't really even use that (it should use the stuff from <elf.h>
from glibc). (Thanks for spotting that! It'll be fixed this week.)

>   * Testing
>
>     I did not see a testsuite for the library, nor any additions to
>     the binutils testsuite for the extensions to objdump and readelf.
>     I would not be comfortable adding such a large body of code to the
>     project without some 

Agreed!!! Writing a testsuite is high on our priority list, but we need
compiler and linker support first. There will definitely be a testsuite
in place, probably in large part autogenerated. (The code is not at all
untested: the testing largely consists of generating CTF for the entire
Linux kernel and then letting the fairly sizeable DTrace testsuite run
over it. However, this is obviously not suitable for upstreaming into
binutils: something more systematic is needed, and will exist.)

>   * Documentation
>
>     It would be really good to have the CTF format documented somewhere
>     (semi) permanent and publicly accessible.  It would also be good if
>     there was a libctf.texi document describing how consumers are expected
>     to use the library, and, ideally, providing code examples.

Agreed!

>   * Usefulness
>
>     This may be a bit contentious - but is the CTF format actually being
>     used anywhere, or likely to be used in the near future ?  If this is
>     a project that is just going to be used by a small group, or even just
>     a single company, then I am worried that the code will just bit rot 
>     away and not be actively maintained.

It would be useful for many projects, but it was not easy to adopt it,
and it has been relatively unknown outside of Solaris and FreeBSD
circles.

We have been using it for many years with the Linux kernel and DTrace.
Other projects may adopt it. It is *useful* for many other projects, but
it was quite difficult to generate CTF-format data (from DWARF using a
variety of painful-to-port Solaris tools). I know I wanted it in the
past for various other profilers I was writing, but I didn't use it
because I had no idea it existed. It seems likely to be *useful* for
debuggers and tracers and introspectors of all sorts: anything that
wants to know what type an ELF object is, or what type function
arguments or return types are, and who wants to be able to decompose
that type into its constitutent pieces and chase pointers from it and
make sense of the results seems likely to be able to make use of CTF.

In particular, GDB, perf, and sysprof could definitely make use of it,
as could systemtap, rr, and honestly I'm only limited here by not
knowing the names of more obscure debugger projects. Right now these
either cannot do useful things with datatypes at all, or require all the
DWARF debuginfo to be present to do anything, and given that many of
these things are systemwide continuous debuggers, the debuginfo is
almost never present when the debugger points at a victim program and
tries to print out argument info or whatever.

I'm fairly confident that having type information for C programs as
widely and easily available as it is, for, say, Lisp programs (with a
cost, often, of only a few kilobytes) is a generally good thing. We're
making a start by adding CTF support to GCC, GNU ld and GDB.


Size-wise it really is pretty small. Let's try a few samples (with a
notably unscientific construction method). The first five numeric
columns are a count of types, and CTF sizes in bytes (the sizes of
specific sections: all uncompressed sizes but the first 'size' number.
"DWARF size" is the size of .debug_info.)

                       CTF
Program          types size (uncompressed)  stringsize  typesize   DWARF size
coreutils ls     396   10324 (26216)        13148       13068      74241
GAS 2.30         1123  55748 (172731)       106079      66652      1001453
emacs 26.1.50    3546  104276 (284479)      142231      142248     3912261
X.org 1.20.3     7266  152196 (473797)      201421      272376     4163434
GhostScript 9.26 7873  181036 (538901)      243293      295608     7943132
Gtk 3.24.7       9236  208612 (620926)      328174      292752     6106925

So a shrinkage over DWARF of roughly 80%, and usually less than 5% of
the size of the binary whose types are described. With planned format
and generation improvements, I expect that a 90% shrinkage over DWARF is
probably achievable without too much effort. (More than that might
require pruning out individual unused bits, a fairly radical change.)