libctf: improving the compressor, optionally-available algorithms

[Matthias Klose]

On 2/5/21 1:17 PM, Nick Alcock via Binutils wrote:
> So in the next few weeks I'll be starting work on the next format
> revision of libctf, v4, bringing back some of the compactness we lost
> when the format's various limits were lifted and adding some new
> features.
> 
> One of the changes I'm considering making for format v4 is an
> improvement in compression algorithm. Right now all but the smallest CTF
> sections are compressed with zlib, which is ubiquitous, in the source
> tree... and notably worse than the current crop of algorithms. I'm
> wondering if I should add better options: probably xz (for when
> compactness matters above all else, including linking speed) and zstd
> (by default). But of course neither of these are in the source tree, so
> they might not be present in consumers' libctfs.
> 
> Now these libraries are so commonplace that they might as well be
> ubiquitous these days, especially zstd: so I was thinking of handling
> this by compressing with zstd by default (if compiled in), and providing
> a linker option (and corresponding new libctf API function) to allow the
> user to pick compression algorithm if they suspect their users might be
> restricted to systems without zstd etc linked against libctf.
> 
> Something like --ctf-compression={xz,zstd,zlib}.
> 
> That way users like distro builders who *know* what their users are
> using, want compactness, and are willing to pay the link-time cost can
> specify -Wl,--ctf-compression=xz, users who are shipping software that
> might have to run anywhere against a libctf that might be linked with
> nothing at all can specify -Wl,--ctf-compression=zlib, and the majority
> can probably say nothing, assume that zstd is available, and get good
> results, on the grounds that basically all distros have it and gdb etc
> are using it so the dev headers are probably widely installed on build
> systems that build binutils anyway.
> 
> Attempts to open CTF sections that use an unsupported compression
> algorithm would return a new error code to indicate as much. (And,
> actually, I could have libctf try to compensate when an unsupported
> algorithm is spotted by trying to dlopen the compression library needed
> if it's not linked in: only a couple of functions are needed to
> decompress xz and zstd, and dlsyming those doesn't seem to be too hard.
> Then we'd only fail to open on platforms without dlsym or without (say)
> libzstd.so.1 available at runtime, whether or not the devel headers were
> installed. I think it reasonable to require the devel headers to
> *compress* CTF with a particular algorithm: that raises no compatibility
> concerns.)
> 
> I wouldn't propose this if the gains didn't seem so substantial: but I
> suspect that moving to a better compressor is a better single-shot
> improvement in compactness than any other single thing I'm going to do
> to the format, and a format rev is the right time to do it.
> 
> To anyone concerned: does this seem sensible? Is it actually totally
> impractical to assume that anything other than zlib is available ever?
> (I hope not).

GCC currently supports zstd for it's streaming format, and what I know from
compressing distro packages, zstd is faster for decompressing with slightly
bigger compressed data.  Also as today, gdb doesn't support zstd.

Matthias