libctf: improving the compressor, optionally-available algorithms

[Nick Alcock]

So in the next few weeks I'll be starting work on the next format
revision of libctf, v4, bringing back some of the compactness we lost
when the format's various limits were lifted and adding some new
features.

One of the changes I'm considering making for format v4 is an
improvement in compression algorithm. Right now all but the smallest CTF
sections are compressed with zlib, which is ubiquitous, in the source
tree... and notably worse than the current crop of algorithms. I'm
wondering if I should add better options: probably xz (for when
compactness matters above all else, including linking speed) and zstd
(by default). But of course neither of these are in the source tree, so
they might not be present in consumers' libctfs.

Now these libraries are so commonplace that they might as well be
ubiquitous these days, especially zstd: so I was thinking of handling
this by compressing with zstd by default (if compiled in), and providing
a linker option (and corresponding new libctf API function) to allow the
user to pick compression algorithm if they suspect their users might be
restricted to systems without zstd etc linked against libctf.

Something like --ctf-compression={xz,zstd,zlib}.

That way users like distro builders who *know* what their users are
using, want compactness, and are willing to pay the link-time cost can
specify -Wl,--ctf-compression=xz, users who are shipping software that
might have to run anywhere against a libctf that might be linked with
nothing at all can specify -Wl,--ctf-compression=zlib, and the majority
can probably say nothing, assume that zstd is available, and get good
results, on the grounds that basically all distros have it and gdb etc
are using it so the dev headers are probably widely installed on build
systems that build binutils anyway.

Attempts to open CTF sections that use an unsupported compression
algorithm would return a new error code to indicate as much. (And,
actually, I could have libctf try to compensate when an unsupported
algorithm is spotted by trying to dlopen the compression library needed
if it's not linked in: only a couple of functions are needed to
decompress xz and zstd, and dlsyming those doesn't seem to be too hard.
Then we'd only fail to open on platforms without dlsym or without (say)
libzstd.so.1 available at runtime, whether or not the devel headers were
installed. I think it reasonable to require the devel headers to
*compress* CTF with a particular algorithm: that raises no compatibility
concerns.)

I wouldn't propose this if the gains didn't seem so substantial: but I
suspect that moving to a better compressor is a better single-shot
improvement in compactness than any other single thing I'm going to do
to the format, and a format rev is the right time to do it.

To anyone concerned: does this seem sensible? Is it actually totally
impractical to assume that anything other than zlib is available ever?
(I hope not).