Bug 29478 - Slow query of a debuginfo in a big rpm with very many files
Summary: Slow query of a debuginfo in a big rpm with very many files
Status: RESOLVED NOTABUG
Alias: None
Product: elfutils
Classification: Unclassified
Component: debuginfod (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-08-12 08:58 UTC by Martin Liska
Modified: 2022-08-12 12:18 UTC (History)
3 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Liska 2022-08-12 08:58:10 UTC
Consider a very common test case where you debug a binary and the binary depends on linux-vdso.so.1, well, actually each Linux binary depends on it.

$ file /usr/lib/modules/5.19.0-1-default/vdso/vdso64.so
/usr/lib/modules/5.19.0-1-default/vdso/vdso64.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=6c5e8eefd4b928a12d9eec149ba8eab49b3febcc, stripped

it belongs to the following debuginfo package:

$ du -hs ~/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm 
1.6G	/home/marxin/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm

and we (openSUSE) use bzip2 compression for it (mainly due to compatibility with older system where one can install a recent kernel).

Anyway it contains quite many files:

$ rpm -qpl ~/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm | wc -l
15906

and it occupies 5.5 GB when extracted.

$ DEBUGINFOD_VERBOSE=1 gdb `which htop`
GNU gdb (GDB; openSUSE Tumbleweed) 12.1
...
url 0 https://debuginfod.opensuse.org/buildid/6c5e8eefd4b928a12d9eec149ba8eab49b3febcc/debuginfo
query 1 urls in parallel
server response HTTP response code said error
url 0 The requested URL returned error: 504
not found No such file or directory (err=-2)

So as seen, there's a timeout of the request. Apparently, libarchive uncompresses the content of each entry:

perf confirms that:

    23.14%  libbz2.so.1.0.6   [.] BZ2_bzDecompress
     4.86%  libbz2.so.1.0.6   [.] 0x0000000000015f62
     2.21%  [kernel]          [k] virtnet_poll
     1.86%  [kernel]          [k] __raw_callee_save___pv_queued_spin_unlock

and I can reproduce it locally with:

$ cat /tmp/demo.c
#include <archive.h>
#include <archive_entry.h>
#include <stdlib.h>

int main()
{
  struct archive *a;
  struct archive_entry *entry;
  int r;

  a = archive_read_new();
  archive_read_support_filter_all(a);
  archive_read_support_format_all(a);
  r = archive_read_open_filename(a, "/home/marxin/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm", 10240); // Note 1
  if (r != ARCHIVE_OK)
    exit(1);
  while (archive_read_next_header(a, &entry) == ARCHIVE_OK) {
    printf("%s\n",archive_entry_pathname(entry));
    archive_read_data_skip(a);  // Note 2
  }
  r = archive_read_free(a);  // Note 3
  if (r != ARCHIVE_OK)
    exit(1);
}

Note rpm -qpl finishes very quickly:

$ time rpm -qpl ~/BIG/test/kernel-default-debuginfo-5.19.0-1.1.x86_64.rpm >/dev/null

real	0m0.064s
user	0m0.060s
sys	0m0.004s

If I see correctly, the function handle_buildid_r_match skips all entries different from b_source1, but it's really slow
due to mentioned libarchive slownes.
Comment 1 Frank Ch. Eigler 2022-08-12 11:23:48 UTC
Unfortunately, we're not aware of an easy way around this.  There is no way to seek in the compressed files to an arbitrary index, so one is stuck reading from the beginning to the file of interest.

I think that, unless the kernel packaging changes, one needs to rely on (trigger!) debuginfod to fetch & cache highly likely files such as the vdso*.  Some work is underway to allow this automation, but until then, make sure your debuginfod server has a large enough fdcache, and you don't groom too frequently (which clears out the fdcache).

See also https://bugzilla.redhat.com/show_bug.cgi?id=1970578
Comment 2 Martin Liska 2022-08-12 12:18:50 UTC
Thanks for the reply, after reading of the RH bug, I can see you faced the very same problem as me ;)