Found some extra details to add.

- The first time we're reading the symbols for this object we lookup separate debug info via Build ID.
- libdebuginfod is used to download a debug info file which is then passed to symfile_bfd_open which throws an error due to failing bfd_check_format.

So we manage to read in symbols from the executable file, but we error out while trying to read in debug info from a separate debug info file. I assume that this means we skip some of the initialisation of the shared library. Not sure whether that's part of this issue.

I'll try to cook up a smaller reproducer for this.

The main question I have is why re-reading minimal symbols happens.
I guess this 'if' fails to trigger:

  if (objfile->per_bfd->minsyms_read
      && ei->stabsect == NULL
      && ei->mdebugsect == NULL
      && ei->ctfsect == NULL)
    {
      if (symtab_create_debug)
	gdb_printf (gdb_stdlog,
		    "... minimal symbols previously read\n");
      return;
    }

... but for which reason exactly?

That would help make a reproducer, I guess.
You could probably trigger it by making a new inferior
and using 'file' to open the same ELF file again.

Turns out that this is caused by a truncated separate debug info file provided by debuginfod.

---

Reproduction details:

------------------------
// lib.cpp:
void foo(void) {}
------------------------
// main.cpp:
void foo(void);
int main(void) { foo(); return 0; }
------------------------
# Makefile
all:
        c++ -shared lib.cpp -o libtest.so
        c++ -L. main.cpp -ltest -o test
------------------------

Check the Build ID of libtest.so and populate an empty debuginfo file in the debuginfod cache:

$ mkdir -p /home/mhov/.cache/debuginfod_client/cac153df26d5e817739e1b234746cb368da05664/
$ touch /home/mhov/.cache/debuginfod_client/cac153df26d5e817739e1b234746cb368da05664/debuginfo

Now launch the program in GDB and see that it fail to load the separate debug info file. Do a set solib-search-path to trigger a reload of the libraries, and see glibc complain:

$ LD_LIBRARY_PATH=. DEBUGINFOD_URLS=foo gdb --silent ./test
Reading symbols from ./test...
Download failed: No route to host.  Continuing without debug info for /home/mhov/work/GDB29257/./test.
(No debugging symbols found in ./test)
(gdb) start
Temporary breakpoint 1 at 0x1151
Starting program: /home/mhov/work/GDB29257/test
Download failed: No route to host.  Continuing without debug info for /home/mhov/work/GDB29257/system-supplied DSO at 0x7ffff7fce000.
Error while reading shared library symbols for ./libtest.so:
`/home/mhov/.cache/debuginfod_client/cac153df26d5e817739e1b234746cb368da05664/debuginfo': can't read symbols: file format not recognized.

Temporary breakpoint 1, 0x0000555555555151 in main ()
(gdb) set solib-search-path
Reading symbols from ./libtest.so...
Error while reading shared library symbols for ./libtest.so:
`/home/mhov/.cache/debuginfod_client/cac153df26d5e817739e1b234746cb368da05664/debuginfo': can't read symbols: file format not recognized.
Reading symbols from ./libtest.so...
free(): double free detected in tcache 2
Aborted (core dumped)

Sometime I must run set solib-search-path multiple times, and glibc reports different failure modes.

---

Analysis:

When opening a separate debug file via Build ID we normally verify its build id after opening it:

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/build-id.c;hb=ce35d7163e779b1321058b22f005c70ce1524b25#l107

This is part of `find_separate_debug_file_by_buildid`, so if it returns a valid string (path) we can be fairly sure that the object is somewhat valid and it's safe to pass it to `symfile_bfd_open`.

When a build id file has been provided by debuginfod, however, we don't use `find_separate_debug_file_by_buildid`, and we first check it with `build_id_verify` after already having tried to open it with `symfile_bfd_open`.

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/elfread.c;hb=ce35d7163e779b1321058b22f005c70ce1524b25#l1329


This means that the failure modes of a truncated separate debug info file are different depending on whether the debug info file was found "locally" or via the debuginfod cache.

- For the "local" case the failure mode is that `find_separate_debug_file_by_buildid` returns an empty string.
- For the debuginfod case the failure mode is that `symfile_bfd_open` raises an exception.

The exception is caught here:

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/solib.c;hb=ce35d7163e779b1321058b22f005c70ce1524b25#l696

That means that we got interrupted at this point in the sequence of loading symbols:

```
solib_read_symbols
    ...
    read_symbols
        elf_symfile_read
            elf_read_minimal_symbols  (this function completes fully)
            symfile_bfd_open          (for separate debug file fetched by debuginfod)
                throw                 (exception is thrown due to truncated separate debug file)

        (this is where minsyms_read would have been set, but it's skipped by the non-local exit)
    catch exception
```


Since elf_read_minimal_symbols completed successfully we have in fact read in minimal symbols. However, the exception that is caused by the truncated separate debug file prevents read_symbols from tracking this in objfile->per_bfd->minsyms_read.

---

Ideas of ways to fix this:

- Change the order of `symbfile_bfd_open` and `build_id_verify` for files provided from debuginfod. This would prevent `symfile_bfd_open` from being called with a corrupt file and thus prevent the exception from being raised.
- Catch this kind of exception when trying to open the file provided from debuginfod.
- Mark minimal symbols as read (objfile->per_bfd->minsyms_read = true) as soon as elf_read_minimal_symbols has completed. I'm not confident about this. There could be some other state that must also have been completed before we want to set this to true?

Presumably, an empty debuginfo file could end up in the debuginfod cache if you interrupt a download at the right time?

Hi Magne, which version of elfutils are you using? I was only able to reproduce this using an older package (elfutils-debuginfod-client-0.186-1.fc35.x86_64) that uses 000-permission files to represent negative-cache-hits. When you used touch to create the file it didn't have 000 permissions, so debuginfod returns an empty file to gdb as if it's valid. Without manually adding the cache file this situation should not occur.

More recent versions of debuginfod use empty files to represent negative-cache-hits. In this case I can't reproduce the bug because debuginfod correctly doesn't return the empty file to gdb.

(It looks like F36 contains an update version of elfutils-debuginfod-client but F35 still provides an older 000-permisssion build, we should probably update F35's elfutils-debuginfod-client.)

Hi Aaron, 

Thanks for your reply. We're using elfutils-0.186 and it sounds like debuginfodclient switching over to using empty files as miss markers would explain why we see this issue. If mixing use of GDB built with 0.186 and 0.187 then the former will interpret the cache misses from the latter as valid entries.

Sticking to only using one version of debuginfodclient would solve this, but I think there might still be an argument for patching GDB to deal better with being given an empty debug info file.

For our use case we might want to ensure that we're using a custom DEBUGINFOD_CACHE_PATH as we often run on machines alongside other builds of GDB.

> - Mark minimal symbols as read (objfile->per_bfd->minsyms_read = true) as soon as elf_read_minimal_symbols has completed. I'm not confident about this. There could be some other state that must also have been completed before we want to set this to true?

I think the problem with this is that minsyms are implemented weirdly.
There may be multiple sources of minsyms in a given objfile.
Now, probably how this should be done is each reader should make
its own object.  However, what's actually done is that they all
write to the same basic data structure, so it isn't marked as
completed until the very end.

It looks like the debuginfod code thinks that symfile_bfd_open
won't throw but will instead return null:

		  gdb_bfd_ref_ptr debug_bfd (symfile_bfd_open (symfile_path.get ()));

		  if (debug_bfd == nullptr)
		    warning (_("File \"%s\" from debuginfod cannot be opened as bfd"),
			     filename);

... but that's wrong.

I have a patch to catch the exception.

Created attachment 14485 [details]
test patch

Could you try this patch?

(In reply to Tom Tromey from comment #9)
> Created attachment 14485 [details]
> test patch
> 
> Could you try this patch?

LGTM

Thanks Tom, your patch fixes the issue on my machine when applied on top of current master.

https://sourceware.org/pipermail/gdb-patches/2022-December/194650.html

*** Bug 30339 has been marked as a duplicate of this bug. ***

The master branch has been updated by Tom Tromey <tromey@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=f96328accde1e6302b62aa880675594618079cb3

commit f96328accde1e6302b62aa880675594618079cb3
Author: Tom Tromey <tromey@adacore.com>
Date:   Tue Dec 6 12:07:12 2022 -0700

    Avoid double-free with debuginfod
    
    PR gdb/29257 points out a possible double free when debuginfod is in
    use.  Aside from some ugly warts in the symbol code (an ongoing
    issue), the underlying issue in this particular case is that elfread.c
    seems to assume that symfile_bfd_open will return NULL on error,
    whereas in reality it throws an exception.  As this code isn't
    prepared for an exception, bad things result.
    
    This patch fixes the problem by introducing a non-throwing variant of
    symfile_bfd_open and using it in the affected places.
    
    Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=29257

Fixed.