Bisect to:
...
a2fedca99c622e1b523046d09f573b06de0207a6 is the first bad commit
commit a2fedca99c622e1b523046d09f573b06de0207a6
Author: Philippe Waroquiers <philippe.waroquiers@skynet.be>
Date:   Sat Dec 21 10:55:11 2019 +0100

    Implement 'set/show exec-file-mismatch'.
...

Fixed by:
...
diff --git a/gdb/testsuite/gdb.server/solib-list.exp b/gdb/testsuite/gdb.server/solib-list.exp
index 5471734e031..45638894f0b 100644
--- a/gdb/testsuite/gdb.server/solib-list.exp
+++ b/gdb/testsuite/gdb.server/solib-list.exp
@@ -47,6 +47,7 @@ foreach nonstop { 0 1 } { with_test_prefix "non-stop $nonstop" {
     # for ${interp_system}.
     gdb_exit
     gdb_start
+    gdb_test_no_output "set exec-file-mismatch off"
     gdb_reinitialize_dir $srcdir/$subdir
     gdb_load_shlib ${binlibfile}
 
...

Strange ...
I see no failure on debian 10.2:
...
(gdb) PASS: gdb.server/solib-list.exp: non-stop 1: file binfile
target remote localhost:2347
Remote debugging using localhost:2347
warning: Mismatch between current exec-file /bd/home/philippe/gdb/git/build_binutils-gdb/gdb/testsuite/outputs/gdb.server/solib-list/solib-list
and automatically determined exec-file /lib/x86_64-linux-gnu/ld-2.28.so
exec-file-mismatch handling is currently "ask"
Reading symbols from /lib/x86_64-linux-gnu/ld-2.28.so...
Reading symbols from /usr/lib/debug/.build-id/f2/5dfd7b95be4ba386fd71080accae8c0732b711.debug...

Program stopped.
0x00007ffff7fd6090 in _start ()
...
		=== gdb Summary ===

# of expected passes		16


Also, it looks strange that the automatically determined exec-file
is /lib64/ld-2.26.so
(that sounds a somewhat strange executable).

Apart of that, the 'set exec-file-mismatch' setting was done
to allow the user to choose the behaviour so the fix looks reasonable
to me.

(In reply to philippe.waroquiers from comment #3)
> Strange ...
> I see no failure on debian 10.2:
> ...
> (gdb) PASS: gdb.server/solib-list.exp: non-stop 1: file binfile
> target remote localhost:2347
> Remote debugging using localhost:2347
> warning: Mismatch between current exec-file
> /bd/home/philippe/gdb/git/build_binutils-gdb/gdb/testsuite/outputs/gdb.
> server/solib-list/solib-list
> and automatically determined exec-file /lib/x86_64-linux-gnu/ld-2.28.so
> exec-file-mismatch handling is currently "ask"

So, if there is a mismatch, and the handling is ask ... where is the asking being done?

If that's missing in your log, does that not indicate a bug in exec-file-mismatch?

> Reading symbols from /lib/x86_64-linux-gnu/ld-2.28.so...
> Reading symbols from
> /usr/lib/debug/.build-id/f2/5dfd7b95be4ba386fd71080accae8c0732b711.debug...
> 
> Program stopped.
> 0x00007ffff7fd6090 in _start ()
> ...
> 		=== gdb Summary ===
> 
> # of expected passes		16
> 
> 
> Also, it looks strange that the automatically determined exec-file
> is /lib64/ld-2.26.so
> (that sounds a somewhat strange executable).
> 

Yes, but that's explicitly done by the test-case, and that's the same in both logs.

Created attachment 12239 [details]
offending gdb.log

Created attachment 12240 [details]
gdb.log with test-case fix mentioned in comment 2

(In reply to Tom de Vries from comment #4)
> (In reply to philippe.waroquiers from comment #3)
> > Strange ...
> > I see no failure on debian 10.2:

Can you attach the entire gdb.log here as well?

Created attachment 12244 [details]
gdb.log of a succesful run of gdb.server/solib-list.exp on Debian

Hi Tom,

I have attached the gdb.log of the successful run.

Following your remark about why there was no question
(and no reload) of the newly detected file in what I attached,
I debugged gdb.

The fact that I do not see a question on my system is due to the
below code in symfile.c: symbol_file_add_with_addrs

  /* Give user a chance to burp if we'd be
     interactively wiping out any existing symbols.  */

  if ((have_full_symbols () || have_partial_symbols ())
      && mainline
      && from_tty
      && !query (_("Load new symbol table from \"%s\"? "), name))
    error (_("Not confirmed."));

I see on my system when loading solib-list in gdb:
(No debugging symbols found in ..../gdb.server/solib-list/solib-list)

I suppose that the 'No debugging symbols' makes the first test fail.
So, the "Load" question is not asked, and the file is loaded unconditionally.

I suppose that on your system, solib-list contains some debug symbols,
which means the "Load" question is asked, and so the test fails for you due
to this.

(In reply to philippe.waroquiers from comment #9)
> Hi Tom,
> 
> I have attached the gdb.log of the successful run.
> 
> Following your remark about why there was no question
> (and no reload) of the newly detected file in what I attached,
> I debugged gdb.
> 
> The fact that I do not see a question on my system is due to the
> below code in symfile.c: symbol_file_add_with_addrs
> 
>   /* Give user a chance to burp if we'd be
>      interactively wiping out any existing symbols.  */
> 
>   if ((have_full_symbols () || have_partial_symbols ())
>       && mainline
>       && from_tty
>       && !query (_("Load new symbol table from \"%s\"? "), name))
>     error (_("Not confirmed."));
> 
> I see on my system when loading solib-list in gdb:
> (No debugging symbols found in ..../gdb.server/solib-list/solib-list)
> 
> I suppose that the 'No debugging symbols' makes the first test fail.
> So, the "Load" question is not asked, and the file is loaded unconditionally.
> 
> I suppose that on your system, solib-list contains some debug symbols,
> which means the "Load" question is asked, and so the test fails for you due
> to this.

Thanks for the analysis. I can confirm that the solib-list executable does contain some debug symbols for me, and using this patch:
...
diff --git a/gdb/testsuite/gdb.server/solib-list.exp b/gdb/testsuite/gdb.server/solib-list.exp
index 5471734e03..d5d73dd630 100644
--- a/gdb/testsuite/gdb.server/solib-list.exp
+++ b/gdb/testsuite/gdb.server/solib-list.exp
@@ -38,6 +38,8 @@ if { [get_compiler_info]
     return -1
 }
 
+exec strip --strip-debug ${binfile}
+
 set interp_system [section_get ${binfile} .interp]
 verbose -log "system interpreter is: $interp_system"
 
...
I get the same results as you.

(In reply to Tom de Vries from comment #10)
> (In reply to philippe.waroquiers from comment #9)
> > Hi Tom,
> > 
> > I have attached the gdb.log of the successful run.
> > 
> > Following your remark about why there was no question
> > (and no reload) of the newly detected file in what I attached,
> > I debugged gdb.
> > 
> > The fact that I do not see a question on my system is due to the
> > below code in symfile.c: symbol_file_add_with_addrs
> > 
> >   /* Give user a chance to burp if we'd be
> >      interactively wiping out any existing symbols.  */
> > 
> >   if ((have_full_symbols () || have_partial_symbols ())
> >       && mainline
> >       && from_tty
> >       && !query (_("Load new symbol table from \"%s\"? "), name))
> >     error (_("Not confirmed."));
> > 
> > I see on my system when loading solib-list in gdb:
> > (No debugging symbols found in ..../gdb.server/solib-list/solib-list)
> > 
> > I suppose that the 'No debugging symbols' makes the first test fail.
> > So, the "Load" question is not asked, and the file is loaded unconditionally.
> > 
> > I suppose that on your system, solib-list contains some debug symbols,
> > which means the "Load" question is asked, and so the test fails for you due
> > to this.
> 
> Thanks for the analysis. I can confirm that the solib-list executable does
> contain some debug symbols for me, and using this patch:
> ...
> diff --git a/gdb/testsuite/gdb.server/solib-list.exp
> b/gdb/testsuite/gdb.server/solib-list.exp
> index 5471734e03..d5d73dd630 100644
> --- a/gdb/testsuite/gdb.server/solib-list.exp
> +++ b/gdb/testsuite/gdb.server/solib-list.exp
> @@ -38,6 +38,8 @@ if { [get_compiler_info]
>      return -1
>  }
>  
> +exec strip --strip-debug ${binfile}
> +
>  set interp_system [section_get ${binfile} .interp]
>  verbose -log "system interpreter is: $interp_system"
>  
> ...
> I get the same results as you.

And conversely, you should be able to see the FAIL I'm seeing by adding "debug" in the gdb_compile flags:
...
diff --git a/gdb/testsuite/gdb.server/solib-list.exp b/gdb/testsuite/gdb.server/solib-list.exp
index 5471734e03..2f1f3a2bbb 100644
--- a/gdb/testsuite/gdb.server/solib-list.exp
+++ b/gdb/testsuite/gdb.server/solib-list.exp
@@ -33,7 +33,7 @@ set binlibfile [standard_output_file ${testfile}.so]
 
 if { [get_compiler_info]
      || [gdb_compile_shlib "${srcdir}/${subdir}/${srclibfile}" "${binlibfile}" {debug}] != ""
-     || [gdb_compile "${srcdir}/${subdir}/${srcfile}" "${binfile}" executable "shlib=${binlibfile}"] != "" } 
{
+     || [gdb_compile "${srcdir}/${subdir}/${srcfile}" "${binfile}" executable "shlib=${binlibfile} debug"] != "" } {
     untested "failed to compile"
     return -1
 }
...

(In reply to Tom de Vries from comment #4)
> (In reply to philippe.waroquiers from comment #3)
> > Strange ...
> > I see no failure on debian 10.2:
> > ...
> > (gdb) PASS: gdb.server/solib-list.exp: non-stop 1: file binfile
> > target remote localhost:2347
> > Remote debugging using localhost:2347
> > warning: Mismatch between current exec-file
> > /bd/home/philippe/gdb/git/build_binutils-gdb/gdb/testsuite/outputs/gdb.
> > server/solib-list/solib-list
> > and automatically determined exec-file /lib/x86_64-linux-gnu/ld-2.28.so
> > exec-file-mismatch handling is currently "ask"
> 
> So, if there is a mismatch, and the handling is ask ... where is the asking
> being done?
> 
> If that's missing in your log, does that not indicate a bug in
> exec-file-mismatch?
> 

Having figured out the cause of the differences in the logs, I still think this question is relevant.  Posing the same question a bit more elaborately ...

So, looking at the exec-file-mismatch docs:
...
@anchor{set exec-file-mismatch}
If the debugger can determine the name of the executable file running
in the process it is attaching to, and this file name does not match
the name of the current exec-file loaded by @value{GDBN}, the option
@code{exec-file-mismatch} specifies how to handle the mismatch.
...

as well as the 'ask' bit:
...
If @samp{ask}, the default, display a warning
and ask the user whether to load the process executable file;
...

and then looking at the events in the gdb.log:
- gdb can determine the name of the executable file running in the process
  it is attaching to
- gdb can determine that that filename does not match the name of the current
  exec-file loaded by gdb
- gdb reports this mismatch
- gdb reports the mismatch handling strategy: ask
- gdb doesn't ask
I'd say this is a bug in exec-file-mismatch.

Do you agree with this analysis?

(In reply to Tom de Vries from comment #0)
> (gdb) PASS: gdb.server/solib-list.exp: non-stop 0: file binfile
> target remote localhost:2346
> Remote debugging using localhost:2346
> warning: Mismatch between current exec-file
> /data/gdb_versions/devel/build/gdb/testsuite/outputs/gdb.server/solib-list/
> solib-list
> and automatically determined exec-file /lib64/ld-2.26.so
> exec-file-mismatch handling is currently "ask"
> Load new symbol table from "/lib64/ld-2.26.so"? (y or n) n

I've submitted a patch ( https://gnutoolchain-gerrit.osci.io/r/c/binutils-gdb/+/761 ) to answer "y" here, to fix the failure on openSUSE.

Up to you if you prefer to fix this by detecting the question
and answering, or by your first proposal to set exec-file-mismatch
to off (the second approach is simpler).

Note that I quickly discussed this problem with Pedro at FOSDEM.
He suggested we might remove the condition regarding debug info,
so that the question is asked whatever the current executable status
(with or without debug info).
I will further investigate that to see if there was a very specific
reason to have this condition.

I've fixed the FAIL here by working around the issue in the test-case ( https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=6b9374f1e07cb250736815ff8db263199416adc6 ).

Leaving the PR open for the gdb issue.

I evaluated the change to ask the question independently
of the fact that the current loaded file has symbols, i.e. removing
the following part of the condition :
  (have_full_symbols () || have_partial_symbols ())
to ask the "Load new symbol table" question.

This seems to be a significant change in behaviour that triggers
about 1200 new failures in the gdb testsuite.
The ratio (cost+risk)/benefit looks poor for this change.

Another possible way to fix this is to replace the value "ask"
by "load" (and explain in the doc that typically, a question is asked
before loading the new file):

  (gdb) h set exec-file-mismatch 
  Set exec-file-mismatch handling (load|warn|off).
  Specifies how to handle a mismatch between the current exec-file name
  loaded by GDB and the exec-file name automatically determined when attaching
  to a process:

   load - warn the user and load the determined exec-file.
   warn - warn the user, but do not change the exec-file.
   off  - do not check for mismatch.
  (gdb) 


WDYT ?

Philippe proposed we put this one on the list of issues we should consider
for the GDB 10.1 release.

At the moment, I am confused about the relationship between the problem
being reported, the portion of code being discussed, and the CLI change
proposal.
 
Do we understand why GDB thought the executable as ld-2.26.so (the loader)?
I have a suspicion that this is the important question, is it not. In other
words, yes, you guys probably found a bug where GDB is not asking confirmation when it should, but isn't the crux of the matter the fact that GDB thinks
the executable is now ld-2.26.so, which of course doesn't match the executable
it was given. If GDB thought instead that the executable was still `solib-list',
I assume that we wouldn't get better an error about matching executables.

Did I misunderstand what the test is doing? If I did, considering Philippe's
results on his experiment, my knee-jerk reaction is that this is the kind of
change that we should consider carefully before putting it in a release.
The bug itself seems relatively minor.

(In reply to Joel Brobecker from comment #17)
> Do we understand why GDB thought the executable as ld-2.26.so (the loader)?

I suspect that is due to the test-case.  It does things like:
...
set interp_system [section_get ${binfile} .interp]
set res [gdbserver_spawn "${interp_system} ${remote_binfile}"]
...

(In reply to Joel Brobecker from comment #17)
> Philippe proposed we put this one on the list of issues we should consider
> for the GDB 10.1 release.
> 
> At the moment, I am confused about the relationship between the problem
> being reported, the portion of code being discussed, and the CLI change
> proposal.
>  
> Do we understand why GDB thought the executable as ld-2.26.so (the loader)?
> I have a suspicion that this is the important question, is it not. In other
> words, yes, you guys probably found a bug where GDB is not asking
> confirmation when it should, but isn't the crux of the matter the fact that
> GDB thinks
> the executable is now ld-2.26.so, which of course doesn't match the
> executable
> it was given. If GDB thought instead that the executable was still
> `solib-list',
> I assume that we wouldn't get better an error about matching executables.
> 
> Did I misunderstand what the test is doing? If I did, considering Philippe's
> results on his experiment, my knee-jerk reaction is that this is the kind of
> change that we should consider carefully before putting it in a release.
> The bug itself seems relatively minor.

I think we have 2 (unrelated) aspects:
* In this test, GDB determines that the exec file is ld-2.26.so
  I think this is normal: the test explicitly tells to start 
  this "loader" exec file.  See a.o. comment:
    # Unset last_loaded_file because we want GDBserver spawn
    # ${interp_system} not ${last_loaded_file}.

* Following build-id mismatch, GDB loads the new exec file without asking
  a question, as the current exec-file has no debug symbol.

This can be reproduced with "normal" exec files not using gdbserver.
See log below.

So, this bug is only about the fact that exec-file-mismatch "ask"
setting indicates it will ask a question, but no question is asked
when the current exec file has no debug info.

So, we have various ways to handle this bug:
  * modify GDB to always ask a question, even when current exec file has
    no debug info.
    That made a lot of tests fail, so not attractive.
  * keep current behaviour, but just clarify in the doc and help
    that "ask" asks a question only when current exec file has debug info.
  * replace in the 'exec-file-mismatch' setting "ask" by "load"
    and similarly change the doc to explain that a question is asked
    if the current exec file has debug info, otherwise the new file
    is loaded unconditionally.

Philippe



(gdb) atta 16623   <<<<<<<<<<<<<<<<  this runs stripped_sleepers,
                   <<<<<<<<<<<<<<<<  build-id differs from sleepers used by second attach
Attaching to process 16623
...
(gdb) detach
Detaching from program: /home/philippe/gdb/littleprogs/stripped_sleepers, process 16623
[Inferior 1 (process 16623) detached]
(gdb) atta 358
Attaching to program: /home/philippe/gdb/littleprogs/stripped_sleepers, process 358
[New LWP 360]
[New LWP 361]
[New LWP 362]
warning: Mismatch between current exec-file /home/philippe/gdb/littleprogs/stripped_sleepers
and automatically determined exec-file /home/philippe/valgrind/git/trunk_untouched/gdbserver_tests/sleepers
exec-file-mismatch handling is currently "ask"
<<<<<<<<<<<<<<<<<<<  No question here, GDB just loads the sleepers exec file
Reading symbols from /home/philippe/valgrind/git/trunk_untouched/gdbserver_tests/sleepers...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f1bd4c88037 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x55bd26e491a0 <t>)
    at ../sysdeps/unix/sysv/linux/select.c:41
41	../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb)

> * Following build-id mismatch, GDB loads the new exec file without asking
>   a question, as the current exec-file has no debug symbol.

I had to dig into the code to understand the context better, so let me share
that here:

The question that we would expect if exec-file-mismatch is set to "ask"
is actually the same question we already had before and get when loading
a new executable in place of an existing one. More precisely, what I am
seeing validate_exec_file do is:
    1. generate a warning about the discrepancy; and then
    2. if exec-file-mismatch is set to "ask", call symbol_file_add_main

That explains why Philippe was looking at the "Load new symbol table"
check in symbol_file_add_with_addrs.


> This can be reproduced with "normal" exec files not using gdbserver.
> See log below.
>
> So, this bug is only about the fact that exec-file-mismatch "ask"
> setting indicates it will ask a question, but no question is asked
> when the current exec file has no debug info.
>
> So, we have various ways to handle this bug:
>   * modify GDB to always ask a question, even when current exec file has
>     no debug info.
>     That made a lot of tests fail, so not attractive.
>   * keep current behaviour, but just clarify in the doc and help
>     that "ask" asks a question only when current exec file has debug info.
>   * replace in the 'exec-file-mismatch' setting "ask" by "load"
>     and similarly change the doc to explain that a question is asked
>     if the current exec file has debug info, otherwise the new file
>     is loaded unconditionally.

I think we should widen this discussion to a bigger audience, so I'd like
to propose you guys move this to gdb-patches, and publish a link to
the discussion here.

Regarding option (1), I think the impact on the testsuite is a consideration,
but not necessarily a strong one. It might be strong if the failures
it demonstrates provides an explanation to the current behavior. Otherwise,
we should review this proposed change by asking ourselves whether it would
make better sense to always ask the question or not.

For me, option (2) would certainly be better than nothing, but wouldn't
take away the fact that the current behavior is confusing.

For option (3), changing it to "load" doesn't make sense to me, because
then what it feels like to me is that "load" and "warn" are identifical.

I have two alternative proposals:

  (4) Change exec-file-mismatch ask/load/off with an on/off switch
      This is option 3 changed to remove the "ask" option

  (5) Update symbol_file_add_with_addrs to accept an addition flag, which
      forces the question, and pass that flag when called from
      validate_exec_file and exec-file-mismatch is set to "ask"
      (but only if stdin is a tty, of course)

Personally, I think it's nice that users have the opportunity to confirm
if GDB detects a mismatch, so (5) has my preference.

(In reply to Joel Brobecker from comment #20)
>   (5) Update symbol_file_add_with_addrs to accept an addition flag, which
>       forces the question, and pass that flag when called from
>       validate_exec_file and exec-file-mismatch is set to "ask"
>       (but only if stdin is a tty, of course)
> 
> Personally, I think it's nice that users have the opportunity to confirm
> if GDB detects a mismatch, so (5) has my preference.
Looks a good idea.  I will prepare a patch and post it on gdb-patches,
referencing this PR.

(In reply to philippe.waroquiers from comment #21)
> (In reply to Joel Brobecker from comment #20)
> >   (5) Update symbol_file_add_with_addrs to accept an addition flag, which
> >       forces the question, and pass that flag when called from
> >       validate_exec_file and exec-file-mismatch is set to "ask"
> >       (but only if stdin is a tty, of course)
> > 
> > Personally, I think it's nice that users have the opportunity to confirm
> > if GDB detects a mismatch, so (5) has my preference.
> Looks a good idea.  I will prepare a patch and post it on gdb-patches,
> referencing this PR.

Submitted patch in :
https://sourceware.org/pipermail/gdb-patches/2020-June/169773.html

The master branch has been updated by Philippe Waroquiers <philippe@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=a8654e7d784980cb4596f685964200fcc1164c78

commit a8654e7d784980cb4596f685964200fcc1164c78
Author: Philippe Waroquiers <philippe.waroquiers@skynet.be>
Date:   Sun Jun 21 21:26:25 2020 +0200

    Fixes PR 25475: ensure exec-file-mismatch "ask" always asks in case of mismatch.
    
    As explained in https://sourceware.org/bugzilla/show_bug.cgi?id=25475,
    when the currently loaded file has no debug symbol,
    symbol_file_add_with_addrs does not ask a confirmation to the user
    before loading the new symbol file.  The behaviour is not consistent
    when symbol_file_add_with_addrs is called due to exec-file-mismatch "ask"
    setting.
    
    The PR discusses several solutions/approaches.
    The preferred approach (suggested by Joel) is to ensure that GDB always asks
    a confirmation when it loads a new symbol file due to exec-file-mismatch,
    using a new SYMFILE add-flag.
    
    I tested this manually.  If OK, we can remove the bypass introduced by Tom
    in 6b9374f1, in order to always answer to the 'load' question.
    
    gdb/ChangeLog
    2020-06-24  Philippe Waroquiers  <philippe.waroquiers@skynet.be>
    
            * symfile-add-flags.h: New flag SYMFILE_ALWAYS_CONFIRM.
            * exec.c (validate_exec_file): If from_tty, set both
            SYMFILE_VERBOSE (== from_tty) and SYMFILE_ALWAYS_CONFIRM.
            * symfile.c (symbol_file_add_with_addrs): if always_confirm
            and from_tty, unconditionally ask a confirmation.

fixed by a8654e7d784980cb4596f685964200fcc1164c78

Well done, Philippe, and thank you!