☠ Buildbot (Sourceware): binutils-gdb - failed test (failure) test (failure) (master)

[Andrew Burgess]

Mark Wielaard <mark@klomp.org> writes:

> Hi Andrew,
>
> On Fri, 2024-06-14 at 22:04 +0100, Andrew Burgess wrote:
>> Mark Wielaard <mark@klomp.org> writes:
>> > It looks like you latest patch caused breakage on i386 with
>> > RUNTESTFLAGS=--target_board=native-gdbserver and --target_board=native-
>> > extended-gdbserver but a "native" make check-gdb is fine as are all
>> > other arches.
>> 
>> I saw the buildbot report for these failures, but thanks for confirming
>> that these were triggered by my commit, I've seen a few false positives
>> from buildbot occasionally.
>>
>> The bad news is that it's not that easy to reproduce.  I have a i386 VM
>> running Debian here on which I've been testing the patch in question,
>> and at least for some of the tests in question, everything passes fine
>> for me here.
>> 
>> Can you give more details about the setup in which the test is being
>> run?
>
> It is a debian stable i386 install (in a VM on an x86_64 machine).
>
> Debian GNU/Linux 12 (bookworm) Linux 6.1.0-21-686-pae #1 SMP
> PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) GNU C Library (Debian
> GLIBC 2.36-9+deb12u7) stable release version 2.36. g++ 12.2.0 GNU
> objdump (GNU Binutils for Debian) 2.40

I'm still unable to reproduce this failure.

I also am using Debian i386 (12.5) image running within a VM on an
x86-64 machine.

I've done an update which pulled all the version numbers into line with
what you listed above.

>
> Build with:
> https://sourceware.org/cgit/builder/tree/builder/master.cfg#n3362
> $ ../configure --enable-targets=all --disable-sim
> $ make -j4 all-gdb all-gdbserver

I've tried building with these exact flags too, but I've also done a
build with address and undefined behaviour sanitisers in place to see if
they would throw anything up.

>
> A "native" minimal make check passes:
> https://sourceware.org/cgit/builder/tree/builder/master.cfg#n3208
> $ make check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break-
> caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp
> gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp
> gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break-
> main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp
> gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp
> gdb.base/break-probes.exp gdb.gdb/unittest.exp gdb.server/unittest.exp
> '
>
> 		=== gdb Summary ===
>
> # of expected passes		379
> # of unsupported tests		1
>
> But the same with native-gdbserver produces failures:
>
> $ make check-gdb 'TESTS= gdb.base/break-always.exp gdb.base/break-
> caller-line.exp gdb.base/break-entry.exp gdb.base/break.exp
> gdb.base/break-fun-addr.exp gdb.base/break-idempotent.exp
> gdb.base/break-include.exp gdb.base/break-inline.exp gdb.base/break-
> main-file-remove-fail.exp gdb.base/break-on-linker-gcd-function.exp
> gdb.base/breakpoint-in-ro-region.exp gdb.base/breakpoint-shadow.exp
> gdb.base/break-probes.exp gdb.gdb/unittest.exp gdb.server/unittest.exp
> ' RUNTESTFLAGS=--target_board=native-gdbserver
>
> 		=== gdb Summary ===
>
> # of expected passes		97
> # of unexpected failures	65
> # of unsupported tests		3
>
>
> This takes a long time since a lot of tests time out.
>
> The whole gdb.log is attached.
>
> I think the issue is:
>
> target remote localhost:2355
> Remote debugging using localhost:2355
> warning: Architecture rejected target-supplied description
> Remote connection closed

That's super interesting.  When buildbot originally reported this issue
the warning was not present, instead the connection was just being
dropped, like this:

  Listening on port 2346
  target remote localhost:2346
  Remote debugging using localhost:2346
  Remote connection closed
  (gdb) continue
  The program is not being run.
  (gdb) FAIL: gdb.base/break-main-file-remove-fail.exp: cmdline: runto: run to start

This is why I thought there might be a core file to be found; I figured
that gdbserver was crashing for some reason, but it appears that the
behaviour has changed.

>
> At least that warning seems new.

Agreed, that would certainly cause the problem.

>
>>   Is it possible to get access to the machine in question?  I'm
>> assuming that gdbserver is crashing and possibly dumping core, so just
>> getting the core file might help.  Or maybe it's a VM? In which case
>> maybe the image is available somewhere that I might reproduce the
>> environment locally?
>
> I don't see any core files. You can certainly get access to the VM, but
> it takes a little setup since it is located after a jump host. The
> install is a plain Debian i386 image, completely up to date.

I'm really at a loss here for how to move forward without debugging this
on the exact VM where the issue is showing up.

I'm happy to try anything you can suggest, but right now this bug seems
to only occur on the buildbot VM.

Thanks,
Andrew