Bug 23175

Summary: leftover breakpoints lead to target process crashes after detaching when multiple inferiors having breakpoints are running
Product: gdb Reporter: agentzh <agentzh>
Component: breakpointsAssignee: Not yet assigned to anyone <unassigned>
Status: UNCONFIRMED ---    
Severity: normal    
Priority: P2    
Version: HEAD   
Target Milestone: ---   
Host: Target:
Build: Last reconfirmed:
Attachments: Self-contained bash script for reproducing the bug (nginx needed though)

Description agentzh 2018-05-12 21:57:01 UTC
Created attachment 11014 [details]
Self-contained bash script for reproducing the bug (nginx needed though)

Hi there,

I've just noted a bug in the latest git master branch of GDB (commit 861d86514a) when multiple inferiors are having (the same set of) breakpoints. Basically when gdb attaches to and creates breakpoints in both of the 2 processes at the same time, one of the 2 processes's inferior would be left in a bad state when the gdb is interrupted by SIGINT (or Ctrl-C on the terminal) and detached from both of these processes automatically.

The bad state manifests itself by crashing the process when the process not stopped by the SIGINT signal via gdb continues running through the former break point code locations after gdb detaches from the process.

It seems that for some reason, the gdb fails to remove the breakpoints from the inferior that is still running (not stopped by Ctrl-C).

Below is a minimal and self-contained bash script that can reproduce and demonstrate the bug (assuming an nginx executable is visible through the `PATH` environment).

https://gist.github.com/agentzh/2baa7619f49699f144ca02488dd62850

(This test.sh script is also attached to this ticket.)

Create a temporary directory, say, `~/work/`, and then chdir into that directory, and then run the `test.sh` script given above:

```
bash ./test.sh
```

One typical output on my side is as follows:

https://gist.github.com/agentzh/9644a5e564fc8c72f0fcd09fe19c3f30

Please note the following line in the sample output above:

```
curl: (52) Empty reply from server
```

This error was due to the crash of the nginx worker process of the pid 75699, which was confirmed by the subsequent ps output:

```
agentzh   75697  0.0  0.0  35252  2948 ?        S    14:24   0:00 nginx: worker process
agentzh   75715  0.0  0.0  35252  2716 ?        S    14:24   0:00 nginx: worker process
```

Please note that the original worker process 75699 is now replaced by a brand new process of the pid 75715 launched automatically by the master process after the former worker was gone.

We know the original worker process 75699 in the earlier outputs like below:

```
agentzh   75697  0.0  0.0  35252  2716 ?        S    14:24   0:00 nginx: worker process
agentzh   75699  0.0  0.0  35252  2716 ?        S    14:24   0:00 nginx: worker process
```

and also

```
  Num  Description       Executable
  1    process 75697     /usr/local/openresty/nginx/sbin/nginx
* 2    process 75699     /usr/local/openresty/nginx/sbin/nginx
```

The last one is the output of the `info inferiors` gdb command. One interesting fact here is that the other inferior's process (here is 75697) would not enter bad state after gdb was interrupted and detached. The following gdb output confirms that it is the first inferior receiving the SIGINT signal:

```
Thread 1.1 "nginx" received signal SIGINT, Interrupt.
```

It seems like whoever process stopped by the SIGINT would be okay while the other processes would still be running and their breakpoints would never have a chance to get properly cleaned up, which is also confirmed by the following gdb output line:

```
warning: Error removing breakpoint 1
```

More details about my system:

```
$ uname -a
Linux work-fed 4.15.13-300.fc27.x86_64 #1 SMP Mon Mar 26 19:06:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)

$ /opt/gdb-git/bin/gdb --version
GNU gdb (GDB) 8.1.50.20180512-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
```

Please let me know if you need any further information. Also, I'm wondering if there is any work around for something like a `stop inferior N` command that can properly stop the still-running inferiors so that gdb can have a chance to properly clean up the pending breakpoints in them.

Many thanks!
Comment 1 agentzh 2018-05-12 22:10:33 UTC
Sorry, I forgot to add that this issue also appears in formal gdb releases like 8.1 and 8.0.1.

I built the gdb HEAD and 8.1 release from source like this:

```
./configure --with-python=/opt/py3k/bin/python3 --prefix=/opt/gdb-git
make -j9
sudo make install-gdb -j9
```

Where the `/opt/py3k/bin/python3` is built from the official python release tarball like this:

```
wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz
tar -xvf Python-3.6.5.tar.xz
cd Python-3.6.5/
./configure --prefix=/opt/py3k
make -j9
sudo make install
```