Bug 18368

Summary: gdb with nsenter attaches to wrong process
Product: gdb Reporter: Raghavendra <raghu.prabhu13>
Component: gdbAssignee: Not yet assigned to anyone <unassigned>
Status: RESOLVED INVALID    
Severity: normal CC: gbenson, pedro
Priority: P2    
Version: 7.9   
Target Milestone: ---   
Host: Target:
Build: Last reconfirmed:

Description Raghavendra 2015-05-02 02:55:43 UTC
When trying to attach to a process in a different PID namespace, gdb correctly identifies it and warns as "warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable"

however, when used with nsenter (-p for pid namespace), it attaches to the wrong process.


nsenter  -t 600  -p gdb -p 1      (where 600 is the PID of docker daemon, and 1 is the PID of process inside it, whose host pid is 29120).

GNU gdb (GDB) 7.9
.......
......
Attaching to process 1
Reading symbols from /usr/lib/systemd/systemd...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libcap.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpam.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libkmod.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libmount.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libseccomp.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/librt.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Reading symbols from /usr/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libattr.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/liblzma.so.5...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libblkid.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libuuid.so.1...(no debugging symbols found)...done.
0x00007f1439c951d3 in __epoll_wait_nocancel () from /usr/lib/libc.so.6
(gdb) quit
....

As you can see it attaches to systemd of the host than the process inside the container.

nsenter  -t 600  -p gdb -p 29120 also gives the PID namespace warning and doesn't work.
Comment 1 Gary Benson 2015-05-05 09:18:12 UTC
You're using nsenter to enter the PID namespace but you're not entering the corresponding mount namespace; GDB is looking in the wrong /proc.  Try adding "-m" to your nsenter options.

Also, I can't imagine how using the docker daemon as a reference can work.  As I understand it the docker daemon does not switch namespaces, so "nsenter -t $DOCKERPID" should just stay in whatever namespaces you were already in.

If you're building from source, try this branch:

  git clone -b namespaces https://github.com/gbenson/binutils-gdb.git

With that GDB you should be able to attach with "gdb -p $HOSTPID" (where $HOSTPID is 29120 in your example).

Let me know if/how you get this working please.
Comment 2 Raghavendra 2015-05-05 14:50:21 UTC
>Also, I can't imagine how using the docker daemon as a reference can work.  As >I understand it the docker daemon does not switch namespaces, so "nsenter -t 
>$DOCKERPID" should just stay in whatever namespaces you were already in

You are right. The namespace remains same for daemon as is the docker client. 


>You're using nsenter to enter the PID namespace but you're not entering the >corresponding mount namespace; GDB is looking in the wrong /proc.  Try adding >"-m" to your nsenter options.

I did try - nsenter-t 25057 -p -m  gdb -p 1 - but this fails since gdb is not present there.
(where 25057 is the pid of process inside container - in this case sleep 100000)


>If you're building from source, try this branch:

>  git clone -b namespaces https://github.com/gbenson/binutils-gdb.git

I will try this and get back.
Comment 3 Gary Benson 2015-05-05 15:06:37 UTC
You do need to have GDB or gdbserver inside the container if you're going to use nsenter.  (If just gdbserver, start GDB with no options and do "target remote | nsenter ... gdbserver - --attach 1")
Comment 4 Raghavendra 2015-05-05 15:26:02 UTC
So, I did try with namespaces branch and it works great.


sudo gdb -p 31802
GNU gdb (GDB) 7.9.50.20150429-cvs
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 31802
Reading symbols from target:/usr/bin/sleep...Reading symbols from target:/usr/bin/sleep...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Reading symbols from target:/lib64/libc.so.6...(no debugging symbols found)...done.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable
0x00007fdaa0a74470 in __nanosleep_nocancel () from target:/lib64/libc.so.6
(gdb) bt
#0  0x00007fdaa0a74470 in __nanosleep_nocancel () from target:/lib64/libc.so.6
#1  0x0000000000403e5f in rpl_nanosleep ()
#2  0x0000000000403cc0 in xnanosleep ()
#3  0x00000000004016cd in main ()
(gdb) quit
A debugging session is active.

        Inferior 1 [process 31802] will be detached.

Quit anyway? (y or n) y
Detaching from program: target:/usr/bin/sleep, process 31802
   
==================

a) I also verified this from inside container by installing gdb there. (You need a privileged container for that which is strange).

b) It still shows  the warning - "warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable", is it still required?


c) Also, -Werror build fails for that branch: https://github.com/gbenson/binutils-gdb/issues/1


Thanks.
Comment 5 Gary Benson 2015-05-07 09:23:06 UTC
(In reply to Raghavendra from comment #4)
> So, I did try with namespaces branch and it works great.
[snip]

Great!

> a) I also verified this from inside container by installing gdb there.
> (You need a privileged container for that which is strange).

What error do you get if the container isn't privileged?

> b) It still shows  the warning - "warning: Target and debugger are in
> different PID namespaces; thread lists and other data are likely
> unreliable", is it still required?

I'm not sure.  The way libthread_db works I think everything should be
consistent, but I'm still experimenting.

> c) Also, -Werror build fails for that branch:
> https://github.com/gbenson/binutils-gdb/issues/1

I'll look into it.

I'm closing this bug now (the original issue isn't a GDB bug) but we can carry on discussing things here.
Comment 6 Raghavendra 2015-05-08 09:49:54 UTC
For unprivileged container (ie. without --privileged or --cap-add sys_ptrace):

I see

gdb -p 1
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 1
ptrace: Operation not permitted.
(gdb) bt
No stack.
(gdb)


=========



>I'm not sure.  The way libthread_db works I think everything should be
>consistent, but I'm still experimenting.

Does this require just the gdb to be installed or any other binutil components updated as well? 


Regarding

>You do need to have GDB or gdbserver inside the container if you're going to >use nsenter.  (If just gdbserver, start GDB with no options and do "target >remote | nsenter ... gdbserver - --attach 1")

I haven't been able to get this working since it required gdbserver inside container.  Can you show how this is done?
Comment 7 Gary Benson 2015-05-13 10:16:49 UTC
You just need to install the gdbserver command (/usr/bin/gdbserver usually).
For Fedora and RHEL this is "yum install gdb-gdbserver", for Debian etc I think it's "apt-get install gdbserver".