Bug 18368

You're using nsenter to enter the PID namespace but you're not entering the corresponding mount namespace; GDB is looking in the wrong /proc.  Try adding "-m" to your nsenter options.

Also, I can't imagine how using the docker daemon as a reference can work.  As I understand it the docker daemon does not switch namespaces, so "nsenter -t $DOCKERPID" should just stay in whatever namespaces you were already in.

If you're building from source, try this branch:

  git clone -b namespaces https://github.com/gbenson/binutils-gdb.git

With that GDB you should be able to attach with "gdb -p $HOSTPID" (where $HOSTPID is 29120 in your example).

Let me know if/how you get this working please.

>Also, I can't imagine how using the docker daemon as a reference can work.  As >I understand it the docker daemon does not switch namespaces, so "nsenter -t 
>$DOCKERPID" should just stay in whatever namespaces you were already in

You are right. The namespace remains same for daemon as is the docker client. 


>You're using nsenter to enter the PID namespace but you're not entering the >corresponding mount namespace; GDB is looking in the wrong /proc.  Try adding >"-m" to your nsenter options.

I did try - nsenter-t 25057 -p -m  gdb -p 1 - but this fails since gdb is not present there.
(where 25057 is the pid of process inside container - in this case sleep 100000)


>If you're building from source, try this branch:

>  git clone -b namespaces https://github.com/gbenson/binutils-gdb.git

I will try this and get back.

You do need to have GDB or gdbserver inside the container if you're going to use nsenter.  (If just gdbserver, start GDB with no options and do "target remote | nsenter ... gdbserver - --attach 1")

So, I did try with namespaces branch and it works great.


sudo gdb -p 31802
GNU gdb (GDB) 7.9.50.20150429-cvs
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 31802
Reading symbols from target:/usr/bin/sleep...Reading symbols from target:/usr/bin/sleep...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Reading symbols from target:/lib64/libc.so.6...(no debugging symbols found)...done.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable
0x00007fdaa0a74470 in __nanosleep_nocancel () from target:/lib64/libc.so.6
(gdb) bt
#0  0x00007fdaa0a74470 in __nanosleep_nocancel () from target:/lib64/libc.so.6
#1  0x0000000000403e5f in rpl_nanosleep ()
#2  0x0000000000403cc0 in xnanosleep ()
#3  0x00000000004016cd in main ()
(gdb) quit
A debugging session is active.

        Inferior 1 [process 31802] will be detached.

Quit anyway? (y or n) y
Detaching from program: target:/usr/bin/sleep, process 31802
   
==================

a) I also verified this from inside container by installing gdb there. (You need a privileged container for that which is strange).

b) It still shows  the warning - "warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable", is it still required?


c) Also, -Werror build fails for that branch: https://github.com/gbenson/binutils-gdb/issues/1


Thanks.

(In reply to Raghavendra from comment #4)
> So, I did try with namespaces branch and it works great.
[snip]

Great!

> a) I also verified this from inside container by installing gdb there.
> (You need a privileged container for that which is strange).

What error do you get if the container isn't privileged?

> b) It still shows  the warning - "warning: Target and debugger are in
> different PID namespaces; thread lists and other data are likely
> unreliable", is it still required?

I'm not sure.  The way libthread_db works I think everything should be
consistent, but I'm still experimenting.

> c) Also, -Werror build fails for that branch:
> https://github.com/gbenson/binutils-gdb/issues/1

I'll look into it.

I'm closing this bug now (the original issue isn't a GDB bug) but we can carry on discussing things here.

For unprivileged container (ie. without --privileged or --cap-add sys_ptrace):

I see

gdb -p 1
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 1
ptrace: Operation not permitted.
(gdb) bt
No stack.
(gdb)


=========



>I'm not sure.  The way libthread_db works I think everything should be
>consistent, but I'm still experimenting.

Does this require just the gdb to be installed or any other binutil components updated as well? 


Regarding

>You do need to have GDB or gdbserver inside the container if you're going to >use nsenter.  (If just gdbserver, start GDB with no options and do "target >remote | nsenter ... gdbserver - --attach 1")

I haven't been able to get this working since it required gdbserver inside container.  Can you show how this is done?

You just need to install the gdbserver command (/usr/bin/gdbserver usually).
For Fedora and RHEL this is "yum install gdb-gdbserver", for Debian etc I think it's "apt-get install gdbserver".