Strange problem which involves libpthread, fork() and linker flags.

[Goffredo Baroncelli <kreijack at inwind dot it>]

Hi Roland,

I found a bug, which could be caused by the pthread fork function (>= 2.22) .

Let me to reassume the history; I encountered this bug because mosh stopped to work after debian updated the libc to the 2.22 [1][2]. After few tests I discovered that the problem was related to a strange combination of switch and libs (see below).

The minimal test case to reproduce the problem is the following:

$ cat boom.c
extern void dofork();

int main() {
	dofork();
}

$ cat dofork.c
#include <unistd.h>

void dofork() {
	fork();
}

$ gcc -fPIC -c dofork.c
$ gcc -shared -Wl,-z,now -o libdofork.so dofork.o
$ gcc -o boom boom.c -lpthread -L$(pwd) -ldofork
$ LD_LIBRARY_PATH=$(pwd) ./boom
Segmentation fault

Expected result: the program doesn't have to crash
Result: instead the program crashes :-)

(I know that I didn't have used any pthread function; but even using one of these the crash still happens; also even if I put -pthread and -lpthread the bug still exists)

Further information:

$ LD_LIBRARY_PATH=$(pwd) ldd ./boom linux-vdso.so.1 (0x00007ffe817dc000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f16b38ed000)
libdofork.so => /home/ghigo/mosh/libdofork.so (0x00007f16b36ec000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f16b3347000)
/lib64/ld-linux-x86-64.so.2 (0x0000562eba12a000)


The fatal combination seems to be "-lpthread", "-Wl,-z,now" a call to fork() and the glibc-2.22. The crash happens near the fork. 

The bug happened in mosh because:
- mosh is linked against libprotobuffer and libutempter
- mosh uses the "-Wl,-z,now" switch
- libprotobuffer via pkg-config suggests the -lpthread switch
- and libutempter uses the fork() function. 
Together created the condition for the bug.

Looking at the commits between the 2.21 and 2.22 regarding nptl/pt-fork.c, I found the following glibc commit:

   commit beff1d132c16aedd87a3f1bc7b572c8e69819015
   Author: Roland McGrath <roland@hack.frob.com>
   Date: Fri Feb 6 10:53:07 2015 -0800

        Clean up NPTL fork to be compat-only

Reverting it, the problem seems to disappear.

Unfortunately I was unable to fully understand this kind of code. So I have only the evidence of the fact but I don't understand the reason.

What do you think ?

BR
G.Baroncelli



[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=817929
[2] https://github.com/mobile-shell/mosh/issues/727

On 2016-03-17 16:41, Goffredo Baroncelli wrote:
> On 2016-03-17 09:39, Sergey Gvozdetskiy wrote:
>> Hello, Goffredo, and everybody!
>> this bug doesn't happen with libc-2.21 and gcc version 4.9.3 too.
> 
> Yes, I downgrade my libc to 2.21, and the crash doesn't happen.
> 
> 
> [....]
> 
>>
>> Return value after execute ./boom was 36, 40, 43, etc.
> 
> This is right, the main didn't return anything, so I think that the exit value is casual
> 
>>
>> Same strace out(with minimal differents) I had, while use gcc version 5.2.1.
>> Return value of it, was 0.
>>
>> Ubuntu 15.10, gcc's from repo.
>>
>> It for more details, may be helpful.))
>> Best regards!
> Thanks, so the problem seems to related to libc 2.22.x
> 
>>
>> On 03/17/2016 01:50 AM, Goffredo Baroncelli wrote:
>>> Hi All,
>>>
>>> I hope that this is the right place where post this kind of question. If this is not the case, sorry for the inconvenience and please give me a suggestion where I have to put this question.
>>>
>>> Investigating the reason why mosh crashed on my debian machine [*], I was able to create a test case to reproduce the crash.
>>>
>>> I have to point out that the problem which I found was not related to mosh, but (I suppose) to a strange interaction between some linker flag and the using of the pthread library.
>>>
>>>
>>> $ cat boom.c
>>> extern void dofork();
>>>
>>> int main() {
>>>       dofork();
>>> }
>>>
>>> $ cat dofork.c
>>> #include <unistd.h>
>>>
>>> void dofork() {
>>>      fork();
>>> }
>>>
>>> $ gcc -fPIC -c dofork.c
>>> $ gcc -shared -Wl,-z,now -o libdofork.so dofork.o
>>> $ gcc -o boom boom.c -lpthread -L$(pwd) -ldofork
>>> $ LD_LIBRARY_PATH=$(pwd) ./boom
>>> Segmentation fault
>>>
>>> $ LD_LIBRARY_PATH=$(pwd) ldd ./boom linux-vdso.so.1 (0x00007ffe817dc000)
>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f16b38ed000)
>>> libdofork.so => /home/ghigo/mosh/libdofork.so (0x00007f16b36ec000)
>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f16b3347000)
>>> /lib64/ld-linux-x86-64.so.2 (0x0000562eba12a000)
>>>
>>>
>>> $ dpkg --list | grep libc6
>>> ii  libc6:amd64                                   2.22-3
>>>
>>> I was able to reproduce this bug also in a Fedora F23 machine (=libc 2.22). The bug happens even with different compiler versions (gcc-4.8, gcc-6, gcc-5.3, clang-3.8). In another regular debian machine equipped with libc-2.19 the crash doesn't happen. All the machines tested are linux x86-64.
>>>
>>> In order to reproduce this problem:
>>> - use the "-Wl,-z,now" in the shared library
>>> - put fork() in the shared library
>>> - compile the main program with "-lpthread" before the "-l<shared library>" (the order matters !)
>>>
>>> So my questions are:
>>> - I am wrong to suppose that this code should not crash ?
>>> - I know that -lpthread is not needed, but is it forbid to use it even if not needed ? With -pthread only, the code works without problem (mosh too !)
>>> - Could the problem be libc/pthread related ?
>>>
>>>
>>> BR
>>> G.Baroncelli
>>>
>>> [*] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=817929
>>> mosh problem
>>
>>
> 
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5