pthread forced stack unwind issue on C++ pthread cancellation

[Rajesh Ghanekar <rajeshsg at gmail dot com>]

Hi,
   I have C++ pthread program in which I do rpc calls (clnt_call) and
another thread
does cancellation of the thread doing RPC call. If I am in RPC call
(doing read or poll),
and thread cancellation happens, thread cleanup functions doesn't get called.
This happens only with clnt_call, any other syscall/library functions
does proper
cleanup.

Here is the stack frames for RPC call, unwind only happens till last read:

(gdb) bt
#0  0xf7d525c1 in __libc_siglongjmp (env=0xf7d25444, val=1) at longjmp.c:29
#1  0xf7fb0988 in unwind_stop (version=1, actions=<value optimized
out>, exc_class=4157756484, exc_obj=0xf7d25dc0, context=0xf7d24c94,
stop_parameter=0xf7d25444)
    at unwind.c:96
#2  0xf7e84d8a in _Unwind_ForcedUnwind_Phase2 (exc=0xf7d25dc0,
context=0xf7d24c94) at ../../../../src/libgcc/../gcc/unwind.inc:163
#3  0xf7e84e8a in _Unwind_ForcedUnwind (exc=0xf7d25dc0,
stop=0xf7fb08f0 <unwind_stop>, stop_argument=0xf7d25444) at
../../../../src/libgcc/../gcc/unwind.inc:207
#4  0xf7fb2dc2 in _Unwind_ForcedUnwind (exc=0xf7d25dc0,
stop=0xf7fb08f0 <unwind_stop>, stop_argument=0xf7d25444) at
../nptl/sysdeps/pthread/unwind-forcedunwind.c:132
#5  0xf7fb0898 in *__GI___pthread_unwind (buf=<value optimized out>)
at unwind.c:130
#6  0xf7fa83bb in __do_cancel (sig=32, si=0xf7d24dfc, ctx=0xf7d24e7c)
at ../nptl/pthreadP.h:265
#7  sigcancel_handler (sig=32, si=0xf7d24dfc, ctx=0xf7d24e7c) at nptl-init.c:202
#8  <signal handler called>
#9  0xf7fdf430 in __kernel_vsyscall ()
#10 0xf7de70fb in read () from /lib32/libc.so.6
#11 0xf7e1a9fe in readtcp (ctptr=0x804b960 "\005", buf=0x804c9e0
"\200", len=4000) at clnt_tcp.c:494
#12 0xf7e20b1a in fill_input_buf (rstrm=0x804b9f8) at xdr_rec.c:569
#13 0xf7e20cf0 in get_input_bytes (rstrm=0x804b9f8, addr=<value
optimized out>, len=4) at xdr_rec.c:588
#14 0xf7e20d56 in set_input_fragment (rstrm=0x1c) at xdr_rec.c:607
#15 0xf7e20eba in xdrrec_getbytes (xdrs=0x804b9ac, addr=0xf7d2526c
"@\272\004\b\354R\322\367\254\271\004\b\220R\322\367w\375\341\367\254\271\004\b\210R\322\367,",
len=4)
    at xdr_rec.c:264
#16 0xf7e20fd7 in xdrrec_getlong (xdrs=0x804c9e0, lp=0xf7d25288) at
xdr_rec.c:220
#17 0xf7e1fd77 in xdr_u_long (xdrs=0x1c, ulp=0xf7d252ec) at xdr.c:194
#18 0xf7e1cf1c in xdr_replymsg (xdrs=0x804b9ac, rmsg=0xf7d252ec) at
rpc_prot.c:137
#19 0xf7e1a788 in clnttcp_call (h=0x804b1f0, proc=0,
xdr_args=0x80486d8 <xdr_void@plt>, args_ptr=0x0, xdr_results=0x80486d8
<xdr_void@plt>, results_ptr=0x0, timeout=...)
    at clnt_tcp.c:291
#20 0x080488f1 in thread_start_function (arg=0x0) at prg.c:62
#21 0xf7fa980e in start_thread (arg=0xf7d25b70) at pthread_create.c:300
#22 0xf7df6a1e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Stack frames for working cancellation on printfs (not doing RPC call) placed is:

(gdb) bt
#0  0xf7d525c1 in __libc_siglongjmp (env=0xf7d25444, val=1) at longjmp.c:29
#1  0xf7fb0988 in unwind_stop (version=1, actions=<value optimized
out>, exc_class=4157756484, exc_obj=0xf7d25dc0, context=0xf7d25220,
stop_parameter=0xf7d25444)
    at unwind.c:96
#2  0xf7e84d8a in _Unwind_ForcedUnwind_Phase2 (exc=0xf7d25dc0,
context=0xf7d25220) at ../../../../src/libgcc/../gcc/unwind.inc:163
#3  0xf7e85056 in _Unwind_Resume (exc=0xf7d25dc0) at
../../../../src/libgcc/../gcc/unwind.inc:232
#4  0x08048802 in thread_start_function (arg=0x0) at prg.c:70
#5  0xf7fa980e in start_thread (arg=0xf7d25b70) at pthread_create.c:300
#6  0xf7df6a1e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

When I compile with -fno-exceptions, cleanup happens fine even with RPC calls.
I am testing on x86_64 machine, but code compiled with -m32. If I compile
the code with default (-m64) option, it runs fine.

Is there any issue with C++ thread cancellation on 64-bit machines
when running 32-bit
programs? I am using glibc 2.10.1-0ubuntu16 and g++ is 4:4.4.1-1ubuntu2.

Any help appreciated.

Thanks,
Rajesh