SIGBUS and hang in dl* functions after truncate
Justin Fries
justinf@us.ibm.com
Mon Nov 25 21:41:11 GMT 2024
Hello --
A user reported to me a hang at exit() which was due to a script of theirs truncating a shared library while in use.
My process dlopen()s a library supplied by the user, calls dlsym() to resolve and run a function in it, and later uses dlsym() to resolve another symbol. Between the two dlsym() calls their script renamed and truncated the library to zero bytes, causing a SIGBUS in the second call.
In a single-threaded version of the process I can handle the SIGBUS and siglongjmp() back, issuing an error message about the library and even going on to call dlopen/dlsym/dlclose with other libraries. When the process is multi-threaded, the locks serializing the dl* functions are not released and all subsequent dl* functions hang including the implicit call to dlclose() by exit(). That’s entirely reasonable.
The user’s script, which was trying to update the library with a new version, has since been corrected, and I have made a change to ensure the process can terminate successfully after this error. Normally I would consider that an end, but I noticed that the SIGBUS thrown by glibc is not seen on some other systems (e.g. macOS and AIX).
I thought I would inquire to see whether glibc could avoid the SIGBUS as on those other systems, or failing that whether I can handle the SIGBUS in a way that lets glibc release its locks, currently elided by my use of siglongjmp() as shown in the attached test case. Thanks in advance for any advice you can share.
Justin Fries
IBM MQ Development: Distributed Service
IBM Automation
Here is the Makefile, source, and results from the test case, which is gratuitously threaded to ensure glibc serializes dl* calls:
Makefile:
<code>
.PHONY: all clean
all: libtest1.so libtest2.so runtests
runtests: runtests.c
gcc -g -Wall -Werror -lpthread -ldl -o runtests runtests.c
test.o: test.c
gcc -g -Wall -Werror -c -fpic -o test.o test.c
libtest1.so: test.o
gcc -Wall -Werror -shared -o libtest1.so test.o
libtest2.so: test.o
gcc -Wall -Werror -shared -o libtest2.so test.o
clean:
rm test.o libtest1.so libtest2.so runtests
</code>
test.c:
<code>
#include <stdio.h>
void testfunc_a(char *program, int tid, char *message)
{
printf("%s [tid-%d]: >> testfunc_a says \"%s\"\n", program, tid, message);
return;
}
void testfunc_b(char *program, int tid, char *message)
{
printf("%s [tid-%d]: >> testfunc_b says \"%s\"\n", program, tid, message);
return;
}
</code>
runtests.c:
<code>
// runtests 1.1
// 22 November 2024
//
// This program attempts to simulate a failure reported by a customer
// who had inadvertently truncated a loaded library to zero bytes while
// it was being used, causing a SIGBUS in dlsym when resolving another
// symbol inside the library and a subsequent hang in exit().
//
// To build and test:
// make clean; make all; ./runtests
#include <pthread.h>
#include <errno.h>
#include <stdio.h>
#include <dlfcn.h>
#include <stdlib.h>
#include <setjmp.h>
#include <signal.h>
#include <unistd.h>
#include <sys/types.h>
#if !defined(thread_local)
#include <threads.h>
#endif
// Array of library tests to be done on individual threads
struct { int tid; int status; pthread_t threadref; const char *library; void *handle; int failp; } test[] = {
{ .tid = 1, /* The main thread, which doesn't test any libraries */ },
{ .tid = 2, .library = "./libtest1.so", .handle = NULL, .failp = 1, },
{ .tid = 3, .library = "./libtest2.so", .handle = NULL, .failp = 0, },
{ .library = NULL, },
};
#define PROGRAM_NAME "runtests"
// Print a message prefaced by the program name and current tid
#define tprintf(format, ...) \
printf(PROGRAM_NAME " [tid-%d]: " format "\n", test[idx].tid, ##__VA_ARGS__);
thread_local sigjmp_buf env;
thread_local int idx;
void *testlibrary(void*);
void signalhandler(int);
int main(int argc, char **argv)
{
int status = EXIT_SUCCESS;
struct sigaction handler;
int cur;
int rc;
idx = 0;
// Turn off stdout buffering and set a recovery environment
(void)setvbuf(stdout, NULL, _IONBF, 0);
tprintf("Version 1.1 built " __DATE__);
if (sigsetjmp(env, 1))
{
status = EXIT_FAILURE;
goto out;
}
// Handle any synchronous signals
sigemptyset(&handler.sa_mask);
handler.sa_handler = signalhandler;
sigaction(SIGBUS, &handler, NULL);
sigaction(SIGFPE, &handler, NULL);
sigaction(SIGILL, &handler, NULL);
sigaction(SIGSEGV, &handler, NULL);
// Test each library in its own thread so that glibc has to serialise things
for (cur = 1; test[cur].library; cur++)
{
tprintf("Starting thread %d to test library %s", test[cur].tid, test[cur].library);
rc = pthread_create(&test[cur].threadref, NULL, testlibrary, (void *)&test[cur].tid);
if (rc != 0)
{
status = EXIT_FAILURE;
tprintf("Failed to create thread %d: errno=%d", test[cur].tid, errno);
test[cur].tid = 0;
continue;
}
tprintf("Created thread %d to test library %s", test[cur].tid, test[cur].library);
// Join each thread before creating the next
tprintf("Waiting for thread %d to exit", test[cur].tid);
rc = pthread_join(test[cur].threadref, NULL);
if (rc != 0)
{
status = EXIT_FAILURE;
tprintf("Failed to join thread %d: errno=%d", test[cur].tid, errno);
continue;
}
tprintf("Joined thread %d which tested library %s with status %d",
test[cur].tid, test[cur].library, test[cur].status);
}
// Unload each library
for (cur = 1; test[cur].library; cur++)
{
if (!test[cur].handle)
continue;
rc = sigsetjmp(env, 1);
if (rc == 0)
{
tprintf("Unloading library %s with handle %p", test[cur].library, test[cur].handle);
rc = dlclose(test[cur].handle);
if (rc != 0)
{
status = EXIT_FAILURE;
tprintf("Failed to dlclose library %s: errno=%d", test[cur].library, errno);
continue;
}
tprintf("Unloaded library %s", test[cur].library);
test[cur].handle = NULL;
}
else
{
status = EXIT_FAILURE;
tprintf("Failed to unload library %s due to signal %d", test[cur].library, rc);
}
}
out:
// Exit nicely if possible, or _exit
rc = sigsetjmp(env, 1);
if (rc == 0)
{
tprintf("Exiting with status %d", status);
exit(status);
}
else
{
status = EXIT_FAILURE;
tprintf("Exiting with status %d", status);
_exit(status);
}
}
void *testlibrary(void *arg)
{
void (*faddr_a)(char *, int, char *);
void (*faddr_b)(char *, int, char *);
char *fname_a = "testfunc_a";
char *fname_b = "testfunc_b";
int rc;
idx = *(int *)arg - 1;
test[idx].status = 0;
// Set a thread recovery environment
rc = sigsetjmp(env, 1);
if (rc != 0)
{
tprintf("Exiting test due to signal %d", rc);
test[idx].status = 128 + rc;
goto out;
}
// Load the library and resolve and call the first function
tprintf("Loading library %s", test[idx].library);
test[idx].handle = dlopen(test[idx].library, RTLD_NOW);
if (test[idx].handle == NULL)
{
tprintf("Failed to dlopen %s: errno=%d", test[idx].library, errno);
test[idx].status = 1;
goto out;
}
tprintf("Loaded library %s with handle %p", test[idx].library, test[idx].handle);
tprintf("Resolving symbol %s with handle %p", fname_a, test[idx].handle);
faddr_a = dlsym(test[idx].handle, fname_a);
if (faddr_a == NULL)
{
tprintf("Failed to dlsym %s with handle %p: errno=%d", fname_a, test[idx].handle, errno);
test[idx].status = 2;
goto out;
}
tprintf("Resolved symbol %s with handle %p to address %p", fname_a, test[idx].handle, faddr_a);
faddr_a(PROGRAM_NAME, test[idx].tid, "Hello.");
// Optionally simulate the customer failure by truncating the in-use library
if (test[idx].failp)
{
tprintf("Truncating library %s to force failure", test[idx].library);
rc = truncate(test[idx].library, 0);
if (rc != 0)
{
tprintf("Failed to truncate %s: errno=%d", test[idx].library, errno);
test[idx].status = 3;
goto out;
}
tprintf("Truncated library %s while it is loaded", test[idx].library);
}
// Try to resolve and call another function in the library
tprintf("Resolving symbol %s with handle %p", fname_b, test[idx].handle);
faddr_b = dlsym(test[idx].handle, fname_b);
if (faddr_b == NULL)
{
tprintf("Failed to dlsym %s with handle %p: errno=%d", fname_b, test[idx].handle, errno);
test[idx].status = 4;
goto out;
}
tprintf("Resolved symbol %s with handle %p to address %p", fname_b, test[idx].handle, faddr_b);
faddr_b(PROGRAM_NAME, test[idx].tid, "Good-bye.");
out:
tprintf("Returning from thread function");
return NULL;
}
void signalhandler(int signal)
{
printf(PROGRAM_NAME " [tid-%d]: !! Signal %d received\n", test[idx].tid, signal);
// Go back to the current jump buffer
siglongjmp(env, signal);
}
</code>
Here are the results from my test run showing the largely expected hang on subsequent dl* calls, run on Red Hat Enterprise Linux 9.4 on x86_64 using glibc version 2.34 release 100.el9_4.3:
<code>
sh> make clean; make all; ./runtests &
rm test.o libtest1.so<https://libtest1.so/> libtest2.so<https://libtest2.so/> runtests
gcc -g -Wall -Werror -c -fpic -o test.o test.c
gcc -Wall -Werror -shared -o libtest1.so<https://libtest1.so/> test.o
gcc -Wall -Werror -shared -o libtest2.so<https://libtest2.so/> test.o
gcc -g -Wall -Werror -lpthread -ldl -o runtests runtests.c
[2] 145685
sh> runtests [tid-1]: Version 1.1 built Nov 22 2024
runtests [tid-1]: Starting thread 2 to test library ./libtest1.so
runtests [tid-1]: Created thread 2 to test library ./libtest1.so
runtests [tid-1]: Waiting for thread 2 to exit
runtests [tid-2]: Loading library ./libtest1.so
runtests [tid-2]: Loaded library ./libtest1.so with handle 0x7f6d00000b80
runtests [tid-2]: Resolving symbol testfunc_a with handle 0x7f6d00000b80
runtests [tid-2]: Resolved symbol testfunc_a with handle 0x7f6d00000b80 to address 0x7f6d05ffa109
runtests [tid-2]: >> testfunc_a says "Hello."
runtests [tid-2]: Truncating library ./libtest1.so to force failure
runtests [tid-2]: Truncated library ./libtest1.so while it is loaded
runtests [tid-2]: Resolving symbol testfunc_b with handle 0x7f6d00000b80
runtests [tid-2]: !! Signal 7 received
runtests [tid-2]: Exiting test due to signal 7
runtests [tid-2]: Returning from thread function
runtests [tid-1]: Joined thread 2 which tested library ./libtest1.so with status 135
runtests [tid-1]: Starting thread 3 to test library ./libtest2.so
runtests [tid-1]: Created thread 3 to test library ./libtest2.so
runtests [tid-1]: Waiting for thread 3 to exit
runtests [tid-3]: Loading library ./libtest2.so
sh> ps
PID TTY TIME CMD
144730 pts/1 00:00:00 bash
145685 pts/1 00:00:00 runtests
145688 pts/1 00:00:00 ps
sh> gdb -p 145685
GNU gdb (GDB) Red Hat Enterprise Linux 10.2-13.el9
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 145685
[New LWP 145687]
Error while mapping shared library sections:
`./libtest1.so': not in executable format: file format not recognized
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
__futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=145687, futex_word=0x7f6d05bff910) at futex-internal.c:57
57 return INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected,
(gdb) thread apply all where
Thread 2 (Thread 0x7f6d05bff640 (LWP 145687) "runtests"):
#0 futex_wait (private=0, expected=2, futex_word=0x7f6d06034988 <_rtld_local+2440>) at ../sysdeps/nptl/futex-internal.h:146
#1 __GI___lll_lock_wait (futex=futex@entry=0x7f6d06034988 <_rtld_local+2440>, private=0) at lowlevellock.c:50
#2 0x00007f6d05c8d04d in lll_mutex_lock_optimized (mutex=0x7f6d06034988 <_rtld_local+2440>) at pthread_mutex_lock.c:49
#3 ___pthread_mutex_lock (mutex=0x7f6d06034988 <_rtld_local+2440>) at pthread_mutex_lock.c:129
#4 0x00007f6d06009ef3 in _dl_open (file=0x40301e "./libtest2.so", mode=-2147483646, caller_dlopen=0x401ae4 <testlibrary+359>, nsid=-2, argc=1, argv=0x7fff84652e38, env=0x7fff84652e48) at dl-open.c:844
#5 0x00007f6d05c85cbc in dlopen_doit (a=a@entry=0x7f6d05bfed00) at dlopen.c:56
#6 0x00007f6d05d56148 in __GI__dl_catch_exception (exception=exception@entry=0x7f6d05bfec60, operate=<optimized out>, args=<optimized out>) at /usr/src/debug/glibc-2.34-100.el9_4.3.x86_64/elf/dl-error-skeleton.c:208
#7 0x00007f6d05d56213 in __GI__dl_catch_error (objname=0x7f6d05bfecb8, errstring=0x7f6d05bfecc0, mallocedp=0x7f6d05bfecb7, operate=<optimized out>, args=<optimized out>) at /usr/src/debug/glibc-2.34-100.el9_4.3.x86_64/elf/dl-error-skeleton.c:227
#8 0x00007f6d05c8578e in _dlerror_run (operate=operate@entry=0x7f6d05c85c60 <dlopen_doit>, args=args@entry=0x7f6d05bfed00) at dlerror.c:138
#9 0x00007f6d05c85d71 in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>) at dlopen.c:71
#10 ___dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:81
#11 0x0000000000401ae4 in testlibrary (arg=0x405110 <test+80>) at runtests.c:173
#12 0x00007f6d05c89c02 in start_thread (arg=<optimized out>) at pthread_create.c:443
#13 0x00007f6d05d0ec40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 1 (Thread 0x7f6d05ff1800 (LWP 145685) "runtests"):
#0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=145687, futex_word=0x7f6d05bff910) at futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f6d05bff910, expected=145687, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at futex-internal.c:87
#2 0x00007f6d05c867ff in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6d05bff910, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at futex-internal.c:139
#3 0x00007f6d05c8b6d3 in __pthread_clockjoin_ex (threadid=140106224629312, thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:102
#4 0x0000000000401565 in main (argc=1, argv=0x7fff84652e38) at runtests.c:94
(gdb)
</code>
More information about the Libc-help
mailing list