Bug 27653 - i18n causes deadlocks with malloc interposers due to missing reentrancy
Summary: i18n causes deadlocks with malloc interposers due to missing reentrancy
Status: REOPENED
Alias: None
Product: glibc
Classification: Unclassified
Component: libc (show other bugs)
Version: 2.30
: P2 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-26 06:45 UTC by Haishan
Modified: 2022-07-18 09:50 UTC (History)
5 users (show)

See Also:
Host: arm64 Kunpeng920
Target:
Build: Gcc-9.2 Glibc-2.30
Last reconfirmed: 2021-08-12 00:00:00


Attachments
Interceptor library working-around glibc's intl issues for malloc interceptors (733 bytes, text/x-csrc)
2021-08-13 14:40 UTC, Mathieu Desnoyers
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Haishan 2021-03-26 06:45:55 UTC
Hello,
centos seems to be incompatible with libasan(8.0.1).
Run a linux command(like ps,free) with LD_PRELOAD envrionable variable on centos aarch64 kunpeng920 (gcc-9.2/glibc-2.30), 
It will locked.

Run my scripts:
export ASAN_OPTIONS=detect_leaks=0:halt_on_error=0
export LD_LIBRARY_PATH=/usr/lib64/clang/8.0.1/lib/
export LD_PRELOAD=/lib64/clang/8.0.1/lib/libclang_rt.asan-aarch64.so
ps

------------------------------------------------------------------------
(gdb) bt
#0  0x0000ffffb244ecd8 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1  0x0000ffffb279501c in textdomain () from /lib64/libc.so.6
#2  0x0000ffffb29f72a0 in __interceptor_textdomain (domainname=0x40cab8 "procps-ng") at ../lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:413
#3  0x0000000000402b10 in main ()
(gdb) info threads
  Id   Target Id                              Frame
* 1    Thread 0xffffb2126480 (LWP 25205) "ps" 0x0000ffffb279501c in textdomain () from /lib64/libc.so.6
There is only one thread. how could it generate deadlock?

But unset LD_PRELOAD, ps runs ok.
Try update compiler-rt to 10.0.0, this deadlock still exists.

Thank you for helping.
Comment 1 Andreas Schwab 2021-04-06 14:37:13 UTC
Please report that to the llvm project.  The ASAN runtime calls dlerror (which uses gettext) during init, which causes a deadlock.

#0  __dcigettext (domainname=0xfffff73fb1e8 <_libc_intl_domainname> "libc", 
    msgid1=0xfffff3b01ea0 "undefined symbol: swift_demangle", 
    msgid2=msgid2@entry=0x0, plural=plural@entry=0, n=n@entry=0, 
    category=category@entry=5) at dcigettext.c:485
#1  0x0000fffff72f6068 in __GI___dcgettext (domainname=<optimized out>, 
    msgid=<optimized out>, category=category@entry=5) at dcgettext.c:47
#2  0x0000fffff7450b48 in __dlerror () at dlerror.c:102
#3  0x0000fffff7588ccc in AsanInitInternal ()
    at ../projects/compiler-rt/lib/asan/asan_rtl.cpp:521
#4  0x0000fffff7580698 in __interceptor_malloc ()
    at ../projects/compiler-rt/lib/asan/asan_malloc_linux.cpp:144
#5  0x0000fffff72f5d58 in set_binding_values (
    domainname=0xfffff6c21290 "libgpg-error", dirnamep=0xffffffffefb8, 
    codesetp=0x0) at bindtextdom.c:202
#6  0x0000fffff72f5fcc in set_binding_values (codesetp=0x0, 
    dirnamep=0xffffffffefb8, domainname=<optimized out>) at bindtextdom.c:82
#7  __bindtextdomain (domainname=<optimized out>, dirname=<optimized out>)
    at bindtextdom.c:320
#8  0x0000fffff6c0c860 in gpg_err_init () from /usr/lib64/libgpg-error.so.0
#9  0x0000fffff7fcc630 in call_init (l=<optimized out>, argc=argc@entry=1, 
    argv=argv@entry=0xfffffffff068, env=env@entry=0xfffffffff078)
    at dl-init.c:74
#10 0x0000fffff7fcc72c in call_init (env=0xfffffffff078, argv=0xfffffffff068, 
    argc=1, l=<optimized out>) at dl-init.c:37
#11 _dl_init (main_map=0xfffff7fff280, argc=1, argv=0xfffffffff068, 
    env=0xfffffffff078) at dl-init.c:121
#12 0x0000fffff7fbee08 in _dl_start_user () from /lib/ld-linux-aarch64.so.1
Comment 2 Mathieu Desnoyers 2021-08-12 20:17:15 UTC
This bug was incorrectly triaged. The glibc i18n code uses non-recursive locks, and causes issues when malloc interposers perform a i18n translation lookup. This happens when dlsym() fails.
Comment 3 Mathieu Desnoyers 2021-08-12 20:30:27 UTC
I have provided a RFC patch against glibc which solves this issue: https://sourceware.org/pipermail/libc-alpha/2021-August/130129.html
Comment 4 Mathieu Desnoyers 2021-08-13 14:40:19 UTC
Created attachment 13617 [details]
Interceptor library working-around glibc's intl issues for malloc interceptors
Comment 5 Haishan 2021-08-30 03:29:40 UTC
   thank you. this patch works for me. so this patch will upstream to
   glibc github?
Comment 6 Mathieu Desnoyers 2021-08-30 14:09:51 UTC
(In reply to Haishan from comment #5)
>    thank you. this patch works for me. so this patch will upstream to
>    glibc github?

Just to confirm, did you test the interceptor library work-around or the glibc fix I submitted here: https://sourceware.org/pipermail/libc-alpha/2021-August/130129.html ?

About the glibc fix, the mailing list discussion is ongoing with the glibc maintainers.
Comment 7 Vincent Lefèvre 2022-03-04 13:26:25 UTC
Under Debian/unstable (Debian package libc6 2.33-7), when running some executables with LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.6, I can also see this issue. The gdb backtrace shows __futex_abstimed_wait_common64 / __GI___futex_abstimed_wait64 / __pthread_rwlock_wrlock_full64 / __GI___pthread_rwlock_wrlock / set_binding_values in each case.
Comment 8 Jean-Michaël Celerier 2022-07-18 09:32:54 UTC
Same issue here, glibc 2.35-6 and clang 14. I tried the interposer library fix but couldn't make it work - for some reason whichever libasan.so I try to preload I get issues:


    LD_PRELOAD=/usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/libasan.so:/home/jcelerier/libasan-glibc-fix.so ./ossia-score
    ==19609==Your application is linked against incompatible ASan runtimes.

     LD_PRELOAD=/usr/lib/clang/14.0.6/lib/linux/libclang_rt.asan-x86_64.so:/home/jcelerier/libasan-glibc-fix.so ./ossia-score 
    ==19481==Your application is linked against incompatible ASan runtimes.


If I don't add the asan runtime I still get the hang..
Comment 9 Jean-Michaël Celerier 2022-07-18 09:50:33 UTC
The extremely aggressive option of forcibly interposing a

    char* dlerror(void) { return ""; }

does work, though