In chasing a malloc bookkeeping corruption bug in X, I hit the following deadlock: #1 0x00a66873 in __lll_lock_wait_private () from /lib/libc.so.6 #2 0x009ef8b4 in _L_lock_9686 () from /lib/libc.so.6 #3 0x009ed914 in malloc () from /lib/libc.so.6 #4 0x009e1718 in vasprintf () from /lib/libc.so.6 #5 0x009c3eeb in asprintf () from /lib/libc.so.6 #6 0x0099dc3d in __assert_fail () from /lib/libc.so.6 #7 0x009ec47d in _int_malloc () from /lib/libc.so.6 #8 0x009ed91e in malloc () from /lib/libc.so.6 #9 0x0095ba15 in pcfReadFont () from /usr/lib/libXfont.so.1 #10 0x0095667b in ?? () from /usr/lib/libXfont.so.1 #11 0x00949d03 in ?? () from /usr/lib/libXfont.so.1 #12 0x0095a11f in BitmapOpenScalable () from /usr/lib/libXfont.so.1 [...] Not awesome. There's a number of ways around this, the most straightforward of which seems to be gcc variable-length arrays and sprintf'ing the error string into that. Attached patch does this. Other options would include walking the args to __assert_fail() directly. I don't really have an opinion, and am willing to implement whatever is preferred.
Created attachment 4150 [details] glibc-mallocless-assert.patch
It's not at all clear to me why it doesn't use __fxprintf directly, or indeed just use __dprintf.
The reason malloc is used is so that in case you see a core file but no console output you know what was going on. Using the stack memory is not sufficient in this case. The __abort_msg variable was deliberately introduced and is visible to the debugger and other tools.
Fixed upstream.