extra syscall_cancel frames
Florian Weimer
fweimer@redhat.com
Thu Mar 20 16:58:31 GMT 2025
* Mark Wielaard:
> Hi,
>
> With latest glibc on fedora rawhide (glibc-2.40.9000-37.fc43.x86_64) I
> am seeing some extra frames in the call stack that I wonder whether to
> specially handle in valgrind.
>
> Before we would report on some bad syscall argument like:
>
> ==1929378== Syscall param sendmsg(msg) points to uninitialised byte(s)
> ==1929378== at 0x4971514: sendmsg (sendmsg.c:28)
> ==1929378== by 0x40128B: main (sendmsg.c:46)
> ==1929378== Address 0x1ffefff640 is on thread 1's stack
> ==1929378== in frame #1, created by main (sendmsg.c:13)
>
> Now it looks like:
>
> ==2670784== Syscall param sendmsg(msg) points to uninitialised byte(s)
> ==2670784== at 0x48D9AE6: __internal_syscall_cancel (cancellation.c:64)
> ==2670784== by 0x48D9B03: __syscall_cancel (cancellation.c:75)
> ==2670784== by 0x49628F0: sendmsg (sendmsg.c:28)
> ==2670784== by 0x4005CB: main (sendmsg.c:46)
> ==2670784== Address 0x1ffeffff40 is on thread 1's stack
> ==2670784== in frame #3, created by main (sendmsg.c:13)
>
> Which I think is not as helpful to the user.
> So I am wondering whether those extra frames should be handled
> specially in valgrind and filtered out. But were these extra stack
> frames added explicitly? And are they easily detected (symbol name
> starting with __ and containing syscall might be a good hearistic)?
I think __internal_syscall_cancel should get inlined into
__syscall_cancel.
There is also another out-of-line system call in __syscall_cancel_arch,
which you probably don't see in your example because the process is
single-threaded.
It is necessary to concentrate all cancelable system calls in one place
for correctness reasons because we need to know if the cancelling signal
arrives within the system call or immediately after it. It's the only
way to tell whether the effect of the system call has taken place or
not. With all system calls in one place, this is a simple address
check. With the previous inlining-based approach, we would have to have
some sort of lookup table to determine whether the cancellation attempt
happened while the system call was executing or not.
This is relevant bug:
Race conditions in pthread cancellation
<https://sourceware.org/bugzilla/show_bug.cgi?id=12683>
And this commit fixed it:
commit 89b53077d2a58f00e7debdfe58afabe953dac60d
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date: Tue Jun 25 16:17:44 2024 -0300
nptl: Fix Race conditions in pthread cancellation [BZ#12683]
The current racy approach is to enable asynchronous cancellation
before making the syscall and restore the previous cancellation
type once the syscall returns, and check if cancellation has happen
during the cancellation entrypoint.
As described in BZ#12683, this approach shows 2 problems:
1. Cancellation can act after the syscall has returned from the
kernel, but before userspace saves the return value. It might
result in a resource leak if the syscall allocated a resource or a
side effect (partial read/write), and there is no way to program
handle it with cancellation handlers.
2. If a signal is handled while the thread is blocked at a cancellable
syscall, the entire signal handler runs with asynchronous
cancellation enabled. This can lead to issues if the signal
handler call functions which are async-signal-safe but not
async-cancel-safe.
For the cancellation to work correctly, there are 5 points at which the
cancellation signal could arrive:
[ ... )[ ... )[ syscall ]( ...
1 2 3 4 5
1. Before initial testcancel, e.g. [*... testcancel)
2. Between testcancel and syscall start, e.g. [testcancel...syscall start)
3. While syscall is blocked and no side effects have yet taken
place, e.g. [ syscall ]
4. Same as 3 but with side-effects having occurred (e.g. a partial
read or write).
5. After syscall end e.g. (syscall end...*]
And libc wants to act on cancellation in cases 1, 2, and 3 but not
in cases 4 or 5. For the 4 and 5 cases, the cancellation will eventually
happen in the next cancellable entrypoint without any further external
event.
The proposed solution for each case is:
1. Do a conditional branch based on whether the thread has received
a cancellation request;
2. It can be caught by the signal handler determining that the saved
program counter (from the ucontext_t) is in some address range
beginning just before the "testcancel" and ending with the
syscall instruction.
3. SIGCANCEL can be caught by the signal handler and determine that
the saved program counter (from the ucontext_t) is in the address
range beginning just before "testcancel" and ending with the first
uninterruptable (via a signal) syscall instruction that enters the
kernel.
4. In this case, except for certain syscalls that ALWAYS fail with
EINTR even for non-interrupting signals, the kernel will reset
the program counter to point at the syscall instruction during
signal handling, so that the syscall is restarted when the signal
handler returns. So, from the signal handler's standpoint, this
looks the same as case 2, and thus it's taken care of.
5. For syscalls with side-effects, the kernel cannot restart the
syscall; when it's interrupted by a signal, the kernel must cause
the syscall to return with whatever partial result is obtained
(e.g. partial read or write).
6. The saved program counter points just after the syscall
instruction, so the signal handler won't act on cancellation.
This is similar to 4. since the program counter is past the syscall
instruction.
So The proposed fixes are:
1. Remove the enable_asynccancel/disable_asynccancel function usage in
cancellable syscall definition and instead make them call a common
symbol that will check if cancellation is enabled (__syscall_cancel
at nptl/cancellation.c), call the arch-specific cancellable
entry-point (__syscall_cancel_arch), and cancel the thread when
required.
2. Provide an arch-specific generic system call wrapper function
that contains global markers. These markers will be used in
SIGCANCEL signal handler to check if the interruption has been
called in a valid syscall and if the syscalls has side-effects.
A reference implementation sysdeps/unix/sysv/linux/syscall_cancel.c
is provided. However, the markers may not be set on correct
expected places depending on how INTERNAL_SYSCALL_NCS is
implemented by the architecture. It is expected that all
architectures add an arch-specific implementation.
3. Rewrite SIGCANCEL asynchronous handler to check for both canceling
type and if current IP from signal handler falls between the global
markers and act accordingly.
4. Adjust libc code to replace LIBC_CANCEL_ASYNC/LIBC_CANCEL_RESET to
use the appropriate cancelable syscalls.
5. Adjust 'lowlevellock-futex.h' arch-specific implementations to
provide cancelable futex calls.
Some architectures require specific support on syscall handling:
* On i386 the syscall cancel bridge needs to use the old int80
instruction because the optimized vDSO symbol the resulting PC value
for an interrupted syscall points to an address outside the expected
markers in __syscall_cancel_arch. It has been discussed in LKML [1]
on how kernel could help userland to accomplish it, but afaik
discussion has stalled.
Also, sysenter should not be used directly by libc since its calling
convention is set by the kernel depending of the underlying x86 chip
(check kernel commit 30bfa7b3488bfb1bb75c9f50a5fcac1832970c60).
* mips o32 is the only kABI that requires 7 argument syscall, and to
avoid add a requirement on all architectures to support it, mips
support is added with extra internal defines.
Checked on aarch64-linux-gnu, arm-linux-gnueabihf, powerpc-linux-gnu,
powerpc64-linux-gnu, powerpc64le-linux-gnu, i686-linux-gnu, and
x86_64-linux-gnu.
[1] https://lkml.org/lkml/2016/3/8/1105
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Thanks,
Florian
More information about the Libc-alpha
mailing list