Bug 12683

Summary: Race conditions in pthread cancellation
Product: glibc Reporter: Rich Felker <bugdal>
Component: nptlAssignee: Not yet assigned to anyone <unassigned>
Status: NEW ---    
Severity: critical CC: adhemerval.zanella, carlos, dan, drepper.fsp, fweimer, mail, siddhesh, sstewartgallus00, zatrazz
Priority: P2 Flags: fweimer: security-
Version: unspecified   
Target Milestone: ---   
Host: Target:
Build: Last reconfirmed:
Bug Depends on:    
Bug Blocks: 14147    
Attachments: Demonstration of file descriptor leak due to problem 1
Demonstration of problem 2

Description Rich Felker 2011-04-18 22:28:16 UTC
Created attachment 5676 [details]
Demonstration of file descriptor leak due to problem 1

The current approach to implementing pthread cancellation points is to enable asynchronous cancellation prior to making the syscall, and restore the previous cancellation type once the syscall returns. I've asked around and heard conflicting answers as to whether this violates the requirements in POSIX (I believe it does), but either way, from a quality of implementation standpoint this approach is very undesirable due to at least 2 problems, the latter of which is very serious:

1. Cancellation can act after the syscall has returned from kernelspace, but before userspace saves the return value. This results in a resource leak if the syscall allocated a resource, and there is no way to patch over it with cancellation handlers. Even if the syscall did not allocate a resource, it may have had an effect (like consuming data from a socket/pipe/terminal buffer) which the application will never see.

2. If a signal is handled while the thread is blocked at a cancellable syscall, the entire signal handler runs with asynchronous cancellation enabled. This could be extremely dangerous, since the signal handler may call functions which are async-signal-safe but not async-cancel-safe. Even worse, the signal handler may call functions which are not even async-signal-safe (like stdio) if it knows the interrupted code could only be using async-signal-safe functions, and having a thread asynchronously terminated while modifying such functions' internal data structures could lead to serious program malfunction.

I am attaching simple programs which demonstrate both issues.

The solution to problem 2 is making the thread's current execution context (e.g. stack pointer) at syscall time part of the cancellability state, so that cancellation requests received while the cancellation point is interrupted by a signal handler can identify that the thread is not presently in the cancellable context.

The solution to problem 1 is making successful return from kernelspace and exiting the cancellable state an atomic operation. While at first this seems impossible without kernel support, I have a working implementation in musl (http://www.etalabs.net/musl) which solves both problems.
Comment 1 Rich Felker 2011-04-18 22:34:44 UTC
Created attachment 5677 [details]
Demonstration of problem 2

This program should hang, or possibly print x=0 if scheduling is really wacky. If it exits printing a nonzero value of the volatile variable x, this means the signal handler wrongly executed under asynchronous cancellation.
Comment 2 Rich Felker 2011-09-21 18:30:01 UTC
It's been 5 months since I filed this bug and there's been no response. I believe this issue it important enough to at least deserve a response. From my perspective, it makes NPTL's pthread_cancel essentially unusable. I've even included a proposed solution (albeit not a patch). Getting a confirmation that you acknowledge the issue exists and are open to a solution would open the door for somebody to start the work to integrate the solution with glibc/NPTL and eventually get it fixed.
Comment 3 Rich Felker 2012-04-29 02:55:59 UTC
Ping. Now that there's some will to revisit bugs that have been long-ignored, is anyone willing to look into and confirm the problem I've reported here? I believe problem 2 is extremely serious and could lead to timing-based attacks that corrupt memory and result in deadlocks or worse. Problem 1 is also serious for long-lived processes with high reliability requirements that use thread cancellation, as rare but undetectable resource leaks are nearly inevitable and will accumulate over time.
Comment 4 Rich Felker 2012-09-22 23:13:30 UTC
I just added a detailed analysis of this bug on my blog at http://ewontfix.com/2/
Comment 5 Carlos O'Donell 2013-08-16 15:32:31 UTC
There is interest in fixing this issue. It's just that it's complicated :-)
Comment 6 Carlos O'Donell 2013-08-16 15:34:25 UTC
Let us see if I can't get resources to fix this for 2.19. We've seen some tst-cancel17 problems specifically around this issue where cancellation is delivered between syscall return and error storage and that causes problems.
Comment 7 Rich Felker 2013-08-16 16:22:05 UTC
Glad to hear that. Have you taken a look at musl's cancellation implementation? The same mechanism could be used in glibc, or I think it could be modified somewhat to use DWARF2 CFI instead of the asm labels. The basic approach is that the cancellation signal handler examines the saved program counter register and determines whether it's in the critical range starting just before the pre-syscall check of the cancellation flag and the syscall instruction (based on asm labels for these two endpoints). The kernel then handles the atomicity of side effects for us: if the signal interrupts the syscall, the kernel must either complete what it's doing and return (positioning the program counter just past the address range that would allow cancellation to be acted upon), or reset the program counter to just before the syscall instruction and setup the register contents for restarting after the signal handler (in which case cancellation can be acted upon).
Comment 8 Carlos O'Donell 2013-08-16 16:59:54 UTC
(In reply to Rich Felker from comment #7)
> Glad to hear that. Have you taken a look at musl's cancellation
> implementation? The same mechanism could be used in glibc, or I think it
> could be modified somewhat to use DWARF2 CFI instead of the asm labels. The
> basic approach is that the cancellation signal handler examines the saved
> program counter register and determines whether it's in the critical range
> starting just before the pre-syscall check of the cancellation flag and the
> syscall instruction (based on asm labels for these two endpoints). The
> kernel then handles the atomicity of side effects for us: if the signal
> interrupts the syscall, the kernel must either complete what it's doing and
> return (positioning the program counter just past the address range that
> would allow cancellation to be acted upon), or reset the program counter to
> just before the syscall instruction and setup the register contents for
> restarting after the signal handler (in which case cancellation can be acted
> upon).

I have not looked at musl's cancellation implementation.

I assume you are parsing the rt_sigframe set down by the Linux kernel and extracting information from that?
Comment 9 Rich Felker 2013-08-16 17:14:52 UTC
> I have not looked at musl's cancellation implementation.
> 
> I assume you are parsing the rt_sigframe set down by the Linux kernel and
> extracting information from that?

We are using the ucontext_t received via the third argument to the
SA_SIGINFO type signal handler.
Comment 10 Carlos O'Donell 2013-08-16 18:09:46 UTC
(In reply to Rich Felker from comment #9)
> > I have not looked at musl's cancellation implementation.
> > 
> > I assume you are parsing the rt_sigframe set down by the Linux kernel and
> > extracting information from that?
> 
> We are using the ucontext_t received via the third argument to the
> SA_SIGINFO type signal handler.

Same thing. Thanks.
Comment 11 Carlos O'Donell 2014-01-10 21:31:46 UTC
Alex Oliva and I talked about this particular issue today.

We believe that an entirely userspace solution is possible without assistance from the kernel, but it requires signal wrappers.

Signal wrappers are code that execute before and after a signal handler and does things like save and restore errno (the one use we have for them currently). The signal wrappers would assist in handling deferred cancellation.

The proposed solution would look like this:

* Stop enabling/disabling asynchronous cancellation around syscalls.

* When a blocking library function who is also a cancellation point is entered a word in the thread's TCB (call it IN_SYSCALL) is set to the value of the stack pointer (we assume no further stack adjustments are made before the function exits). The value of IN_SYSCALL is cleared just before the function returns. Deferred cancellation is still checked before and after the syscall.

* Add a signal wrapper to all signals that checks to see if IN_SYSCALL == SP stored in the ucontext_t and if it does it immediately cancels the thread. The check is done upon entry and exit of the wrapper to reduce cancellation latency. Just before unwinding the IN_SYCALL value is cleared.

* When a thread starts we install a SIGCANCEL (SIGRTMIN) handler like we did before, but this handler checks to see if the thread's IN_SYSCALL matches the SP stored in ucontext_t, indicating that cancellation was requested while executing in the cancellation region of a blocking syscall (and no other signal handler executing). In that case the signal handler cancels the thread immediately. If IN_SYSCALL != SP then another signal handler is running and we defer the cancellation to the signal wrapper or syscall wrapper. The SIGCANCEL handler operates as it previously did when asynchronous cancellation was enabled.

Resolved use cases:

- Cancellation delivered between first instruction of function and IN_SYSCALL set: Syscall wrapper code will check for cancellation and act upon it.

- Cancellation delivered between IN_SYSCALL set and syscall: The SIGCANCEL handler will immediately cancel the thread.

- Cancellation delivered between syscall and clearing IN_SYSCALL: The SIGCANCEL handler will immediately cancel the thread.

- Cancellation delivered between clearing of IN_SYSCALL and function return: The next cancellation point will act upon the cancellation (still meets POSIX requirement given escape clause of "The thread is suspended at a cancellation point and the event for which it is waiting occurs").

- Cancellation delivered and thread stopped at syscall is executing multiple nested signal handlers and the first signal handler has not checked IN_SYSCALL yet: Only the first signal delivered will have IN_SYSCALL == SP be true. The SIGCANCEL handler will do nothing. The first signal handler's wrapper will detect the cancellation is active and act upon it as it exits (only after all the other signal handlers have completed).

- Cancellation delivered and thread stopped at syscall is executing multiple nested signal handlers and the first signal handler is exiting and has already checked IN_SYSCALL: The syscall will be interrupted and return. The syscall wrapper will act upon the cancellation request. The goal here is to have the signal handlers finish executing without interruption.

Unresolved use cases:

- Related to bug 14147 -- Cancellation delivered while thread is blocked on an async-safe function (in fact it's only executing async-safe functions during the time a signal can be delivered for this to be valid) and executing a signal handler that longjmp's out of the function. In this case IN_SYSCALL is still set to SP and not cleared. If by luck SP ends up the same, and another thread delivers a cancellation request the SIGCANCEL handler will immediately cancel the thread even though it was not in a cancellation region.

- What if you are executing fork and someone tries to cancel you?

A potential resolution to the first unresolved use case is to use a cleanup handler to reset IN_SYSCALL since such a handler is run when longjmp unwinds the frames. However we then need to consider cancellation during the execution of the cleanup.

I haven't fully thought through what to do with the forking a multithreaded program case, but we should try to see if we can make it work.

Note: Setting IN_SYSCALL must be atomic.
Comment 12 Rich Felker 2014-01-10 22:37:44 UTC
Your proposed solution is a lot more complex and invasive than mine; it's actually almost equivalent to the first-generation solution I used in musl for the problem, which turned out to be a bad idea, and thus got scrapped.

Most importantly, aside from being complex and ugly, it does not actually solve the worst problem, because this case is wrong:

"Cancellation delivered between syscall and clearing IN_SYSCALL: The SIGCANCEL handler will immediately cancel the thread."

In this case, unless the syscall failed with EINTR, you must not act on the cancellation request. Doing so is non-conforming to the requirement that the side effects upon cancellation match the side effects on EINTR (which is just a fancy way of saying, approximately, that cancellation can only take place if the syscall has already done its job, e.g. closing a fd, transferring some bytes, etc.).

In addition, I suspect your solution has further flaws like what happens when you longjmp out of a signal handler that interrupted an AS-safe syscall which is a cancellation point. These issues can be solved with more complexity (extra work in longjmp), but the solution I've proposed is much simpler and has no corner cases that are difficult to handle.
Comment 13 Carlos O'Donell 2014-01-12 18:31:10 UTC
(In reply to Rich Felker from comment #12)
> Your proposed solution is a lot more complex and invasive than mine; it's
> actually almost equivalent to the first-generation solution I used in musl
> for the problem, which turned out to be a bad idea, and thus got scrapped.

Experience is knowing what not to do :-)

> Most importantly, aside from being complex and ugly, it does not actually
> solve the worst problem, because this case is wrong:

I like it when we can talk concretely about use cases.

> "Cancellation delivered between syscall and clearing IN_SYSCALL: The
> SIGCANCEL handler will immediately cancel the thread."
> 
> In this case, unless the syscall failed with EINTR, you must not act on the
> cancellation request. Doing so is non-conforming to the requirement that the
> side effects upon cancellation match the side effects on EINTR (which is
> just a fancy way of saying, approximately, that cancellation can only take
> place if the syscall has already done its job, e.g. closing a fd,
> transferring some bytes, etc.).

I was not aware of this requirement. Is this written in POSIX or did this come about from discussion with the Austin group around the problems with close() being cancelled? Can you provide a reference to this?

> In addition, I suspect your solution has further flaws like what happens
> when you longjmp out of a signal handler that interrupted an AS-safe syscall
> which is a cancellation point. These issues can be solved with more
> complexity (extra work in longjmp), but the solution I've proposed is much
> simpler and has no corner cases that are difficult to handle.

Could you propose your design as a glibc wiki page so that we can look at it and critique it? I'd be happy to adopt your solution, but I Want to review it and put it through the same kind of use-cases as we discussed here.
Comment 14 Rich Felker 2014-01-12 23:54:53 UTC
From XSH 2.9.5 Thread Cancellation:

"The side-effects of acting upon a cancellation request while suspended during a call of a function are the same as the side-effects that may be seen in a single-threaded program when a call to a function is interrupted by a signal and the given function returns [EINTR]. Any such side-effects occur before any cancellation cleanup handlers are called."

This is the important paragraph. By requiring that the side-effects on cancellation match the side effects on EINTR, the standard requires that cancellation cannot be acted upon if other irreversible side effects have already taken place. For example, if a file descriptor has been closed, data transferred, etc. then cancellation can't happen. The following paragraph explains further:

"Whenever a thread has cancelability enabled and a cancellation request has been made with that thread as the target, and the thread then calls any function that is a cancellation point (such as pthread_testcancel() or read()), the cancellation request shall be acted upon before the function returns."

This is simple. If cancellation is already pending when a cancellation point is called, it must be acted upon. The next part is less clear:

"If a thread has cancelability enabled and a cancellation request is made with the thread as a target while the thread is suspended at a cancellation point, the thread shall be awakened and the cancellation request shall be acted upon. It is unspecified whether the cancellation request is acted upon or whether the cancellation request remains pending and the thread resumes normal execution if:

* The thread is suspended at a cancellation point and the event for which it is waiting occurs

* A specified timeout expired

before the cancellation request is acted upon."

This is covering the case of blocking syscalls. If a cancellation request arrives during a blocking syscall, it's normally acted upon, but there's one race condition being described: it's possible that the "event being waited for" arrives just before the cancellation request arrives, but before the target thread unblocks. In this case, it's implementation-defined whether cancellation is acted upon (in which case, by the first paragraph, the event remains pending) or the event is acted upon (in which case the cancellation request remains pending). The reason for there being two bullet points above is that some blocking syscalls wait for either an event or a timeout (think of sem_timedwait or recv with a timeout set by setsockopt), and in that case, the timeout can also be 'consumed' and cause the cancellation to remain pending.

Anyway, this race condition is the whole matter at hand here. The two possibilities allowed (again, due to the limitations imposed by the first paragrah) are acting on the event while leaving cancellation pending, or acting on cancellation while leaving the event pending. But glibc also has a race window where it can act on both the event, producing side effects (because the kernel already has), and act on cancellation. This makes it non-conforming and makes it impossible to use cancellation safely.
Comment 15 Carlos O'Donell 2014-01-13 01:52:33 UTC
(In reply to Rich Felker from comment #14)

Thanks for the recap.

> Anyway, this race condition is the whole matter at hand here. The two
> possibilities allowed (again, due to the limitations imposed by the first
> paragrah) are acting on the event while leaving cancellation pending, or
> acting on cancellation while leaving the event pending. But glibc also has a
> race window where it can act on both the event, producing side effects
> (because the kernel already has), and act on cancellation. This makes it
> non-conforming and makes it impossible to use cancellation safely.

So does this imply that the cancellation *must* happen at some point after errno is known? Thus if a cancellation arrives and we're already in the syscall there is nothing to do but let the syscall return and let the syscall wrapper handle the cancellation. That seems reasonable to me.
Comment 16 Rich Felker 2014-01-13 04:37:30 UTC
There are several points at which the cancellation signal could arrive:

1. Before the final "testcancel" before the syscall is made.
2. Between the "testcancel" and the syscall.
3. While the syscall is blocked and no side effects have yet taken place.
4. While the syscall is blocked but with some side effects already having taken place (e.g. a partial read or write).
5. After the syscall has returned.

You want to act on cancellation in cases 1-3 but not in case 4 or 5. Handling case 1 is of course trivial, since you're about to do a conditional branch based on whether the thread has received a cancellation request; nothing needs to be done in the signal handler (but it also wouldn't hurt to handle it from the signal handler). Case 2 can be caught by the signal handler determining that the saved program counter (from the ucontext_t) is in some address range beginning just before the "testcancel" and ending with the syscall instruction.

The rest of the cases are the "tricky" part but it turns out they too are easy:

Case 3: In this case, except for certain syscalls that ALWAYS fail with EINTR even for non-interrupting signals, the kernel will reset the program counter to point at the syscall instruction during signal handling, so that the syscall is restarted when the signal handler returns. So, from the signal handler's standpoint, this looks the same as case 2, and thus it's taken care of.

Case 4: In this case, the kernel cannot restart the syscall; when it's interrupted by a signal, the kernel must cause the syscall to return with whatever partial result it obtained (e.g. partial read or write). In this case, the saved program counter points just after the syscall instruction, so the signal handler won't act on cancellation.

Case 5: OK, I lied. This one is trivial too since the program counter is past the syscall instruction already.

What about syscalls that fail with EINTR even when the signal handler is non-interrupting? In this case, the syscall wrapper code can just check the cancellation flag when the errno result is EINTR, and act on cancellation if it's set. Note that an exception needs to be made for close(), where EINTR should be treated as EINPROGRESS and thus not permit cancellation to take place.

BTW, I should justify why the signal handler should be non-interrupting (SA_RESTART): if it weren't, you would risk causing spurious EINTR in programs not written to handle it, e.g. if the user incorrectly send signal 32/33 to the process or if pthread_cancel were called while cancellation is disabled in the target thread. The kernel folks have spent a great deal of effort getting rid of spurious EINTRs (which cause all sorts of ugly bugs) and it would be a shame to reintroduce them. Also it doesn't buy you anything moving the cancellation action to the EINTR check after the syscall returns; the same check in the signal handler that handles case 2 above also handles the case of restartable syscalls correctly, for free.
Comment 17 Carlos O'Donell 2014-01-14 14:51:43 UTC
(In reply to Rich Felker from comment #16)
> There are several points at which the cancellation signal could arrive:
> 
> 1. Before the final "testcancel" before the syscall is made.
> 2. Between the "testcancel" and the syscall.
> 3. While the syscall is blocked and no side effects have yet taken place.
> 4. While the syscall is blocked but with some side effects already having
> taken place (e.g. a partial read or write).
> 5. After the syscall has returned.
> 
> You want to act on cancellation in cases 1-3 but not in case 4 or 5.
> Handling case 1 is of course trivial, since you're about to do a conditional
> branch based on whether the thread has received a cancellation request;
> nothing needs to be done in the signal handler (but it also wouldn't hurt to
> handle it from the signal handler). Case 2 can be caught by the signal
> handler determining that the saved program counter (from the ucontext_t) is
> in some address range beginning just before the "testcancel" and ending with
> the syscall instruction.
> 
> The rest of the cases are the "tricky" part but it turns out they too are
> easy:
> 
> Case 3: In this case, except for certain syscalls that ALWAYS fail with
> EINTR even for non-interrupting signals, the kernel will reset the program
> counter to point at the syscall instruction during signal handling, so that
> the syscall is restarted when the signal handler returns. So, from the
> signal handler's standpoint, this looks the same as case 2, and thus it's
> taken care of.
> 
> Case 4: In this case, the kernel cannot restart the syscall; when it's
> interrupted by a signal, the kernel must cause the syscall to return with
> whatever partial result it obtained (e.g. partial read or write). In this
> case, the saved program counter points just after the syscall instruction,
> so the signal handler won't act on cancellation.
> 
> Case 5: OK, I lied. This one is trivial too since the program counter is
> past the syscall instruction already.

Excellent. I like your idea then. It seems like a list of PC's using either markers or dwarf2 is the way to go here.

> What about syscalls that fail with EINTR even when the signal handler is
> non-interrupting? In this case, the syscall wrapper code can just check the
> cancellation flag when the errno result is EINTR, and act on cancellation if
> it's set. Note that an exception needs to be made for close(), where EINTR
> should be treated as EINPROGRESS and thus not permit cancellation to take
> place.

We'll need a big disclaimer about close and a detailed comment. I know some of the details there, specifically that although EINTR has been returned the close will complete.

> BTW, I should justify why the signal handler should be non-interrupting
> (SA_RESTART): if it weren't, you would risk causing spurious EINTR in
> programs not written to handle it, e.g. if the user incorrectly send signal
> 32/33 to the process or if pthread_cancel were called while cancellation is
> disabled in the target thread. The kernel folks have spent a great deal of
> effort getting rid of spurious EINTRs (which cause all sorts of ugly bugs)
> and it would be a shame to reintroduce them. Also it doesn't buy you
> anything moving the cancellation action to the EINTR check after the syscall
> returns; the same check in the signal handler that handles case 2 above also
> handles the case of restartable syscalls correctly, for free.

That makes sense.
Comment 18 Jackie Rosen 2014-02-16 19:42:14 UTC Comment hidden (spam)
Comment 19 Steven Stewart-Gallus 2014-07-19 18:43:59 UTC
I am confused but if the proposed fix for this bug is implemented than
that means that my bug at
https://sourceware.org/bugzilla/show_bug.cgi?id=17168 where I can't
cancel FUTEX_WAITs would be automatically fixed right? I would have to
do no extra effort to let my blocking system call be cancellable? So
this proposed fix would have a side benefit of giving me
cancellability for free? Or would this be a bug or at least a breaking
change?
Comment 20 Rich Felker 2014-07-19 18:54:33 UTC
Steven, I don't think this bug is related to your issue. If bug 9712
(of which your 17168 seems to be a duplicate) is resolved by adding
futex and glibc makes futex a cancellation point, THEN the resolution
of this bug (12683) would make it safe to use cancellation with the
futex function in a way that's race-free.

I think this is a strong argument for resolving 9712 by adding futex:
unless it's part of libc, there's no safe way to make it cancellable,
because wrapping syscall() with async cancellation will introduce an
application-level bug comparable to this bug.
Comment 21 Steven Stewart-Gallus 2014-07-20 18:15:16 UTC
Okay Rich Felker, I was confused because the implementation in Musl
seems to look it'd would make syscall users that block cancellation
points. So, if GLibc does something similar to your solution they
would have to explicitly block cancels in the syscall function to
preserve compatibility (and in the future GLibc might possibly
consider moving over to making syscall automatically cancellable for
blocking system calls but that'd be a separate issue)?
Comment 22 Rich Felker 2014-07-20 18:41:08 UTC
Steven, I'm not sure I understand what you're saying. This issue report is not about changing which syscalls/functions are cancellable. For the standard functions that is specified by POSIX, and for extensions, the natural choices were already made and changing them would be problematic to their users. The topic at hand is just fixing the mechanism by which cancellation is performed so that there are not race conditions.

If your question is about the syscall() function that applications can use to make syscalls directly, there is no open issue for making it cancellable, and as above, changing this would be problematic. One could envision a request for a separate version of the syscall() function which is cancellable, but as far as I know nobody has requested this and I think it's a bad idea to be adding features that encourage applications to make syscalls directly (since this is usually non-portable between archs due to subtle differences in the calling conventions and other issues like whether the libc-level structs match the syscall-level ones for a given arch).
Comment 23 Adhemerval Zanella Netto 2014-08-28 14:03:20 UTC
I am currently working on a fix based on musl implementation, which from comments #16 and #17 seems a good approach.  My initial idea is to use PC markers instead of DWARF2, since it see it as more clean approach. However, this is require a lot of cleanup.

I plan push implementations for powerpc64 and x86_64 and ask for arch maintainers for more arch specific work. I also plan to write a wiki page describing the work done and summarizing the discussion on this bug report.
Comment 24 Carlos O'Donell 2014-08-28 15:02:33 UTC
(In reply to Adhemerval Zanella Netto from comment #23)
> I am currently working on a fix based on musl implementation, which from
> comments #16 and #17 seems a good approach.  My initial idea is to use PC
> markers instead of DWARF2, since it see it as more clean approach. However,
> this is require a lot of cleanup.
> 
> I plan push implementations for powerpc64 and x86_64 and ask for arch
> maintainers for more arch specific work. I also plan to write a wiki page
> describing the work done and summarizing the discussion on this bug report.

The biggest problem with DWARF2 is the parser, and making it accessible from the signal handler. I strongly suggest using PC, and a list of exception regions generated from markers in the assembly (similar to kernel exception regions).
Comment 25 Dan Searle 2015-01-15 13:20:17 UTC
I think we have stubmled upon this bug, or something related to it. Can someone please confirm I'm on the right track here?

We have a multithreaded server application which calls recv() and poll() from async cancellable threads, each thread handles a single connection with a master thread accpeting new connections and adding them to a job queue.

More and more often now we are seeing the server lock up and on inspection two or more threads seem deadlocked in some race condition inside libc recv() and or poll().

One example here shows two back traces from gdb from the two threads that seemed deadlocked chewing 100% CPU:

Thread 1 bt:
#0  __pthread_disable_asynccancel () at ../nptl/sysdeps/unix/sysv/linux/x86_64/cancellation.S:98
#1  0x00007f895ba987fd in __libc_recv (fd=0, fd@entry=33, buf=buf@entry=0x7cada02b, n=n@entry=1024, flags=1537837035,
    flags@entry=16384) at ../sysdeps/unix/sysv/linux/x86_64/recv.c:35
#2  0x000000000040ec54 in recv (__flags=16384, __n=1024, __buf=0x7cada02b, __fd=33)
    at /usr/include/x86_64-linux-gnu/bits/socket2.h:44
[snip]

Thread 2 bt:
#0  0x00007f895ba987eb in __libc_recv (fd=fd@entry=31, buf=buf@entry=0x7ca5e02b, n=n@entry=1024, flags=-1, flags@entry=16384)
    at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33
#1  0x000000000040ec54 in recv (__flags=16384, __n=1024, __buf=0x7ca5e02b, __fd=31)
    at /usr/include/x86_64-linux-gnu/bits/socket2.h:44
[snip]

There can be more than two threads involved, but I'm unsure if it can happen with just one thread locked up, but it's always inside recv() or poll() and sometimes in __pthread_disable_asynccancel() within either of those.

Could I work around this problem by changing the threads to syncronmous cancellable or try to work around the need to cancel the treads at all?
Comment 26 Rich Felker 2015-01-15 13:31:26 UTC
I don't think this is the bug you're seeing. If it were, use of async cancellation would only make it worse. But the symptoms you'd see from this bug would be things like side effects of a function having happened despite it getting cancelled.

If you're seeing 100% cpu load from threads in recv, the most likely explanation is that the socket you're reading from is in EOF status (remote sending end closed), so that recv immediately returns zero. Repeatedly attempting to read in this situation would be an application bug, not anything related to glibc.
Comment 27 Dan Searle 2015-01-15 14:01:41 UTC
(In reply to Rich Felker from comment #26)
> I don't think this is the bug you're seeing. If it were, use of async
> cancellation would only make it worse. But the symptoms you'd see from this
> bug would be things like side effects of a function having happened despite
> it getting cancelled.
> 
> If you're seeing 100% cpu load from threads in recv, the most likely
> explanation is that the socket you're reading from is in EOF status (remote
> sending end closed), so that recv immediately returns zero. Repeatedly
> attempting to read in this situation would be an application bug, not
> anything related to glibc.

Thanks Rich, your suggestion made me think to look through the code paths again and you are quite right, there was an infinite loop in there, not obvious but I found it.

In light of the current problems with cancellable threads and syscalls, I'm going to disable cancelation during the main job execution (where all the recv() can poll()) call are, just in case this bug is causing problems I'm unaware of.

Many thanks, you saved me a lot of hair pulling :)
Comment 28 Adhemerval Zanella 2017-01-12 17:16:24 UTC
Back to the issue reported on the bug report, I am still working on a fix for this issue and I am currently rebasing my changes against master and checking if there is any missing issue that I need to take care of.

I have decided to extend this fix in basically two subtasks that complement
each other:

  1. Consolidate all the cancellation syscall implementations on C code to
     remove the requirement of syscall-cancel.h header.  This is orthogonal
     to cancellation fix and aims to make the architecture adjustment simpler
     (less assembly-macro code that emulates functions calls).

  2. The cancellation (bz12683) fix itself which will require multiple patches
     for each architecture.

For 1. I have a set of 33 patches, which 2 I already sent for review without reply [1] [2].  The rest is similar to consolidation patches I have been working to set a C implementation plus architecture defines to set the correct syscall usage.

For 2. the patch is incomplete regarding architecture support. Based on current implementation, I have added fixes for only x86, powerpc, s390, aarch64, and arm.  I summarize the steps required for each architecture fix at [3] and I plan to review it after the patch rebase I am working it.

However for correct fix I am foreseeing some issues that I will need to take care when more architectures are fixed:

  1. i386 syscall code when using cancellation is suboptimal using this
     cancellation mechanism due the not usage of vDSO.  This has been
     discussed on LKML on how kernel could help userland to accomplish
     it, but afaik discussion was stalled.

  2. Current code assumes maximum of 6 arguments syscalls for the 
     helper bridge.  This is true for most architecture, however afaik
     MIPS has 7 argument cancellable syscall, which will require code
     adjustment.  To avoid more performance issues, ideally both the
     bridge function and related declaration should be arch-specific
     (6 arguments if not specific, otherwise 7).

I am currently tracking some issues that have arise from code rebase and I will check on the architectures I have access (x86, powerpc64le, arm, aarch64).  For 1. I also plan to have a full make/check using all the supported architectures and as soon I have all fixed I plan to update my sourceware branch [5] and re-start discussion for 2.26.

My plan is at least to have 1. on 2.26. I will update the bugzilla with this report.

[1] https://sourceware.org/ml/libc-alpha/2016-08/msg00443.html
[2] https://sourceware.org/ml/libc-alpha/2016-08/msg00442.html
[3] https://sourceware.org/glibc/wiki/PortStatus
[4] https://lkml.org/lkml/2016/3/8/1105
[5] https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/azanella/bz12683
Comment 29 cvs-commit@gcc.gnu.org 2018-02-26 21:10:11 UTC
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, azanella/bz12683 has been created
        at  d95b8c5f0fa8a8a2b99e63461df3fbe917990a5e (commit)

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=d95b8c5f0fa8a8a2b99e63461df3fbe917990a5e

commit d95b8c5f0fa8a8a2b99e63461df3fbe917990a5e
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue Jan 24 21:05:56 2017 -0200

    nptl: Remove THREAD_ATOMIC_* macros
    
    This patch removes the ununsed THREAD_ATOMIC_* macros now that nptl
    code is using C11 atomics.
    
    Checked on x86_64-linux-gnu and i686-linux-gnu.
    
    	* manual/pattern.texi (THREAD_ATOMIC_BIT_SET): Remove.
    	(THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	* nptl/pthreadP.h (THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	(THREAD_ATOMIC_BIT_SET): Likewise.
    	* sysdeps/i386/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	(THREAD_ATOMIC_AND): Likewise.
    	* sysdeps/x86_64/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	(THREAD_ATOMIC_AND): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=fdafb04cb8269446e056273bcd923a4186113790

commit fdafb04cb8269446e056273bcd923a4186113790
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue Jan 24 09:17:37 2017 -0200

    nptl: cancelhandling refactor
    
    This patch basically uses glibc C11 atomic function to access and
    modify struct pthread cancelhandling variable.  All plain access
    are handled using the atomic_load_* functions and THREAD_ATOMIC_BIT_SET
    is replaced by atomic_fetch_or_acquire.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, and powerpc64le-linux-gnu.
    
    	* nptl/descr.h (CANCELED_BITMASK): Rename to THREAD_CANCELED.
    	(EXITING_BITMASK): Rename to THREAD_EXITING.
    	(TERMINATED_BITMARK): Rename to THREAD_TERMINATED.
    	* nptl/allocatestack.c (setxid_mark_thread): Use THREAD_* instead of
    	*_BITMASK and use glibc C11 atomic function to access pthread
    	cancelhandling.
    	* nptl/libc-cancellation.c (__syscall_cancel): Likewise.
    	* nptl/nptl-init.c (sigcancel_handler): Likewise.
    	* nptl/pthread_cancel.c (pthread_cancel): Likewise.
    	* nptl/pthread_create.c (__free_tcb): Likewise.
    	(START_THREAD_DEFN): Likewise.
    	(__pthread_create_2_1): Likewise.
    	* nptl/pthread_detach.c (pthread_detach): Likewise.
    	* nptl/pthread_exit.c (__pthread_exit): Likewise.
    	* nptl/pthread_join.c (pthread_join): Likewise.
    	* nptl_db/td_thr_get_info.c (td_thr_get_info): Likewise.
    	* nptl_db/td_thr_getfpregs.c (td_thr_getfpregs): Likewise.
    	* nptl_db/td_thr_getgregs.c (td_thr_getregs): Likewise.
    	* nptl_db/td_thr_setfpregs.c (td_thr_setfpregs): Likewise.
    	* nptl_db/td_thr_setgregs.c (td_thr_setgregs): Likewise.
    	* sysdeps/unix/sysv/linux/arm/syscall_cancel.S
    	(__syscall_cancel_arch): Change CANCELED_BITMASK to
    	THREAD_CANCELED in comment.
    	* sysdeps/unix/sysv/linux/syscall_cancel.c
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/x86_64/nptl/tcb-offsets.sym (TCB_CANCELED_BITMASK):
    	Remove.
    	(TCB_EXITING_BITMASK): Likewise.
    	(TCB_TERMINATED_BITMASK): Likewise.
    	* sysdeps/unix/sysv/linux/i386/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/powerpc/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/sh/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/sparc32/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9037ec95f5d0e8ac5d846f3b2181e468a15e6819

commit 9037ec95f5d0e8ac5d846f3b2181e468a15e6819
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue Jul 18 16:33:41 2017 -0300

    nptl: Remove setxid bit from cancelhandling
    
    This patch removes the setxid signaling out of cancelhandling flag
    and use its own member of pthread instead.  The idea is 1 simplify
    setxid handling to now mix atomic operation with thread cancellation
    since both are orthogonal, 2. isolate cancelhandling variable to only
    track thread cancel state.
    
    It also uses glibc C11 atomic operation while accessing new setxid_op
    field.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, and powerpc64le-linux-gnu.
    
     	* nptl/allocatestack.c (get_cached_stack): Set sexid_op on new
     	allocated stack.
    	(setxid_mark_thread): Use setxid_op to mark thread as executing
    	a sexid operation insteaf of using cancelhandling.  Also adapt
     	algorithm to use C11 atomic analogous functions.
     	(setxid_unmark_thread): Likewise.
     	(setxid_signal_thread): Likewise.
     	* nptl/nptl-init.c (sighandler_setxid): Likewise.
     	* nptl/pthread_create.c (sighandler_setxid): Likewise.
     	* nptl/descr.h (SETXID_BIT): Remove define.
     	(SETXID_BITMASK): Likewise.
     	(setxid_op): New member.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=c3f33edb679e95ded02b15fd5fe8d760a759ac3c

commit c3f33edb679e95ded02b15fd5fe8d760a759ac3c
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Jan 20 17:24:35 2017 -0200

    nptl: Move cancel state and type out cancelhandling
    
    This patch move both the cancel state (PTHREAD_CANCEL_ENABLE and
    PTHREAD_CANCEL_DISABLE) and cancel type (PTHREAD_CANCEL_DEFERRED and
    PTHREAD_CANCEL_ASYNCHRONOUS) out of cancelhandling member.  The idea
    is to avoid atomic handling of cancelhandling where these states
    are concerned since with exclusive member they are not concurrent
    accessed:
    
      * pthread_setcancel{type,state} and internal functions
        __pthread_{un}register_cancel_defer (and its analogous compat
        versions) only change the type/state of the calling thread.
      * __syscall_cancel also only requires to check the state of
        calling thread.
      * sigcancel_handler is executed only for the thread about to be
        potentially canceled.
      * pthread_join deadlock checks only requires access the state for
        the calling thread.
      * Same behavior for pthread_testcancel.
    
    With this change the cancelhandling member from pthread struct is
    used solely for cancelation handling and setxid signaling.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, and powerpc64le-linux-gnu.
    
    	* nptl/allocatestack.c (get_cached_stack): Set both cancelstate
    	and canceltype on new stack allocation.
    	* nptl/cleanup_defer.c (__pthread_register_cancel_defer): Set
    	canceltype directly instead of using cancelhandling member.
    	(__pthread_unregister_cancel_restore): Likewise.
    	* nptl/cleanup_defer_compat.c (_pthread_cleanup_push_defer):
    	Likewise.
    	(_pthread_cleanup_pop_restore): Likewise.
    	* nptl/descr.h (CANCELSTATE_BIT): Remove flag.
    	(CANCELSTATE_BITMASK): Likewise.
    	(CANCELTYPE_BIT): Likewise.
    	(CANCELTYPE_BITMSK): Likewise.
    	(CANCEL_RESTMASK): Likewise.
    	(CANCEL_ENABLED_AND_CANCELED): Likewise.
    	(CANCEL_ENABLED_AND_CANCELED_AND_ASYNCHRONOUS): Likewise.
    	(cancelstate): New member.
    	(canceltype): Likewise.
    	* nptl/libc-cancellation.c (__syscall_cancel): Use cancelstate
    	instead of cancelhandling member.
    	* nptl/nptl-init.c (sigcancel_handler): Likewise.
    	* nptl/pthreadP.h (__do_cancel): Likewise.
    	(CANCELLATION_P): Likewise.
    	* nptl/pthread_join.c (pthread_join): Remove CANCELLATION_P
    	usage.
    	* nptl/pthread_testcancel.c (__pthread_testcancel): Likewise.
    	* nptl/pthread_setcancelstate.c (__pthread_setcancelstate):
    	Use cancelstate member instead of cancelhandling.
    	* nptl/pthread_setcanceltype.c (__pthread_setcanceltype): Use
    	canceltype member instead of canceltype.
    	* sysdeps/x86_64/nptl/tcb-offsets.sym (TCB_CANCELSTATE_BITMASK):
    	Remove.
    	(TCB_CANCELTYPE_BITMASK): Likewise.
    	(TCB_CANCEL_RETMASK): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=ca81de5ed9021e4b96e7e3b0ce49eea806f92460

commit ca81de5ed9021e4b96e7e3b0ce49eea806f92460
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Feb 26 16:24:15 2018 -0300

    nptl: riscv: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the riscv modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    The default syscall_cancel.c should be as expected for RISCV. With
    GCC 7.3.1 syscall_cancel.c generates the following code:
    
    0000000000000000 <__GI___syscall_cancel_arch>:
       0:   8346                    mv      t1,a7
    
    0000000000000002 <__syscall_cancel_arch_start>:
       2:   00052883                lw      a7,0(a0)
       6:   0048f893                andi    a7,a7,4
       a:   00089c63                bnez    a7,22 <.L0 >
       e:   88ae                    mv      a7,a1
      10:   8532                    mv      a0,a2
      12:   85b6                    mv      a1,a3
      14:   863a                    mv      a2,a4
      16:   86be                    mv      a3,a5
      18:   8742                    mv      a4,a6
      1a:   879a                    mv      a5,t1
      1c:   00000073                ecall
    
    0000000000000020 <__syscall_cancel_arch_end>:
      20:   8082                    ret
    
      22:   1141                    addi    sp,sp,-16
      24:   e406                    sd      ra,8(sp)
      26:   00000097                auipc   ra,0x0
      2a:   000080e7                jalr    ra # 26 <.L0 >
    
    Checked with a build for riscv64-linux-gnu with run-built-tests=no.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/riscv/sigcontextinfo.h (ucontext_get_pc):
    	New fuction.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=33154acabe37c1bc4b4c180229d7f120b512dab9

commit 33154acabe37c1bc4b4c180229d7f120b512dab9
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Feb 3 21:07:07 2017 -0200

    nptl: hppa: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the hppa modifications required for the BZ#12683.
    It basically adds the required __pthread_get_pc function.
    
    HPPA requires an arch-specific syscall_cancel because the
    INTERNAL_SYSCALL_NCS adds some instruction to fetch the returned
    syscalls value.  The implementation were based on on default C
    version built with GCC 6.1
    
    Checked on hppa-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/hppa/sigcontextinfo.h: New file.
    	* sysdeps/unix/sysv/linux/hppa/syscall_cancel.S: Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=5f51f6b99358e17c177010167bd5dc56fecf1788

commit 5f51f6b99358e17c177010167bd5dc56fecf1788
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:03:07 2017 -0200

    nptl: mips: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the mips modifications required for the BZ#12683.
    It adds the required ucontext_get_pc function, a mips32 cancellable
    syscall wrapper and 7 argument cancellable syscall support.
    
    To avoid code pessimization and add a requirement on all architectures
    to support {INLINE,INTERNAL)_SYSCALL with 7 argument, its support is
    added through a flag, HAVE_CANCELABLE_SYSCALL_WITH_7_ARGS, which changes
    the signature and prototype of the requires macros and functions
    (SYSCALL_CANCEL, __syscall_cancel and __syscall_cancel_arch). As
    default 6 arguments cancellable syscalls are use.
    
    MIPS o32 requires an arch-specific implementation because
    INTERNAL_SYSCALL_NCS adds an 'addiu' just after the syscall
    instruction which invalidates the checks on sigcancel_handler.
    
    Checked against a build and make check run-built-tests=no for
    mips-gnu-linux, mips64-linux-gnu, mips64-n32-linux-gnu.  I also
    ran some basic o32 and n64 cancellation tests on a simulated
    mips64 qemu system.
    
    	[BZ #12683]
    	* nptl/libc-cancellation.c (__syscall_cancel): Define and use 7
    	argument syscall if architecture requires it.
    	* nptl/pthreadP.h (__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysdep.h (__syscall_cancel, __SYSCALL_CANCEL*): Define
    	with 7 argument if architecture requires it.
    	(__SYSCALL_CANCEL7_ARG_DEF): New macro.
    	(__SYSCALL_CANCEL7_ARG): Likewise.
    	(__SYSCALL_CANCEL7_ARG7): Likewise.
    	* sysdeps/unix/sysv/linux/syscall_cancel.c (__syscall_cancel_arch):
    	Likewise.
    	* sysdeps/mips/nptl/tls.h (READ_THREAD_POINTER): Check __mips_isa_rev
    	existance for macro definition.
    	* sysdeps/unix/sysv/linux/mips/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    	* sysdeps/unix/sysv/linux/mips/mips32/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/mips/mips32/sysdep.h
    	(HAVE_CANCELABLE_SYSCALL_WITH_7_ARGS): Define.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=e4db079ad7eea4a58dc6713dc2f8f1817e8de480

commit e4db079ad7eea4a58dc6713dc2f8f1817e8de480
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:03:03 2017 -0200

    nptl: sh: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the sh modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function and a
    cancellable syscall wrapper.
    
    SH requires an arch-specific syscall_cancel because the
    INTERNAL_SYSCALL_NCS adds the required or instruction to workaround
    a hardware bug [1].  The implementation were based on on default C
    version built with GCC 6.2.1.
    
    Checked against a build and make check run-built-tests=no for
    sh4-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/sh/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/sh/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    
    [1] http://documentation.renesas.com/eng/products/mpumcu/tu/tnsh7456ae.pdf

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=793aeb3010780a615f2f4b2236cad3f1707e88e4

commit 793aeb3010780a615f2f4b2236cad3f1707e88e4
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:53 2017 -0200

    nptl: nios2: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the nios2 modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    The default syscall_cancel.c should be as expected for NIOS2. With
    GCC 6.2.1 syscall_cancel.c generates the following code:
    
    ---
    00000000 <__GI___syscall_cancel_arch>:
       0:   defffe04        addi    sp,sp,-8
       4:   dd800015        stw     r22,0(sp)
       8:   002ce03a        nextpc  r22
       c:   02000034        movhi   r8,0
      10:   42000004        addi    r8,r8,0
      14:   dfc00115        stw     ra,4(sp)
      18:   b22d883a        add     r22,r22,r8
    
    0000001c <__syscall_cancel_arch_start>:
      1c:   20c00017        ldw     r3,0(r4)
      20:   18c0010c        andi    r3,r3,4
      24:   18000f1e        bne     r3,zero,64 <__syscall_cancel_arch_end+0x18>
      28:   3015883a        mov     r10,r6
      2c:   2805883a        mov     r2,r5
      30:   da400517        ldw     r9,20(sp)
      34:   380b883a        mov     r5,r7
      38:   da000417        ldw     r8,16(sp)
      3c:   d9c00317        ldw     r7,12(sp)
      40:   d9800217        ldw     r6,8(sp)
      44:   5009883a        mov     r4,r10
      48:   003b683a        trap    0
    
    0000004c <__syscall_cancel_arch_end>:
      4c:   38000126        beq     r7,zero,54 <__syscall_cancel_arch_end+0x8>
      50:   0085c83a        sub     r2,zero,r2
      54:   dfc00117        ldw     ra,4(sp)
      58:   dd800017        ldw     r22,0(sp)
      5c:   dec00204        addi    sp,sp,8
      60:   f800283a        ret
      64:   b0800017        ldw     r2,0(r22)
      68:   103ee83a        callr   r2
    ---
    
    Checked against a build and make check run-built-tests=no for
    nios2-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/nios2/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=14f629ccf56a06dba93cf6c1808496acb81edec8

commit 14f629ccf56a06dba93cf6c1808496acb81edec8
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:36 2017 -0200

    nptl: sparc: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the sparc modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function, cancellable
    syscall wrappers, and a fix for pause.
    
    Sparc requires arch-specific syscall_cancel implementation because
    INLINE_SYSCALL_NCS uses the __SYSCALL_STRING (defined different
    for sparc32 and sparc64) and it issues additional instructions
    after the syscall one to check the resulting error code.  When used
    in the default syscall_cancel.c implementation the label
    __syscall_cancel_arch_end is not placed just after the syscall as
    expected.  Both 32 and 64 bits version were based on default C version
    built with GCC 6.1.
    
    Also, different than other architectures, SPARC passes the sigcontext_t
    struct pointer as third argument in the signal handler set with
    SA_SIGINFO (some info at [1]) for 64 bits and the pt_regs in 32 bits.
    From Linux code:
    
    * arch/sparc/kernel/signal_64.c
    
    428         /* 3. signal handler back-trampoline and parameters */
    429         regs->u_regs[UREG_FP] = ((unsigned long) sf) - STACK_BIAS;
    430         regs->u_regs[UREG_I0] = ksig->sig;
    431         regs->u_regs[UREG_I1] = (unsigned long) &sf->info;
    432
    433         /* The sigcontext is passed in this way because of how it
    434          * is defined in GLIBC's /usr/include/bits/sigcontext.h
    435          * for sparc64.  It includes the 128 bytes of siginfo_t.
    436          */
    437         regs->u_regs[UREG_I2] = (unsigned long) &sf->info;
    
    * arch/sparc/kernel/signal_32.c:
    
    392         regs->u_regs[UREG_FP] = (unsigned long) sf;
    393         regs->u_regs[UREG_I0] = ksig->sig;
    394         regs->u_regs[UREG_I1] = (unsigned long) &sf->info;
    395         regs->u_regs[UREG_I2] = (unsigned long) &sf->regs;
    396
    397         regs->pc = (unsigned long) ksig->ka.sa.sa_handler;
    398         regs->npc = (regs->pc + 4);
    
    So to access the signal mask in the signal frame, a arch-specific
    ucontext_get_mask is defined which obtain the sa_mask from the context.
    
    Checked on a SPARC T5 for sparc64-linux-gnu and sparcv9-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/sparc/sparc32/sigcontextinfo.h
    	(ucontext_get_pc): New function.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/sigcontextinfo.h
    	(ucontext_get_pc): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/sparc32/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/syscall_cancel.S: Likwise.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/pause.c: New file.
    
    [1] https://www.spinics.net/lists/sparclinux/msg05037.html

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9f21ab0e6f18e0f548d36a6f460b7ac6ae5f476e

commit 9f21ab0e6f18e0f548d36a6f460b7ac6ae5f476e
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:28 2017 -0200

    nptl: tile: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the tile modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function
    
    The default syscall_cancel.c should be as expected for TILE. With
    GCC 6.1 syscall_cancel.c generates the following code:
    
    * TILEGX
    
    0000000000000000 <__GI___syscall_cancel_arch>:
       0:   c7bf46ce576bfd9d        { move r29, sp ; addi r28, sp, -24 ; st sp, lr }
       8:   283bf825401e0db6        { addi sp, sp, -32 ; move r10, r1 }
      10:   eeedf85b85c18d9b        { addi r27, sp, 24 ; move lr, r2 ; st r28, r29 }
      18:   eef5f88155bbf0c1        { move r1, r3 ; move r2, r4 ; st r27, r30 }
      20:   283bf8c25107f143        { move r3, r5 ; move r4, r6 }
      28:   283bf8e2d1483000        { move r5, r7 }
    
    0000000000000030 <__syscall_cancel_arch_start>:
      30:   9c5e4000340c3000        { ld4s r11, r0 }
      38:   18182165d1483000        { andi r11, r11, 4 }
      40:   17c00163d1483000        { bnez r11, 78 <__syscall_cancel_arch_end+0x20> }
      48:   283bfee051483000        { move r0, lr }
      50:   286b180051485000        { swint1 }
    
    0000000000000058 <__syscall_cancel_arch_end>:
      58:   180906ced1401ff7        { subx lr, zero, r1 ; addi r29, sp, 32 }
      60:   87b8c6ce4dd77040        { cmovnez r0, r1, lr ; addi r28, sp, 24 ; ld lr, r29 }
      68:   9ef6400035cc3000        { ld r30, r28 }
      70:   286a6ee040120db6        { addi sp, sp, 32 ; jrp lr }
      78:   2000000051483000        { jal 78 <__syscall_cancel_arch_end+0x20> }
    
    * TILEGX32
    
    00000000 <__GI___syscall_cancel_arch>:
       0:   cbbfc6ce576bfd9d        { move r29, sp ; addxi r28, sp, -8 ; st sp, lr }
       8:   283bf825402f0db6        { addxi sp, sp, -16 ; move r10, r1 }
      10:   eeedf860d5cbf0b7        { move lr, r2 ; move r1, r3 ; st r28, r29 }
      18:   283bf8a1d107f102        { move r2, r4 ; move r3, r5 }
      20:   283bf8e2d107f184        { move r4, r6 ; move r5, r7 }
    
    00000028 <__syscall_cancel_arch_start>:
      28:   9c5e4000340c3000        { ld4s r11, r0 }
      30:   18182165d1483000        { andi r11, r11, 4 }
      38:   17c0016351483000        { bnez r11, 68 <__syscall_cancel_arch_end+0x18> }
      40:   283bfee051483000        { move r0, lr }
      48:   286b180051485000        { swint1 }
    
    00000050 <__syscall_cancel_arch_end>:
      50:   28660fe540210d9d        { addxi r29, sp, 16 ; subx r10, zero, r1 }
      58:   9fbe40004dd4a040        { cmovnez r0, r1, r10 ; ld lr, r29 }
      60:   286a6ee040210db6        { addxi sp, sp, 16 ; jrp lr }
      68:   2000000051483000        { jal 68 <__syscall_cancel_arch_end+0x18> }
    
    Checked against a build and make check run-built-tests=no for
    tile-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/tile/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=5868f2b379c4dc9fed02585f5a9f6d41951a4b7f

commit 5868f2b379c4dc9fed02585f5a9f6d41951a4b7f
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:17 2017 -0200

    nptl: microblaze: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the microblaze modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    Microblaze requires an arch specific assembly implementation because
    the archicture INTERNAL_SYSCALL_NCS implementation adds a nop after
    the brki instruction because the instruction expect a delay branch
    slot.  I based this implementation on generated assembly using GCC 6.1.
    
    Checked against a build and make check run-built-tests=no for
    microblaze-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/microblaze/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/microblaze/sigcontextinfo.h
    	(ucontext_get_pc): New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a9d375373acbfc69dded5b12efbeef5c6ecdcf87

commit a9d375373acbfc69dded5b12efbeef5c6ecdcf87
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:08 2017 -0200

    nptl: m68k: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the m68k modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function
    
    The default syscall_cancel.c should is as expected for m68k with
    GCC 6.2.1.  It generates the following code for syscall_cancel.os:
    
    ---
    00000000 <__GI___syscall_cancel_arch>:
       0:   4e56 0000       linkw %fp,#0
       4:   48e7 3c00       moveml %d2-%d5,%sp@-
    
    00000008 <__syscall_cancel_arch_start>:
       8:   206e 0008       moveal %fp@(8),%a0
       c:   2010            movel %a0@,%d0
       e:   0800 0002       btst #2,%d0
      12:   6628            bnes 3c <__syscall_cancel_arch_end+0xa>
      14:   206e 0024       moveal %fp@(36),%a0
      18:   2a2e 0020       movel %fp@(32),%d5
      1c:   282e 001c       movel %fp@(28),%d4
      20:   262e 0018       movel %fp@(24),%d3
      24:   242e 0014       movel %fp@(20),%d2
      28:   222e 0010       movel %fp@(16),%d1
      2c:   202e 000c       movel %fp@(12),%d0
      30:   4e40            trap #0
    
    00000032 <__syscall_cancel_arch_end>:
      32:   4cee 003c fff0  moveml %fp@(-16),%d2-%d5
      38:   4e5e            unlk %fp
      3a:   4e75            rts
      3c:   61ff 0000 0000  bsrl 3e <__syscall_cancel_arch_end+0xc>
    ---
    
    Checked against a build and make check run-built-tests=no for
    m68k-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/m68k/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=5bd68272946551afa1d5578fc67b4acacf46e09a

commit 5bd68272946551afa1d5578fc67b4acacf46e09a
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:01:58 2017 -0200

    nptl: alpha: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the alpha modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function
    
    The default syscall_cancel.c should be fine for alpha and GCC 6.1
    generates the following cod for syscall_cancel.os:
    
    ---
    0000000000000000 <__GI___syscall_cancel_arch>:
       0:   00 00 bb 27     ldah    gp,0(t12)
       4:   00 00 bd 23     lda     gp,0(gp)
       8:   f0 ff de 23     lda     sp,-16(sp)
       c:   00 04 f1 47     mov     a1,v0
      10:   00 00 5e b7     stq     ra,0(sp)
    
    0000000000000014 <__syscall_cancel_arch_start>:
      14:   00 00 30 a0     ldl     t0,0(a0)
      18:   01 00 e1 43     sextl   t0,t0
      1c:   01 90 20 44     and     t0,0x4,t0
      20:   0f 00 20 f4     bne     t0,60 <__syscall_cancel_arch_end+0x20>
      24:   10 04 f2 47     mov     a2,a0
      28:   11 04 f3 47     mov     a3,a1
      2c:   12 04 f4 47     mov     a4,a2
      30:   10 00 9e a6     ldq     a4,16(sp)
      34:   13 04 f5 47     mov     a5,a3
      38:   18 00 be a6     ldq     a5,24(sp)
      3c:   83 00 00 00     callsys
    
    0000000000000040 <__syscall_cancel_arch_end>:
      40:   21 05 e0 43     negq    v0,t0
      44:   00 00 5e a7     ldq     ra,0(sp)
      48:   c0 04 61 46     cmovne  a3,t0,v0
      4c:   10 00 de 23     lda     sp,16(sp)
      50:   01 80 fa 6b     ret
      54:   00 00 fe 2f     unop
      58:   1f 04 ff 47     nop
      5c:   00 00 fe 2f     unop
      60:   00 00 7d a7     ldq     t12,0(gp)
      64:   00 40 5b 6b     jsr     ra,(t12),68 <__syscall_cancel_arch_end+0x28>
      68:   1f 04 ff 47     nop
      6c:   00 00 fe 2f     unop
    ---
    
    Checked on alpha-linux-gnu, no regression found.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/alpha/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=3a5a8aa8f6e665fb0fdf3ab6befbaa4a9b91a913

commit 3a5a8aa8f6e665fb0fdf3ab6befbaa4a9b91a913
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:01:44 2017 -0200

    nptl: ia64: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the s390 modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function. a workaround
    for mismatched sigcontext::sa_flags defiition between kernel and
    GLIBC (tracked by BZ#21634), and an arch specific syscall_cancel
    implementation.
    
    IA64 requires an arch-specific syscall_cancel implemetantion because
    {INLINE,INTERNAL}_SYSCALL is implemented by branching to a gate
    DSO (similar to i386) which renders the pointer comparison in
    SIGCANCEL handler wrong.  This incurs in performance penalty due the
    use of a break instruction insteaf of a eds one, however
    cancellable syscalls should potentially incur in syscalls blocking.
    
    Checked with a ia64-linux-gnu build with run-tests-built=no.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/ia64/sigcontextinfo.h (ucontext_get_pc,
    	ucontext_get_mask): New functions.
    	* sysdeps/unix/sysv/linux/ia64/syscall_cancel.S: New file.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=d47c1d0fa0b4ce173b9fd06caaf7a999a6a78af8

commit d47c1d0fa0b4ce173b9fd06caaf7a999a6a78af8
Author: Adhemerval Zanella <adhemerval.zanella@linaro.com>
Date:   Wed Aug 12 10:51:38 2015 -0300

    nptl: s390: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the s390 modifications required for the BZ#12683 fix.
    It basically adds the required ucontext_get_pc function.
    
    The built cancelable syscall wrapper for s390 using GCC 7.2.1 and
    default configuration flags shows the wrappers on expected corrected
    places:
    
    ---
    __GI___syscall_cancel_arch:
    .LFB39:
            .cfi_startproc
            stm     %r6,%r15,24(%r15)
            .cfi_offset 6, -72
            .cfi_offset 7, -68
            .cfi_offset 8, -64
            .cfi_offset 9, -60
            .cfi_offset 10, -56
            .cfi_offset 11, -52
            .cfi_offset 12, -48
            .cfi_offset 13, -44
            .cfi_offset 14, -40
            .cfi_offset 15, -36
            ahi     %r15,-96
            .cfi_def_cfa_offset 192
    
            .global __syscall_cancel_arch_start
    .type __syscall_cancel_arch_start,@function
    __syscall_cancel_arch_start:
            l       %r0,0(%r2)
            tml     %r0,4
            jne     .L5
            lr      %r1,%r3
            lr      %r2,%r4
            lr      %r3,%r5
            lr      %r4,%r6
            l       %r5,192(%r15)
            l       %r6,196(%r15)
            l       %r7,200(%r15)
            svc    0
    
            .global __syscall_cancel_arch_end
    .type __syscall_cancel_arch_end,@function
    __syscall_cancel_arch_end:
            l       %r4,152(%r15)
            lm      %r6,%r15,120(%r15)
            .cfi_remember_state
            .cfi_restore 15
            .cfi_restore 14
            .cfi_restore 13
            .cfi_restore 12
            .cfi_restore 11
            .cfi_restore 10
            .cfi_restore 9
            .cfi_restore 8
            .cfi_restore 7
            .cfi_restore 6
            .cfi_def_cfa_offset 96
            br      %r4
    .L5:
            .cfi_restore_state
            brasl   %r14,__syscall_do_cancel
            .cfi_endproc
    ---
    
    The s390x version also shows similar placement:
    
    ---
    __GI___syscall_cancel_arch:
            .cfi_startproc
            stmg    %r6,%r15,48(%r15)
            .cfi_offset 6, -112
            .cfi_offset 7, -104
            .cfi_offset 8, -96
            .cfi_offset 9, -88
            .cfi_offset 10, -80
            .cfi_offset 11, -72
            .cfi_offset 12, -64
            .cfi_offset 13, -56
            .cfi_offset 14, -48
            .cfi_offset 15, -40
            aghi    %r15,-160
            .cfi_def_cfa_offset 320
    
            .global __syscall_cancel_arch_start
    .type __syscall_cancel_arch_start,@function
    __syscall_cancel_arch_start:
            l       %r0,0(%r2)
            tmll    %r0,4
            jne     .L5
            lgr     %r1,%r3
            lgr     %r2,%r4
            lgr     %r3,%r5
            lgr     %r4,%r6
            lg      %r5,320(%r15)
            lg      %r6,328(%r15)
            lg      %r7,336(%r15)
            svc    0
    
            .global __syscall_cancel_arch_end
    .type __syscall_cancel_arch_end,@function
    __syscall_cancel_arch_end:
            lg      %r4,272(%r15)
            lmg     %r6,%r15,208(%r15)
            .cfi_remember_state
            .cfi_restore 15
            .cfi_restore 14
            .cfi_restore 13
            .cfi_restore 12
            .cfi_restore 11
            .cfi_restore 10
            .cfi_restore 9
            .cfi_restore 8
            .cfi_restore 7
            .cfi_restore 6
            .cfi_def_cfa_offset 160
            br      %r4
    .L5:
            .cfi_restore_state
            brasl   %r14,__syscall_do_cancel
            .cfi_endproc
    ---
    
    Checked on s390-linux-gnu and s390x-linux-gnu build with
    run-tests-built=no.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/s390/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=4c686923b51f81cca01ad125f403b50fe05eeef4

commit 4c686923b51f81cca01ad125f403b50fe05eeef4
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri May 8 17:12:31 2015 -0300

    nptl: arm: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the ARM modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function and adjust
    the generic syscall_cancel build.
    
    For ARM we need to build syscall_cancel in ARM mode (-marm) to avoid
    INTERNAL_SYSCALL to issue the syscall through the helper gate
    __libc_do_syscall (which invalidates the mark checks on SIGCANCEL
    handler).
    
    Checked on arm-linux-gnueabihf.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/arm/Makefile (CFLAGS-syscall_cancel.c): New
    	rule.
    	* sysdeps/unix/sysv/linux/arm/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=1147b8b7f935fadfc4bc75f44bd50e5bebf4474e

commit 1147b8b7f935fadfc4bc75f44bd50e5bebf4474e
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Wed May 6 17:51:29 2015 -0300

    nptl: aarch64: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the aarch64 modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    The built cancelable syscall wrapper for aarch64 using GCC 7.2.1 and
    default configuration flags shows an expected optimized version:
    
    ---
    __GI___syscall_cancel_arch:
    .LFB38:
            .cfi_startproc
            .global __syscall_cancel_arch_start
    .type __syscall_cancel_arch_start,@function
    __syscall_cancel_arch_start:
            ldr     w9, [x0]
            tbnz    x9, 2, .L7
            mov     x8, x1
            mov     x0, x2
            mov     x1, x3
            mov     x2, x4
            mov     x3, x5
            mov     x4, x6
            mov     x5, x7
            svc     0       // syscall nr
            .global __syscall_cancel_arch_end
    .type __syscall_cancel_arch_end,@function
    __syscall_cancel_arch_end:
            ret
    ---
    
    A similar code is obtained with GCC 5.3.1, so I see to need to provide
    an arch-specific syscall_cancel.S for aarch64.
    
    Checked on aarch64-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/aarch64/sysdep-cancel.h (PSEUDO): Redefine
    	to call __syscall_cancel function for cancellable syscalls.
    	(__pthread_get_ip): Add implementation.
    	* sysdeps/unix/sysv/linux/aarch64/sysdep.h (SYSCALL_CANCEL_ERROR): Add
    	definition.
    	(SYSCALL_CANCEL_ERRNO): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=c31e9f5f8549cfe2a3585d497d0764f2496c86a1

commit c31e9f5f8549cfe2a3585d497d0764f2496c86a1
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Sep 18 18:14:19 2015 -0300

    nptl: powerpc: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches adds the powerpc64 modification required for the BZ#12683.
    It basically adds the required __pthread_get_pc function and a arch
    specific syscall_cancel implementation.
    
    The powerpc requires an arch-specific syscall_cancel because
    INTERNAL_SYSCALL_NCS adds a mfcr just after the sc instruction to get
    the CR0.SO bit information from kernel (which signals the error
    return status).  So for cancelled syscalls with side effects,
    __pthread_get_pc will point to mcfr and thus invalidating the checks
    on sigcancel_handler.
    
    Checked on powerpc64le-linux-gnu and powerpc-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/powerpc/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/powerpc/sysdep-cancel.h
    	(__pthread_get_pc): New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=3bf87610132f283253be992d73762c2792752c14

commit 3bf87610132f283253be992d73762c2792752c14
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon May 4 16:30:13 2015 -0300

    nptl: i386: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the i386 modifications required for the BZ#12683.
    It basically provides the required ucontext_get_pc symbol, add the
    cancelable syscall wrapper and fix a thread atomic update macro.
    
    On i386 an arch-specific cancellation implementation is required
    because depending of the glibc configuration and underlying kernel
    the syscall may be done using a vDSO symbol (__kernel_vsyscall).
    By using the vDSO symbol the resulting PC value for an interrupted
    syscall points to an adress outside the expected markers in
    __syscall_cancel_arch.  It has been discussed in LKML [1] on how
    kernel could help userland to accomplish it, but afaik discussion
    was stalled.
    
    Also, since glibc supports i486, the old 'int 0x80' should be used
    in the syscall wrapper.  One option could make minimum default chip
    to pentium II (which implements sysenter) or add a runtime check
    on syscall_cancel.S to use 'int 0x80' or sysenter.
    
    Similar to x86_64, it also remove bogus arch-specific
    THREAD_ATOMIC_BIT_SET where it always reference to current thread
    instead of the one referenced by input 'descr' argument.
    
    Checked on i686-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/i386/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL,
    	THREAD_ATOMIC_BIT_SET): Remove macro.
    	* sysdeps/unix/sysv/linux/i386/Makefile
    	[$(subdir) = elf] (sysdep-rtld_routines): Add libc-do-syscall object.
    	* sysdeps/unix/sysv/linux/i386/lowlevellock.h (lll_wait_tid): Use
    	cancellable futex syscall macro.
    	* sysdeps/unix/sysv/linux/i386/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/i386/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    
    [1] https://lkml.org/lkml/2016/3/8/1105

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=c3619359a7a15ed73b7c5e7356173dcc2adf5d2c

commit c3619359a7a15ed73b7c5e7356173dcc2adf5d2c
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Sat May 9 14:20:26 2015 -0300

    nptl: x32: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches adds the x32 modification required for the BZ#12683.
    It basically adjust the syscall size used to pass the arguments to
    the syscall cancel wrappers by zero extending pointers type while
    preserting values for default types (such as off_t).
    
    Checked on x86_64-linux-gnu-x32.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/x86_64/x32/sysdep.h (__syscall_arg_t):
    	Define type for x32.
    	(__SSC): Add platform specific macro.
    	* include/libc-pointer-arith.h (__integer_if_pointer_type_sub,
    	__integer_if_pointer_type, cast_to_integer): Parametrize integer type
    	cast.
    	(cast_to_uinteger): New macro.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=13c8c4809c7703c09d77a31601cba75cdc3de5d4

commit 13c8c4809c7703c09d77a31601cba75cdc3de5d4
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Sep 29 09:48:34 2014 -0300

    nptl: x86_64: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches adds the x86_64 modification required for the BZ#12683.
    It basically provide the required ucontext_get_pc symbol and remove
    the arch-specific libc-cancellation implementations.
    
    It also remove bogus arch-specific THREAD_ATOMIC_BIT_SET where it
    always reference to current thread instead of the one referenced by
    input 'descr' argument. It works as long the input is the self thread
    pointer, however it creates wrong code is it used along with a
    description to a different one (as on nptl/pthread_cancel.c).
    
    The code generated create an additional load to reference to TLS segment,
    for instance the code:
    
      THREAD_ATOMIC_BIT_SET (THREAD_SELF, cancelhandling, CANCELED_BIT);
    
    Compiles to:
    
      lock;orl $4, %fs:776
    
    Where with patch changes it now compiles to:
    
      mov %fs:16,%rax
      lock;orl $4, 776(%rax)
    
    If some usage indeed proves to be a hotspot we can add an extra macro
    with a more descriptive name (THREAD_ATOMIC_BIT_SET_SELF for instance)
    where x86_64 might optimize it.  In fact all x86_64 THREAD_ATOMIC_* macros
    do not respect the input descr and possible will fail when used with
    a 'descr' difference than THREAD_SELF.
    
    Checked on x86_64-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/x86_64/cancellation.S: Remove file.
    	* sysdeps/unix/sysv/linux/x86_64/libc-cancellation.S: Remove file.
    	* sysdeps/unix/sysv/linux/x86_64/librt-cancellation.S: Remove file.
    	* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (lll_wait_tid):
    	Use cancellable futex wait call.
    	* sysdeps/unix/sysv/linux/x86_64/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    	* sysdeps/x86_64/nptl/tcb-offsets.sym (TCB_CANCELING_BITMASK):
    	Remove.
    	* sysdeps/x86_64/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL,
    	THREAD_ATOMIC_BIT_SET): Remove macros.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=5c75e9c9cbecc95437a4bc5c9b030bd3af63116b

commit 5c75e9c9cbecc95437a4bc5c9b030bd3af63116b
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Sep 18 18:26:35 2015 -0300

    nptl: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches fixes some race conditions in NPTL cancellation code by
    redefining how cancellable syscalls are defined and handled.  Current
    approach is to enable asynchronous cancellation prior to making the syscall
    and restore the previous cancellation type once the syscall returns.
    
    As decribed in BZ#12683, this approach shows 2 important problems:
    
      1. Cancellation can act after the syscall has returned from kernel, but
         before userspace saves the return value.  It might result in a resource
         leak if the syscall allocated a resource or a side effect (partial
         read/write), and there is no way to program handle it with cancellation
         handlers.
    
      2. If a signal is handled while the thread is blocked at a cancellable
         syscall, the entire signal handler runs with asynchronous cancellation
         enabled.  This can lead to issues if the signal handler call functions
         which are async-signal-safe but not async-cancel-safe.
    
    For cancellation to work correctly, there are 5 points at which the
    cancellation signal could arrive:
    
      1. Before the final "testcancel" and before the syscall is made.
      2. Between the "testcancel" and the syscall.
      3. While the syscall is blocked and no side effects have yet taken place.
      4. While the syscall is blocked but with some side effects already having
         taken place (e.g. a partial read or write).
      5. After the syscall has returned.
    
    And GLIBC wants to act on cancellation in cases 1, 2, and 3 but not in case
    4 or 5.  The proposed solution follows:
    
      * Handling case 1 is trivial: do a conditional branch based on whether the
        thread has received a cancellation request;
      * Case 2 can be caught by the signal handler determining that the saved
        program counter (from the ucontext_t) is in some address range beginning
        just before the "testcancel" and ending with the syscall instruction.
      * In this case, except for certain syscalls that ALWAYS fail with EINTR
        even for non-interrupting signals, the kernel will reset the program
        counter to point at the syscall instruction during signal handling, so
        that the syscall is restarted when the signal handler returns. So, from
        the signal handler's standpoint, this looks the same as case 2, and thus
        it's taken care of.
      * In this case, the kernel cannot restart the syscall; when it's
        interrupted by a signal, the kernel must cause the syscall to return
        with whatever partial result it obtained (e.g. partial read or write).
      * In this case, the saved program counter points just after the syscall
        instruction, so the signal handler won't act on cancellation.
        This one is equal to 4. since the program counter is past the syscall
        instruction already.
    
    Another case that needs handling is syscalls that fail with EINTR even
    when the signal handler is non-interrupting. In this case, the syscall
    wrapper code can just check the cancellation flag when the errno result
    is EINTR, and act on cancellation if it's set.
    
    The proposed GLIBC adjustments are:
    
      1. Remove the enable_asynccancel/disable_asynccancel function usage in
         syscall definition and instead make them call a common symbol that will
         check if cancellation is enabled (__syscall_cancel at
         nptl/libc-cancellation.c), call the arch-specific cancellable
         entry-point (__syscall_cancel_arch) and cancel the thread when required.
    
      2. Provide a arch-specific symbol that contains global markers. These
         markers will be used in SIGCANCEL handler to check if the interruption
         has been called in a valid syscall and if the syscalls has been
         completed or not.
         A default version is provided (sysdeps/unix/sysv/linux/syscall_cancel.c),
         however the markers may not be set on correct expected places depeding
         of how INTERNAL_SYSCALL_NCS is implemented by the underlying architecture.
         In this case arch-specific implementation should be provided.
    
      3. Rewrite SIGCANCEL asynchronous handler to check for both cancelling type
         and if current IP from signal handler falls between the global markes
         and act accordingly (sigcancel_handler at nptl/nptl-init.c).
    
      4. Adjust nptl/pthread_cancel.c to send an signal instead of acting
         directly. This avoid synchronization issues when updating the
         cancellation status and also focus the logic on signal handler and
         cancellation syscall code.
    
      5. Adjust pthread code to replace CANCEL_ASYNC/CANCEL_RESET calls to
         appropriated cancelable futex syscalls.
    
      6. Adjust libc code to replace LIBC_CANCEL_ASYNC/LIBC_CANCEL_RESET to
         appropriated cancelable syscalls.
    
      7. Adjust 'lowlevellock-futex.h' arch-specific implementations to provide
         cancelable futex calls (used in libpthread code).
    
    This patch adds the proposed changes to NPTL.  The code leaves all the ports
    broken without further patches in the list.
    
    	[BZ #12683]
    	* nptl/Makefile [routines]: Add syscall_cancel object.
    	[libpthread-routines]: Remove cancellation object.
    	(CFLAGS-cancellation.c): Remove rule.
    	(CFLAGS-syscall_cancel.c): New rule.
    	(tests): Add tst-cancel28.
    	* nptl/Versions [GLIBC_PRIVATE] (libc): Add __syscall_cancel,
    	__syscall_cancel_arch_start, and __syscall_cancel_arch_end.
    	* nptl/cancellation.c: Remove file.
    	* nptl/descr.h (CANCELING_BIT): Remove define.
    	(CANCELING_BITMASK): Likewise.
    	(CANCEL_RESTMASK): Adjust value with CANCELED_BIT remove.
    	* nptl/libc-cancellation.c (__syscall_cancel): Add non-cancellable
    	implementation for loader and cancellable one for libc.
    	(__syscall_do_cancel): New function: cancel call for syscall wrappers.
    	* nptl/lll_timedlock_wait.c (__lll_timedlock_wait): Use cancellable
    	futex operation.
    	(__lll_timedwait_tid): Likewise.
    	* nptl/nptl-init.c (sigcancel_handler): Rewrite function to avoid race
    	conditions.
    	(__pthread_initialize_minimal_internal): Add SA_RESTART to SIGCANCEL
    	handler.
    	* nptl/pt-system.c [LIBC_CANCEL_HANDLED]: Remove definition.
    	* io/creat.c (LIBC_CANCEL_HANDLED): Likewise.
    	* io/ppoll.c [ppoll] (LIBC_CANCEL_HANDLED): Likewise.
    	* misc/pselect [__pselect] (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/posix/pause.c (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/generic/creat.c (LIBC_CANCEL_HANDLED):
    	Likewise.
    	* nptl/pthreadP.h (__do_cancel): Rewrite to both disable asynchronous
    	cancellation and setting the thread as cancelled.
    	(CANCEL_ASYNC): Remove definition.
    	(CANCEL_RESET): Likewise.
    	(LIBC_CANCEL_ASYNC): Likewise.
    	(LIBC_CANCEL_RESET): Likewise.
    	(LIBC_CANCEL_HANDLED): Likewise.
    	(__syscall_cancel_arch): Add prototype.
    	(__pthread_enable_asynccancel): Remove prototype.
    	(__pthread_disable_asynccancel): Likewise.
    	(__libc_enable_asynccancel): Likewise.
    	(__libc_disable_asynccancel): Likewise.
    	(__librt_enable_asynccancel): Likewise.
    	(__librt_disable_asynccancel): Likewise.
    	(__syscall_cancel_arch): Add prototype.
    	(__syscall_do_cancel): Likewise.
    	* nptl/pthread_cancel.c (pthread_cancel): Rewrite to just set
    	CANCELLED_BIT and call __pthread_kill.
    	* nptl/pthread_create.c (start_thread): Likewise.
    	* nptl/pthread_timedjoin.c (pthread_timedjoin_np): Likewise.
    	* nptl/sem_timedwait.c (sem_timedwait):  Likewise.
    	* nptl/sem_wait.c (__new_sem_wait): Likewise.
    	* nptl/sem_waitcommon.c (futex_abstimed_wait): Likewise.
    	* sysdeps/nptl/aio_misc.h (AIO_MISC_WAIT): Likewise.
    	* sysdeps/nptl/gai_misc.h (GAI_MISC_WAIT): Likewise.
    	* sysdeps/posix/sigpause.c (do_sigpause): Likewise.
    	* sysdeps/posix/sigwait.c (__sigwait): Likewise.
    	* sysdeps/posix/waitid.c (__waitid): Likewise.
    	* sysdeps/nptl/lowlevellock.h (lll_wait_tid): Likewise.
    	* sysdeps/posix/open64.c (__libc_open64): Likewise.
    	* sysdeps/unix/sysv/linux/sigwait.c (__sigwait): Likewise.
    	* nptl/pthread_exit.c (pthread_exit): Rewrite to set EXITING_BIT
    	before call __pthread_unwind.
    	* nptl/pthread_join.c (pthread_join): Remove CANCEL_ASYNC/CANCEL_RESET
    	usage.
    	* rt/Makefile [CFLAGS-librt-cancellation.c]: Remove rule.
    	* sysdeps/generic/sysdep-cancel.h (LIBC_CANCEL_ASYNC): Remove define.
    	(LIBC_CANCEL_RESET): Likewise.
    	(LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/clock_nanosleep.c (__clock_nanosleep):
    	Likewise.
    	* sysdeps/unix/sysv/linux/fcntl.c (__libc_fcntl): Likewise.
    	* sysdeps/unix/sysv/linux/generic/wordsize-32/fcntl.c (__libc_fcntl):
    	Likewise.
    	* sysdeps/nptl/Makefile [$(subdir) = rt] (librt-sysdep_routines):
    	Remove librt-cancellation object.
    	[$(subdir) = rt] (librt-cancellation.c): Remove rule.
    	* sysdeps/nptl/librt-cancellation.c: Remove file.
    	* sysdeps/unix/sysv/linux/futex-internal.h (lll_futex_wait_cancel):
    	Use lll_futex_timed_wait_cancel.
    	(futex_reltimed_wait_cancelable): Likewise.
    	(futex_abstimed_wait_cancelable)): Use
    	lll_futex_timed_wait_bitset_cancel.
    	* sysdeps/unix/sysv/linux/lowlevellock-futex.h
    	(lll_futex_wait_cancel): New macro.
    	(lll_futex_timed_wait_cancel): Likewise.
    	(lll_futex_timed_wait_bitset_cancel): Likewise.
    	* sysdeps/unix/sysdep.h (SYSCALL_CANCEL): New macro: cancelable
    	syscall calls.
    	(INTERNAL_SYSCALL_NCS_CALL): New macro.
    	(__syscall_cancel): New prototype.
    	* sysdeps/unix/sysv/linux/socketcall.h (SOCKETCALL): Use __SSC macros.
    	(SOCKETCALL_CANCEL): Use SYSCALL_CANCEL macros.
    	* sysdeps/generic/sysdep-cancel.h (LIBC_CANCEL_ASYNC): Remove define.
    	(LIBC_CANCEL_RESET): Likewise.
    	(LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/pthread_kill.c (__pthread_kill): Allow
    	SIGCANCEL to be sent.
    	* nptl/tst-cancel28.c: New file.
    	* sysdeps/unix/sysv/linux/syscall_cancel.c: Likewise.
    	* support/temp_file.c (create_temp_fifo): New function.
    	* support/temp_file.h (create_temp_fifo): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=0796a7c793e817c84a4e06e405ed902c41d0a8de

commit 0796a7c793e817c84a4e06e405ed902c41d0a8de
Author: Adhemerval Zanella <adhemerval.zanella@linaro.com>
Date:   Mon Sep 21 15:55:58 2015 -0700

    nptl: Fix testcases for new pthread cancellation mechanism
    
    With upcoming fix for BZ#12683, pthread cancellation does not act for:
    
      1. If syscall is blocked but with some side effects already having
         taken place (e.g. a partial read or write).
      2. After the syscall has returned.
    
    The main change is due the fact programs need to act in syscalls with
    side-effects (for instance, to avoid leak of allocated resources or
    handle partial read/write).
    
    This patch changes the NPTL testcase that assumes the old behavior and
    also remove the tst-cancel-wrappers.sh test (which checks for symbols
    that will not exist anymore).  For tst-cancel{2,3} case it remove the
    pipe close because it might cause the write syscall to return with
    side effects if the close is executed before the pthread cancel.
    
    It also changes how to call the read syscall on tst-backtrace{5,6}
    to use syscall instead of read cancelable syscall to avoid need to
    handle the cancelable bridge function calls.  It requires a change
    on powerpc syscall implementation to create a stackframe, since
    powerpc backtrace rely on such information.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, powerpc64le-linux-gnu,
    powerpc-linux-gnu, sparcv9-linux-gnu, and sparc64-linux-gnu.
    
    	* debug/tst-backtrace5.c (handle_signal): Check for syscall
    	instead of read.
    	(fn): Issue the read syscall instead of call the cancellable
    	syscall.
    	* nptl/Makefile [$(run-built-tests) = yes] (tests-special): Remove
    	tst-cancel-wrappers.sh.
    	* nptl/tst-cancel-wrappers.sh: Remove file.
    	* nptl/tst-cancel2.c (do_test): Do not close pipe.
    	* nptl/tst-cancel3.c (do_test): Likewise.
    	* nptl/tst-cancel4.c (tf_write): Handle cancelled syscall with
    	side-effects.
    	(tf_send): Likewise.
    	* sysdeps/unix/sysv/linux/powerpc/syscall.S (syscall): Create stack
    	frame.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9a5fd16b1327cdcab9cd26687c7f70779c2d63dd

commit 9a5fd16b1327cdcab9cd26687c7f70779c2d63dd
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Feb 26 17:52:02 2018 -0300

    powerpc: Create stackframe information on syscall
    
    This patch adds a minimal stackframe creation on powerpc syscall
    implementation so backtrace works correctly on a signal handler.
    
    Checked on powerpc64le-linux-gnu.
    
    	* sysdeps/unix/sysv/linux/powerpc/syscall.S (syscall): Create stack
    	frame.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=7a2c0ea11bac560392d4002626129523968f6c22

commit 7a2c0ea11bac560392d4002626129523968f6c22
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Thu Feb 22 15:47:33 2018 -0300

    nptl: Fix tst-cancel4 sendto tests
    
    Now that send might be implemented calling sendto syscall on Linux,
    I am seeing some issue in some kernel configurations where tst-cancel4
    sendto do not block as expected.
    
    The socket used to force the syscall blocking is used with default
    system configuration for buffer sending size, which might not be
    correct to force blocking.  This patch fixes it by explicit setting
    buffer socket lower than the buffer size used.  It also enabled sendto
    cancellation tests to work in both ways (since internally send is
    implemented routing to sendto on Linux kernel).
    
    The patch also removes unrequired make rules on some archictures
    for send/recv. The generic nptl Makefile already set the compiler flags
    required on some architectures for correct unwinding and libc object
    are not strictly required to support unwind (since pthread_cancel
    requires linking against libpthread).
    
    Checked on aarch64-linux-gnu and x86_64-linux-gnu. I also did a
    sniff test with tst-cancel{4,5} on a simulated mips64-linux-gnu.
    
    	* nptl/tst-cancel4-common.h (set_socket_buffer): New function.
    	* nptl/tst-cancel4-common.c (do_test): Call set_socket_buffer
    	for socketpair endpoint.
    	* nptl/tst-cancel4.c (tf_send): Call set_socket_buffer and use
    	WRITE_BUFFER_SIZE as buffer size for sending socket.
    	(tf_sendto): Use SOCK_STREAM instead of SOCK_DGRAM and fix an
    	issue on system where send is implemented with sendto syscall.
    	* sysdeps/unix/sysv/linux/mips/mips64/Makefile [$(subdir) = socket]
    	(CFLAGS-recv.c, CFLAGS-send.c): Remove rules.
    	[$(subdir) = nptl] (CFLAGS-recv.c, CFLAGS-send.c): Likewise.
    	* sysdeps/unix/sysv/linux/riscv/rv64/Makefile: Remove file.

-----------------------------------------------------------------------
Comment 30 cvs-commit@gcc.gnu.org 2018-05-14 20:43:34 UTC
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".

The branch, azanella/bz12683 has been created
        at  2e8c8e9622f0591fdffb7b5e79d1b6937af52e6a (commit)

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=2e8c8e9622f0591fdffb7b5e79d1b6937af52e6a

commit 2e8c8e9622f0591fdffb7b5e79d1b6937af52e6a
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue Jan 24 21:05:56 2017 -0200

    nptl: Remove THREAD_ATOMIC_* macros
    
    This patch removes the ununsed THREAD_ATOMIC_* macros now that nptl
    code is using C11 atomics.
    
    Checked on x86_64-linux-gnu and i686-linux-gnu.
    
    	* manual/pattern.texi (THREAD_ATOMIC_BIT_SET): Remove.
    	(THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	* nptl/pthreadP.h (THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	(THREAD_ATOMIC_BIT_SET): Likewise.
    	* sysdeps/i386/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	(THREAD_ATOMIC_AND): Likewise.
    	* sysdeps/x86_64/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL): Likewise.
    	(THREAD_ATOMIC_AND): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=c905cddb3868b5128b0460239e0cc5d609a7ef65

commit c905cddb3868b5128b0460239e0cc5d609a7ef65
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue Jan 24 09:17:37 2017 -0200

    nptl: cancelhandling refactor
    
    This patch basically uses glibc C11 atomic function to access and
    modify struct pthread cancelhandling variable.  All plain access
    are handled using the atomic_load_* functions and THREAD_ATOMIC_BIT_SET
    is replaced by atomic_fetch_or_acquire.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, and powerpc64le-linux-gnu.
    
    	* nptl/descr.h (CANCELED_BITMASK): Rename to THREAD_CANCELED.
    	(EXITING_BITMASK): Rename to THREAD_EXITING.
    	(TERMINATED_BITMARK): Rename to THREAD_TERMINATED.
    	* nptl/allocatestack.c (setxid_mark_thread): Use THREAD_* instead of
    	*_BITMASK and use glibc C11 atomic function to access pthread
    	cancelhandling.
    	* nptl/libc-cancellation.c (__syscall_cancel): Likewise.
    	* nptl/nptl-init.c (sigcancel_handler): Likewise.
    	* nptl/pthread_cancel.c (pthread_cancel): Likewise.
    	* nptl/pthread_create.c (__free_tcb): Likewise.
    	(START_THREAD_DEFN): Likewise.
    	(__pthread_create_2_1): Likewise.
    	* nptl/pthread_detach.c (pthread_detach): Likewise.
    	* nptl/pthread_exit.c (__pthread_exit): Likewise.
    	* nptl/pthread_join.c (pthread_join): Likewise.
    	* nptl_db/td_thr_get_info.c (td_thr_get_info): Likewise.
    	* nptl_db/td_thr_getfpregs.c (td_thr_getfpregs): Likewise.
    	* nptl_db/td_thr_getgregs.c (td_thr_getregs): Likewise.
    	* nptl_db/td_thr_setfpregs.c (td_thr_setfpregs): Likewise.
    	* nptl_db/td_thr_setgregs.c (td_thr_setgregs): Likewise.
    	* sysdeps/unix/sysv/linux/arm/syscall_cancel.S
    	(__syscall_cancel_arch): Change CANCELED_BITMASK to
    	THREAD_CANCELED in comment.
    	* sysdeps/unix/sysv/linux/syscall_cancel.c
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/x86_64/nptl/tcb-offsets.sym (TCB_CANCELED_BITMASK):
    	Remove.
    	(TCB_EXITING_BITMASK): Likewise.
    	(TCB_TERMINATED_BITMASK): Likewise.
    	* sysdeps/unix/sysv/linux/i386/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/powerpc/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/sh/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/sparc32/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/syscall_cancel.S
    	(__syscall_cancel_arch): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=2212de8d6ffc4b29baf170422656d784caab4959

commit 2212de8d6ffc4b29baf170422656d784caab4959
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue Jul 18 16:33:41 2017 -0300

    nptl: Remove setxid bit from cancelhandling
    
    This patch removes the setxid signaling out of cancelhandling flag
    and use its own member of pthread instead.  The idea is simplify
    setxid handling to not mix atomic operation with thread cancellation
    since both are orthogonal, and to isolate cancelhandling variable to
    only track thread cancel state.
    
    It also uses glibc C11 atomic operation while accessing new setxid_op
    field.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, and powerpc64le-linux-gnu.
    
     	* nptl/allocatestack.c (get_cached_stack): Set sexid_op on new
     	allocated stack.
    	(setxid_mark_thread): Use setxid_op to mark thread as executing
    	a sexid operation insteaf of using cancelhandling.  Also adapt
     	algorithm to use C11 atomic analogous functions.
     	(setxid_unmark_thread): Likewise.
     	(setxid_signal_thread): Likewise.
     	* nptl/nptl-init.c (sighandler_setxid): Likewise.
     	* nptl/pthread_create.c (sighandler_setxid): Likewise.
     	* nptl/descr.h (SETXID_BIT): Remove define.
     	(SETXID_BITMASK): Likewise.
     	(setxid_op): New member.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=f1b02e0cc90b72148e2755b55dff1faac3b9eb8b

commit f1b02e0cc90b72148e2755b55dff1faac3b9eb8b
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Jan 20 17:24:35 2017 -0200

    nptl: Move cancel state and type out cancelhandling
    
    This patch move both the cancel state (PTHREAD_CANCEL_ENABLE and
    PTHREAD_CANCEL_DISABLE) and cancel type (PTHREAD_CANCEL_DEFERRED and
    PTHREAD_CANCEL_ASYNCHRONOUS) out of cancelhandling member.
    
    The idea is to simplify their access since for most cases there is
    no concurrent access with cancellation mode:
    
      * pthread_setcancel{type,state} and internal functions
        __pthread_{un}register_cancel_defer (and its analogous compat
        versions) only change the type/state of the calling thread.
      * __syscall_cancel also only requires to check the state of
        calling thread.
      * sigcancel_handler is executed only for the thread about to be
        potentially canceled.
      * pthread_join deadlock checks only requires access the state for
        the calling thread.
      * Same behavior for pthread_testcancel.
    
    With this change the cancelhandling member from pthread struct is
    used solely for cancelation handling and setxid signaling.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, and powerpc64le-linux-gnu.
    
    	* nptl/allocatestack.c (get_cached_stack): Set both cancelstate
    	and canceltype on new stack allocation.
    	* nptl/cleanup_defer.c (__pthread_register_cancel_defer): Set
    	canceltype directly instead of using cancelhandling member.
    	(__pthread_unregister_cancel_restore): Likewise.
    	* nptl/cleanup_defer_compat.c (_pthread_cleanup_push_defer):
    	Likewise.
    	(_pthread_cleanup_pop_restore): Likewise.
    	* nptl/descr.h (CANCELSTATE_BIT): Remove flag.
    	(CANCELSTATE_BITMASK): Likewise.
    	(CANCELTYPE_BIT): Likewise.
    	(CANCELTYPE_BITMSK): Likewise.
    	(CANCEL_RESTMASK): Likewise.
    	(CANCEL_ENABLED_AND_CANCELED): Likewise.
    	(CANCEL_ENABLED_AND_CANCELED_AND_ASYNCHRONOUS): Likewise.
    	(cancelstate): New member.
    	(canceltype): Likewise.
    	* nptl/libc-cancellation.c (__syscall_cancel): Use cancelstate
    	instead of cancelhandling member.
    	* nptl/nptl-init.c (sigcancel_handler): Likewise.
    	* nptl/pthreadP.h (__do_cancel): Likewise.
    	(CANCELLATION_P): Likewise.
    	* nptl/pthread_join.c (pthread_join): Remove CANCELLATION_P
    	usage.
    	* nptl/pthread_testcancel.c (__pthread_testcancel): Likewise.
    	* nptl/pthread_setcancelstate.c (__pthread_setcancelstate):
    	Use cancelstate member instead of cancelhandling.
    	* nptl/pthread_setcanceltype.c (__pthread_setcanceltype): Use
    	canceltype member instead of canceltype.
    	* sysdeps/x86_64/nptl/tcb-offsets.sym (TCB_CANCELSTATE_BITMASK):
    	Remove.
    	(TCB_CANCELTYPE_BITMASK): Likewise.
    	(TCB_CANCEL_RETMASK): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=7102b569bba27db931ae98460469796757709741

commit 7102b569bba27db931ae98460469796757709741
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Feb 26 16:24:15 2018 -0300

    nptl: riscv: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the riscv modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    The default syscall_cancel.c should be as expected for RISCV. With
    GCC 7.3.1 syscall_cancel.c generates the following code:
    
    0000000000000000 <__GI___syscall_cancel_arch>:
       0:   8346                    mv      t1,a7
    
    0000000000000002 <__syscall_cancel_arch_start>:
       2:   00052883                lw      a7,0(a0)
       6:   0048f893                andi    a7,a7,4
       a:   00089c63                bnez    a7,22 <.L0 >
       e:   88ae                    mv      a7,a1
      10:   8532                    mv      a0,a2
      12:   85b6                    mv      a1,a3
      14:   863a                    mv      a2,a4
      16:   86be                    mv      a3,a5
      18:   8742                    mv      a4,a6
      1a:   879a                    mv      a5,t1
      1c:   00000073                ecall
    
    0000000000000020 <__syscall_cancel_arch_end>:
      20:   8082                    ret
    
      22:   1141                    addi    sp,sp,-16
      24:   e406                    sd      ra,8(sp)
      26:   00000097                auipc   ra,0x0
      2a:   000080e7                jalr    ra # 26 <.L0 >
    
    Checked with a build for riscv64-linux-gnu with run-built-tests=no.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/riscv/sigcontextinfo.h (ucontext_get_pc):
    	New fuction.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=1092ac32af74ddaa5ccfb63b8ef33b40b9befc8a

commit 1092ac32af74ddaa5ccfb63b8ef33b40b9befc8a
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Feb 3 21:07:07 2017 -0200

    nptl: hppa: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the hppa modifications required for the BZ#12683.
    It basically adds the required __pthread_get_pc function.
    
    HPPA requires an arch-specific syscall_cancel because the
    INTERNAL_SYSCALL_NCS adds some instruction to fetch the returned
    syscalls value.  The implementation were based on on default C
    version built with GCC 6.1
    
    Checked on hppa-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/hppa/sigcontextinfo.h: New file.
    	* sysdeps/unix/sysv/linux/hppa/syscall_cancel.S: Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=f74aa232cde071899f8137920e84bbba3c8ed679

commit f74aa232cde071899f8137920e84bbba3c8ed679
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:03:07 2017 -0200

    nptl: mips: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the mips modifications required for the BZ#12683.
    It adds the required ucontext_get_pc function, a mips32 cancellable
    syscall wrapper and 7 argument cancellable syscall support.
    
    To avoid code pessimization and add a requirement on all architectures
    to support {INLINE,INTERNAL)_SYSCALL with 7 argument, its support is
    added through a flag, HAVE_CANCELABLE_SYSCALL_WITH_7_ARGS, which changes
    the signature and prototype of the requires macros and functions
    (SYSCALL_CANCEL, __syscall_cancel and __syscall_cancel_arch). As
    default 6 arguments cancellable syscalls are use.
    
    MIPS o32 requires an arch-specific implementation because
    INTERNAL_SYSCALL_NCS adds an 'addiu' just after the syscall
    instruction which invalidates the checks on sigcancel_handler.
    
    Checked against a build and make check run-built-tests=no for
    mips-gnu-linux, mips64-linux-gnu, mips64-n32-linux-gnu.  I also
    ran some basic o32 and n64 cancellation tests on a simulated
    mips64 qemu system.
    
    	[BZ #12683]
    	* nptl/libc-cancellation.c (__syscall_cancel): Define and use 7
    	argument syscall if architecture requires it.
    	* nptl/pthreadP.h (__syscall_cancel_arch): Likewise.
    	* sysdeps/unix/sysdep.h (__syscall_cancel, __SYSCALL_CANCEL*): Define
    	with 7 argument if architecture requires it.
    	(__SYSCALL_CANCEL7_ARG_DEF): New macro.
    	(__SYSCALL_CANCEL7_ARG): Likewise.
    	(__SYSCALL_CANCEL7_ARG7): Likewise.
    	* sysdeps/unix/sysv/linux/syscall_cancel.c (__syscall_cancel_arch):
    	Likewise.
    	* sysdeps/mips/nptl/tls.h (READ_THREAD_POINTER): Check __mips_isa_rev
    	existance for macro definition.
    	* sysdeps/unix/sysv/linux/mips/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    	* sysdeps/unix/sysv/linux/mips/mips32/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/mips/mips32/sysdep.h
    	(HAVE_CANCELABLE_SYSCALL_WITH_7_ARGS): Define.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=7806615561d4f40f4d53f3375536cfe08fe127e0

commit 7806615561d4f40f4d53f3375536cfe08fe127e0
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:03:03 2017 -0200

    nptl: sh: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the sh modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function and a
    cancellable syscall wrapper.
    
    SH requires an arch-specific syscall_cancel because the
    INTERNAL_SYSCALL_NCS adds the required or instruction to workaround
    a hardware bug [1].  The implementation were based on on default C
    version built with GCC 6.2.1.
    
    Checked against a build and make check run-built-tests=no for
    sh4-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/sh/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/sh/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    
    [1] http://documentation.renesas.com/eng/products/mpumcu/tu/tnsh7456ae.pdf

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=02badc9d7b0c5379bd46019546ed9d2d8acb6a4a

commit 02badc9d7b0c5379bd46019546ed9d2d8acb6a4a
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:53 2017 -0200

    nptl: nios2: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the nios2 modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    The default syscall_cancel.c should be as expected for NIOS2. With
    GCC 6.2.1 syscall_cancel.c generates the following code:
    
    ---
    00000000 <__GI___syscall_cancel_arch>:
       0:   defffe04        addi    sp,sp,-8
       4:   dd800015        stw     r22,0(sp)
       8:   002ce03a        nextpc  r22
       c:   02000034        movhi   r8,0
      10:   42000004        addi    r8,r8,0
      14:   dfc00115        stw     ra,4(sp)
      18:   b22d883a        add     r22,r22,r8
    
    0000001c <__syscall_cancel_arch_start>:
      1c:   20c00017        ldw     r3,0(r4)
      20:   18c0010c        andi    r3,r3,4
      24:   18000f1e        bne     r3,zero,64 <__syscall_cancel_arch_end+0x18>
      28:   3015883a        mov     r10,r6
      2c:   2805883a        mov     r2,r5
      30:   da400517        ldw     r9,20(sp)
      34:   380b883a        mov     r5,r7
      38:   da000417        ldw     r8,16(sp)
      3c:   d9c00317        ldw     r7,12(sp)
      40:   d9800217        ldw     r6,8(sp)
      44:   5009883a        mov     r4,r10
      48:   003b683a        trap    0
    
    0000004c <__syscall_cancel_arch_end>:
      4c:   38000126        beq     r7,zero,54 <__syscall_cancel_arch_end+0x8>
      50:   0085c83a        sub     r2,zero,r2
      54:   dfc00117        ldw     ra,4(sp)
      58:   dd800017        ldw     r22,0(sp)
      5c:   dec00204        addi    sp,sp,8
      60:   f800283a        ret
      64:   b0800017        ldw     r2,0(r22)
      68:   103ee83a        callr   r2
    ---
    
    Checked against a build and make check run-built-tests=no for
    nios2-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/nios2/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a6ed5172ed96b96d6fe99bb1e83f0af8aec6bb97

commit a6ed5172ed96b96d6fe99bb1e83f0af8aec6bb97
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:36 2017 -0200

    nptl: sparc: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the sparc modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function, cancellable
    syscall wrappers, and a fix for pause.
    
    Sparc requires arch-specific syscall_cancel implementation because
    INLINE_SYSCALL_NCS uses the __SYSCALL_STRING (defined different
    for sparc32 and sparc64) and it issues additional instructions
    after the syscall one to check the resulting error code.  When used
    in the default syscall_cancel.c implementation the label
    __syscall_cancel_arch_end is not placed just after the syscall as
    expected.  Both 32 and 64 bits version were based on default C version
    built with GCC 6.1.
    
    Also, different than other architectures, SPARC passes the sigcontext_t
    struct pointer as third argument in the signal handler set with
    SA_SIGINFO (some info at [1]) for 64 bits and the pt_regs in 32 bits.
    From Linux code:
    
    * arch/sparc/kernel/signal_64.c
    
    428         /* 3. signal handler back-trampoline and parameters */
    429         regs->u_regs[UREG_FP] = ((unsigned long) sf) - STACK_BIAS;
    430         regs->u_regs[UREG_I0] = ksig->sig;
    431         regs->u_regs[UREG_I1] = (unsigned long) &sf->info;
    432
    433         /* The sigcontext is passed in this way because of how it
    434          * is defined in GLIBC's /usr/include/bits/sigcontext.h
    435          * for sparc64.  It includes the 128 bytes of siginfo_t.
    436          */
    437         regs->u_regs[UREG_I2] = (unsigned long) &sf->info;
    
    * arch/sparc/kernel/signal_32.c:
    
    392         regs->u_regs[UREG_FP] = (unsigned long) sf;
    393         regs->u_regs[UREG_I0] = ksig->sig;
    394         regs->u_regs[UREG_I1] = (unsigned long) &sf->info;
    395         regs->u_regs[UREG_I2] = (unsigned long) &sf->regs;
    396
    397         regs->pc = (unsigned long) ksig->ka.sa.sa_handler;
    398         regs->npc = (regs->pc + 4);
    
    So to access the signal mask in the signal frame, a arch-specific
    ucontext_get_mask is defined which obtain the sa_mask from the context.
    
    Checked on a SPARC T5 for sparc64-linux-gnu and sparcv9-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/sparc/sparc32/sigcontextinfo.h
    	(ucontext_get_pc): New function.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/sigcontextinfo.h
    	(ucontext_get_pc): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/sparc32/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/syscall_cancel.S: Likwise.
    	* sysdeps/unix/sysv/linux/sparc/sparc64/pause.c: New file.
    
    [1] https://www.spinics.net/lists/sparclinux/msg05037.html

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=fbdf0307b3e6b880a5c0effce603f441c1dce7ed

commit fbdf0307b3e6b880a5c0effce603f441c1dce7ed
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:17 2017 -0200

    nptl: microblaze: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the microblaze modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    Microblaze requires an arch specific assembly implementation because
    the archicture INTERNAL_SYSCALL_NCS implementation adds a nop after
    the brki instruction because the instruction expect a delay branch
    slot.  I based this implementation on generated assembly using GCC 6.1.
    
    Checked against a build and make check run-built-tests=no for
    microblaze-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/microblaze/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/microblaze/sigcontextinfo.h
    	(ucontext_get_pc): New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=838fce23efc583cb8f016914fc374319b4b85f96

commit 838fce23efc583cb8f016914fc374319b4b85f96
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:02:08 2017 -0200

    nptl: m68k: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the m68k modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function
    
    The default syscall_cancel.c should is as expected for m68k with
    GCC 6.2.1.  It generates the following code for syscall_cancel.os:
    
    ---
    00000000 <__GI___syscall_cancel_arch>:
       0:   4e56 0000       linkw %fp,#0
       4:   48e7 3c00       moveml %d2-%d5,%sp@-
    
    00000008 <__syscall_cancel_arch_start>:
       8:   206e 0008       moveal %fp@(8),%a0
       c:   2010            movel %a0@,%d0
       e:   0800 0002       btst #2,%d0
      12:   6628            bnes 3c <__syscall_cancel_arch_end+0xa>
      14:   206e 0024       moveal %fp@(36),%a0
      18:   2a2e 0020       movel %fp@(32),%d5
      1c:   282e 001c       movel %fp@(28),%d4
      20:   262e 0018       movel %fp@(24),%d3
      24:   242e 0014       movel %fp@(20),%d2
      28:   222e 0010       movel %fp@(16),%d1
      2c:   202e 000c       movel %fp@(12),%d0
      30:   4e40            trap #0
    
    00000032 <__syscall_cancel_arch_end>:
      32:   4cee 003c fff0  moveml %fp@(-16),%d2-%d5
      38:   4e5e            unlk %fp
      3a:   4e75            rts
      3c:   61ff 0000 0000  bsrl 3e <__syscall_cancel_arch_end+0xc>
    ---
    
    Checked against a build and make check run-built-tests=no for
    m68k-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/m68k/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=6da7532402198c74e7c063ade4a1ea6c6906b82a

commit 6da7532402198c74e7c063ade4a1ea6c6906b82a
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:01:58 2017 -0200

    nptl: alpha: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the alpha modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function
    
    The default syscall_cancel.c should be fine for alpha and GCC 6.1
    generates the following cod for syscall_cancel.os:
    
    ---
    0000000000000000 <__GI___syscall_cancel_arch>:
       0:   00 00 bb 27     ldah    gp,0(t12)
       4:   00 00 bd 23     lda     gp,0(gp)
       8:   f0 ff de 23     lda     sp,-16(sp)
       c:   00 04 f1 47     mov     a1,v0
      10:   00 00 5e b7     stq     ra,0(sp)
    
    0000000000000014 <__syscall_cancel_arch_start>:
      14:   00 00 30 a0     ldl     t0,0(a0)
      18:   01 00 e1 43     sextl   t0,t0
      1c:   01 90 20 44     and     t0,0x4,t0
      20:   0f 00 20 f4     bne     t0,60 <__syscall_cancel_arch_end+0x20>
      24:   10 04 f2 47     mov     a2,a0
      28:   11 04 f3 47     mov     a3,a1
      2c:   12 04 f4 47     mov     a4,a2
      30:   10 00 9e a6     ldq     a4,16(sp)
      34:   13 04 f5 47     mov     a5,a3
      38:   18 00 be a6     ldq     a5,24(sp)
      3c:   83 00 00 00     callsys
    
    0000000000000040 <__syscall_cancel_arch_end>:
      40:   21 05 e0 43     negq    v0,t0
      44:   00 00 5e a7     ldq     ra,0(sp)
      48:   c0 04 61 46     cmovne  a3,t0,v0
      4c:   10 00 de 23     lda     sp,16(sp)
      50:   01 80 fa 6b     ret
      54:   00 00 fe 2f     unop
      58:   1f 04 ff 47     nop
      5c:   00 00 fe 2f     unop
      60:   00 00 7d a7     ldq     t12,0(gp)
      64:   00 40 5b 6b     jsr     ra,(t12),68 <__syscall_cancel_arch_end+0x28>
      68:   1f 04 ff 47     nop
      6c:   00 00 fe 2f     unop
    ---
    
    Checked on alpha-linux-gnu, no regression found.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/alpha/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=60eb1c51381fda4131058dffbd46f9d512cef59f

commit 60eb1c51381fda4131058dffbd46f9d512cef59f
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Jan 16 17:01:44 2017 -0200

    nptl: ia64: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the s390 modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function. a workaround
    for mismatched sigcontext::sa_flags defiition between kernel and
    GLIBC (tracked by BZ#21634), and an arch specific syscall_cancel
    implementation.
    
    IA64 requires an arch-specific syscall_cancel implemetantion because
    {INLINE,INTERNAL}_SYSCALL is implemented by branching to a gate
    DSO (similar to i386) which renders the pointer comparison in
    SIGCANCEL handler wrong.  This incurs in performance penalty due the
    use of a break instruction insteaf of a eds one, however
    cancellable syscalls should potentially incur in syscalls blocking.
    
    Checked with a ia64-linux-gnu build with run-tests-built=no.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/ia64/sigcontextinfo.h (ucontext_get_pc,
    	ucontext_get_mask): New functions.
    	* sysdeps/unix/sysv/linux/ia64/syscall_cancel.S: New file.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=ee923bad6397f393d8ae8d38b1ea95c8df0dcf74

commit ee923bad6397f393d8ae8d38b1ea95c8df0dcf74
Author: Adhemerval Zanella <adhemerval.zanella@linaro.com>
Date:   Wed Aug 12 10:51:38 2015 -0300

    nptl: s390: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the s390 modifications required for the BZ#12683 fix.
    It basically adds the required ucontext_get_pc function.
    
    The built cancelable syscall wrapper for s390 using GCC 7.2.1 and
    default configuration flags shows the wrappers on expected corrected
    places:
    
    ---
    __GI___syscall_cancel_arch:
    .LFB39:
            .cfi_startproc
            stm     %r6,%r15,24(%r15)
            .cfi_offset 6, -72
            .cfi_offset 7, -68
            .cfi_offset 8, -64
            .cfi_offset 9, -60
            .cfi_offset 10, -56
            .cfi_offset 11, -52
            .cfi_offset 12, -48
            .cfi_offset 13, -44
            .cfi_offset 14, -40
            .cfi_offset 15, -36
            ahi     %r15,-96
            .cfi_def_cfa_offset 192
    
            .global __syscall_cancel_arch_start
    .type __syscall_cancel_arch_start,@function
    __syscall_cancel_arch_start:
            l       %r0,0(%r2)
            tml     %r0,4
            jne     .L5
            lr      %r1,%r3
            lr      %r2,%r4
            lr      %r3,%r5
            lr      %r4,%r6
            l       %r5,192(%r15)
            l       %r6,196(%r15)
            l       %r7,200(%r15)
            svc    0
    
            .global __syscall_cancel_arch_end
    .type __syscall_cancel_arch_end,@function
    __syscall_cancel_arch_end:
            l       %r4,152(%r15)
            lm      %r6,%r15,120(%r15)
            .cfi_remember_state
            .cfi_restore 15
            .cfi_restore 14
            .cfi_restore 13
            .cfi_restore 12
            .cfi_restore 11
            .cfi_restore 10
            .cfi_restore 9
            .cfi_restore 8
            .cfi_restore 7
            .cfi_restore 6
            .cfi_def_cfa_offset 96
            br      %r4
    .L5:
            .cfi_restore_state
            brasl   %r14,__syscall_do_cancel
            .cfi_endproc
    ---
    
    The s390x version also shows similar placement:
    
    ---
    __GI___syscall_cancel_arch:
            .cfi_startproc
            stmg    %r6,%r15,48(%r15)
            .cfi_offset 6, -112
            .cfi_offset 7, -104
            .cfi_offset 8, -96
            .cfi_offset 9, -88
            .cfi_offset 10, -80
            .cfi_offset 11, -72
            .cfi_offset 12, -64
            .cfi_offset 13, -56
            .cfi_offset 14, -48
            .cfi_offset 15, -40
            aghi    %r15,-160
            .cfi_def_cfa_offset 320
    
            .global __syscall_cancel_arch_start
    .type __syscall_cancel_arch_start,@function
    __syscall_cancel_arch_start:
            l       %r0,0(%r2)
            tmll    %r0,4
            jne     .L5
            lgr     %r1,%r3
            lgr     %r2,%r4
            lgr     %r3,%r5
            lgr     %r4,%r6
            lg      %r5,320(%r15)
            lg      %r6,328(%r15)
            lg      %r7,336(%r15)
            svc    0
    
            .global __syscall_cancel_arch_end
    .type __syscall_cancel_arch_end,@function
    __syscall_cancel_arch_end:
            lg      %r4,272(%r15)
            lmg     %r6,%r15,208(%r15)
            .cfi_remember_state
            .cfi_restore 15
            .cfi_restore 14
            .cfi_restore 13
            .cfi_restore 12
            .cfi_restore 11
            .cfi_restore 10
            .cfi_restore 9
            .cfi_restore 8
            .cfi_restore 7
            .cfi_restore 6
            .cfi_def_cfa_offset 160
            br      %r4
    .L5:
            .cfi_restore_state
            brasl   %r14,__syscall_do_cancel
            .cfi_endproc
    ---
    
    Checked on s390-linux-gnu and s390x-linux-gnu build with
    run-tests-built=no.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/s390/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=683a8f59a9473e0ff6d6a3df26c0294948ddc8d8

commit 683a8f59a9473e0ff6d6a3df26c0294948ddc8d8
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri May 8 17:12:31 2015 -0300

    nptl: arm: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the ARM modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function and adjust
    the generic syscall_cancel build.
    
    For ARM we need to build syscall_cancel in ARM mode (-marm) to avoid
    INTERNAL_SYSCALL to issue the syscall through the helper gate
    __libc_do_syscall (which invalidates the mark checks on SIGCANCEL
    handler).
    
    Checked on arm-linux-gnueabihf.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/arm/Makefile (CFLAGS-syscall_cancel.c): New
    	rule.
    	* sysdeps/unix/sysv/linux/arm/sigcontextinfo.h (ucontext_get_pc):
    	New function.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=0151ef6406a6e32c4601929bcbd3ea2b136f223b

commit 0151ef6406a6e32c4601929bcbd3ea2b136f223b
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Wed May 6 17:51:29 2015 -0300

    nptl: aarch64: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the aarch64 modifications required for the BZ#12683.
    It basically adds the required ucontext_get_pc function.
    
    The built cancelable syscall wrapper for aarch64 using GCC 7.2.1 and
    default configuration flags shows an expected optimized version:
    
    ---
    __GI___syscall_cancel_arch:
    .LFB38:
            .cfi_startproc
            .global __syscall_cancel_arch_start
    .type __syscall_cancel_arch_start,@function
    __syscall_cancel_arch_start:
            ldr     w9, [x0]
            tbnz    x9, 2, .L7
            mov     x8, x1
            mov     x0, x2
            mov     x1, x3
            mov     x2, x4
            mov     x3, x5
            mov     x4, x6
            mov     x5, x7
            svc     0       // syscall nr
            .global __syscall_cancel_arch_end
    .type __syscall_cancel_arch_end,@function
    __syscall_cancel_arch_end:
            ret
    ---
    
    A similar code is obtained with GCC 5.3.1, so I see to need to provide
    an arch-specific syscall_cancel.S for aarch64.
    
    Checked on aarch64-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/aarch64/sysdep-cancel.h (PSEUDO): Redefine
    	to call __syscall_cancel function for cancellable syscalls.
    	(__pthread_get_ip): Add implementation.
    	* sysdeps/unix/sysv/linux/aarch64/sysdep.h (SYSCALL_CANCEL_ERROR): Add
    	definition.
    	(SYSCALL_CANCEL_ERRNO): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=996cfcbe41d1a7561a791ab20de2dab89bde529d

commit 996cfcbe41d1a7561a791ab20de2dab89bde529d
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Sep 18 18:14:19 2015 -0300

    nptl: powerpc: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches adds the powerpc64 modification required for the BZ#12683.
    It basically adds the required __pthread_get_pc function and a arch
    specific syscall_cancel implementation.
    
    The powerpc requires an arch-specific syscall_cancel because
    INTERNAL_SYSCALL_NCS adds a mfcr just after the sc instruction to get
    the CR0.SO bit information from kernel (which signals the error
    return status).  So for cancelled syscalls with side effects,
    __pthread_get_pc will point to mcfr and thus invalidating the checks
    on sigcancel_handler.
    
    Checked on powerpc64le-linux-gnu and powerpc-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/powerpc/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    	* sysdeps/unix/sysv/linux/powerpc/syscall_cancel.S: New file.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=2258ba74573f9265fea31606d2d1352514e9deb0

commit 2258ba74573f9265fea31606d2d1352514e9deb0
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon May 4 16:30:13 2015 -0300

    nptl: i386: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patch adds the i386 modifications required for the BZ#12683.
    It basically provides the required ucontext_get_pc symbol, add the
    cancelable syscall wrapper and fix a thread atomic update macro.
    
    On i386 an arch-specific cancellation implementation is required
    because depending of the glibc configuration and underlying kernel
    the syscall may be done using a vDSO symbol (__kernel_vsyscall).
    By using the vDSO symbol the resulting PC value for an interrupted
    syscall points to an adress outside the expected markers in
    __syscall_cancel_arch.  It has been discussed in LKML [1] on how
    kernel could help userland to accomplish it, but afaik discussion
    was stalled.
    
    Also, since glibc supports i486, the old 'int 0x80' should be used
    in the syscall wrapper.  One option could make minimum default chip
    to pentium II (which implements sysenter) or add a runtime check
    on syscall_cancel.S to use 'int 0x80' or sysenter.
    
    Checked on i686-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/i386/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL,
    	THREAD_ATOMIC_BIT_SET): Remove macro.
    	* sysdeps/unix/sysv/linux/i386/Makefile
    	[$(subdir) = elf] (sysdep-rtld_routines): Add libc-do-syscall object.
    	* sysdeps/unix/sysv/linux/i386/lowlevellock.h (lll_wait_tid): Use
    	cancellable futex syscall macro.
    	* sysdeps/unix/sysv/linux/i386/syscall_cancel.S: New file.
    	* sysdeps/unix/sysv/linux/i386/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    
    [1] https://lkml.org/lkml/2016/3/8/1105

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=f43adba514ff665338e6da714551d8c3f6b158ce

commit f43adba514ff665338e6da714551d8c3f6b158ce
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Sat May 9 14:20:26 2015 -0300

    nptl: x32: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches adds the x32 modification required for the BZ#12683.
    It basically adjust the syscall size used to pass the arguments to
    the syscall cancel wrappers by zero extending pointers type while
    preserting values for default types (such as off_t).
    
    Checked on x86_64-linux-gnu-x32.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/x86_64/x32/sysdep.h (__syscall_arg_t):
    	Define type for x32.
    	(__SSC): Add platform specific macro.
    	* include/libc-pointer-arith.h (__integer_if_pointer_type_sub,
    	__integer_if_pointer_type, cast_to_integer): Parametrize integer type
    	cast.
    	(cast_to_uinteger): New macro.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=299d22ce112252b2a950e4395344dcf3791d73a9

commit 299d22ce112252b2a950e4395344dcf3791d73a9
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Sep 29 09:48:34 2014 -0300

    nptl: x86_64: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches adds the x86_64 modification required for the BZ#12683.
    It basically provide the required ucontext_get_pc symbol and remove
    the arch-specific libc-cancellation implementations.
    
    Checked on x86_64-linux-gnu.
    
    	[BZ #12683]
    	* sysdeps/unix/sysv/linux/x86_64/cancellation.S: Remove file.
    	* sysdeps/unix/sysv/linux/x86_64/libc-cancellation.S: Remove file.
    	* sysdeps/unix/sysv/linux/x86_64/librt-cancellation.S: Remove file.
    	* sysdeps/unix/sysv/linux/x86_64/sigcontextinfo.h (ucontext_get_pc):
    	New function.
    	* sysdeps/x86_64/nptl/tcb-offsets.sym (TCB_CANCELING_BITMASK):
    	Remove.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=6fe99352106cf8f244418f3708b3d5928e82e831

commit 6fe99352106cf8f244418f3708b3d5928e82e831
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Fri Sep 18 18:26:35 2015 -0300

    nptl: Fix Race conditions in pthread cancellation (BZ#12683)
    
    This patches fixes some race conditions in NPTL cancellation code by
    redefining how cancellable syscalls are defined and handled.  Current
    approach is to enable asynchronous cancellation prior to making the syscall
    and restore the previous cancellation type once the syscall returns.
    
    As decribed in BZ#12683, this approach shows 2 important problems:
    
      1. Cancellation can act after the syscall has returned from kernel, but
         before userspace saves the return value.  It might result in a resource
         leak if the syscall allocated a resource or a side effect (partial
         read/write), and there is no way to program handle it with cancellation
         handlers.
    
      2. If a signal is handled while the thread is blocked at a cancellable
         syscall, the entire signal handler runs with asynchronous cancellation
         enabled.  This can lead to issues if the signal handler call functions
         which are async-signal-safe but not async-cancel-safe.
    
    For cancellation to work correctly, there are 5 points at which the
    cancellation signal could arrive:
    
      1. Before the final "testcancel" and before the syscall is made.
      2. Between the "testcancel" and the syscall.
      3. While the syscall is blocked and no side effects have yet taken place.
      4. While the syscall is blocked but with some side effects already having
         taken place (e.g. a partial read or write).
      5. After the syscall has returned.
    
    And GLIBC wants to act on cancellation in cases 1, 2, and 3 but not in case
    4 or 5.  The proposed solution follows:
    
      * Handling case 1 is trivial: do a conditional branch based on whether the
        thread has received a cancellation request;
      * Case 2 can be caught by the signal handler determining that the saved
        program counter (from the ucontext_t) is in some address range beginning
        just before the "testcancel" and ending with the syscall instruction.
      * In this case, except for certain syscalls that ALWAYS fail with EINTR
        even for non-interrupting signals, the kernel will reset the program
        counter to point at the syscall instruction during signal handling, so
        that the syscall is restarted when the signal handler returns. So, from
        the signal handler's standpoint, this looks the same as case 2, and thus
        it's taken care of.
      * In this case, the kernel cannot restart the syscall; when it's
        interrupted by a signal, the kernel must cause the syscall to return
        with whatever partial result it obtained (e.g. partial read or write).
      * In this case, the saved program counter points just after the syscall
        instruction, so the signal handler won't act on cancellation.
        This one is equal to 4. since the program counter is past the syscall
        instruction already.
    
    Another case that needs handling is syscalls that fail with EINTR even
    when the signal handler is non-interrupting. In this case, the syscall
    wrapper code can just check the cancellation flag when the errno result
    is EINTR, and act on cancellation if it's set.
    
    The proposed GLIBC adjustments are:
    
      1. Remove the enable_asynccancel/disable_asynccancel function usage in
         syscall definition and instead make them call a common symbol that will
         check if cancellation is enabled (__syscall_cancel at
         nptl/libc-cancellation.c), call the arch-specific cancellable
         entry-point (__syscall_cancel_arch) and cancel the thread when required.
    
      2. Provide a arch-specific symbol that contains global markers. These
         markers will be used in SIGCANCEL handler to check if the interruption
         has been called in a valid syscall and if the syscalls has been
         completed or not.
         A default version is provided (sysdeps/unix/sysv/linux/syscall_cancel.c),
         however the markers may not be set on correct expected places depeding
         of how INTERNAL_SYSCALL_NCS is implemented by the underlying architecture.
         In this case arch-specific implementation should be provided.
    
      3. Rewrite SIGCANCEL asynchronous handler to check for both cancelling type
         and if current IP from signal handler falls between the global markes
         and act accordingly (sigcancel_handler at nptl/nptl-init.c).
    
      4. Adjust nptl/pthread_cancel.c to send an signal instead of acting
         directly. This avoid synchronization issues when updating the
         cancellation status and also focus the logic on signal handler and
         cancellation syscall code.
    
      5. Adjust pthread code to replace CANCEL_ASYNC/CANCEL_RESET calls to
         appropriated cancelable futex syscalls.
    
      6. Adjust libc code to replace LIBC_CANCEL_ASYNC/LIBC_CANCEL_RESET to
         appropriated cancelable syscalls.
    
      7. Adjust 'lowlevellock-futex.h' arch-specific implementations to provide
         cancelable futex calls (used in libpthread code).
    
    This patch adds the proposed changes to NPTL.  The code leaves all the ports
    broken without further patches in the list.
    
    	[BZ #12683]
    	* manual/llio.texi: Adjust comments regarding
    	pthread_enable_asynccancel and pthread_disable_asynccancel.
    	* nptl/Makefile (routines): Add syscall_cancel object.
    	(libpthread-routines): Add pthread_kill_internal and remove
    	cancellation object.
    	(CFLAGS-cancellation.c): Remove rule.
    	(CFLAGS-syscall_cancel.c): New rule.
    	(tests): Add tst-cancel28.
    	* nptl/Versions [GLIBC_PRIVATE] (libc): Add __syscall_cancel,
    	__syscall_cancel_arch_start, and __syscall_cancel_arch_end.
    	* nptl/cancellation.c: Remove file.
    	* sysdeps/nptl/librt-cancellation.c: Likewise.
    	* nptl/descr.h (CANCELING_BIT, CANCELING_BITMASK): Remove define.
    	(CANCELED_BIT, EXITING_BIT, TERMINATED_BIT, SETXID_BIT,
    	CANCEL_RESTMASK): Adjust value with CANCELED_BIT removal.
    	* nptl/libc-cancellation.c (__syscall_cancel): New symbol: symbol
    	brigde for non-cancellable syscalls.
    	(__syscall_do_cancel): New symbol.
    	* nptl/nptl-init.c (sigcancel_handler): Rewrite function to avoid race
    	conditions.
    	(__pthread_initialize_minimal_internal): Add SA_RESTART to SIGCANCEL
    	handler.
    	(UCONTEXT_SIGMASK): New macro.
    	* nptl/pthreadP.h (__do_cancel): Rewrite to both disable asynchronous
    	cancellation and setting the thread as cancelled.
    	(__do_cancel_with_result): New macro.
    	(CANCEL_ASYNC, CANCEL_RESET, LIBC_CANCEL_ASYNC, LIBC_CANCEL_RESET,
    	LIBC_CANCEL_HANDLED): Remove macros.
    	(__syscall_cancel_arch, __syscall_do_cancel, __pthread_kill_internal):
    	New prototypes.
    	(__pthread_enable_asynccancel, __pthread_disable_asynccancel,
    	__libc_enable_asynccancel, __libc_disable_asynccancel,
    	__librt_enable_asynccancel, __librt_disable_asynccancel): Remove
    	prototypes.
    	* nptl/pthread_cancel.c (pthread_cancel): Rewrite to just set
    	CANCELLED_BIT and call __pthread_kill.
    	* nptl/pthread_create.c (START_THREAD_DEFN): Likewise.
    	* nptl/pthread_exit.c (__pthread_exit): Call __do_cancel_with_result.
    	* nptl/pthread_join_common.c (__pthread_timedjoin_ex): Likewise.
    	* nptl/pthread_kill.c (__pthread_kill): Check internal signals with
    	__is_internal_signal, tail call __pthread_kill_internal, and remove
    	stub_warning.
    	* nptl/pthread_kill_internal.c: New file.
    	* nptl/tst-cancel28.c: Likewise.
    	* rt/Makefile [CFLAGS-librt-cancellation.c]: Remove rule.
    	* support/temp_file.c (support_create_temp_fifo): New function.
    	* support/temp_file.h (support_create_temp_fifo): New prototype.
    	* sysdeps/generic/sigcontextinfo.h (ucontext_get_pc): Likewise.
    	* sysdeps/generic/sysdep-cancel.h (LIBC_CANCEL_ASYNC,
    	LIBC_CANCEL_RESET): Remove define.
    	* sysdeps/htl/pthreadP.h (__pthread_kill_internal): New prototype.
    	* sysdeps/nptl/Makefile [$(subdir) = rt] (librt-sysdep_routines):
    	Remove librt-cancellation object.
    	[$(subdir) = rt] (CFLAGS-librt-cancellation.c): Remove rule.
    	* nptl/lll_timedwait_tid.c (__lll_timedwait_tid): Use cancellable
    	futex operation.
    	* sysdeps/unix/sysdep.h (SYSCALL_CANCEL): Rewrite to call __syscall_cancel.
    	(INTERNAL_SYSCALL_NCS_CALL, __INTERNAL_SYSCALL_NCS*, __SYSCALL_CANCEL*):
    	New macros.
    	* sysdeps/unix/sysv/linux/futex-internal.h (fuex_wait_cancel,
    	futex_reltimed_wait_cancelable, futex_abstimed_wait_cancelable): Use
    	cancelable futex wrapper.
    	* sysdeps/unix/sysv/linux/lowlevellock-futex.h (lll_futex_syscall_cp,
    	lll_futex_wait_cancel, lll_futex_timed_wait_cancel,
    	lll_futex_timed_wait_bitset_cancel): New macros.
    	* sysdeps/unix/sysv/linux/pthread_kill_internal.c: New file.
    	* sysdeps/unix/sysv/linux/socketcall.h (SOCKETCALL): Use __SSC macros.
    	(SOCKETCALL_CANCEL): Use SYSCALL_CANCEL macros.
    	(__SOCKETCALL_CANCEL*): New macros.
    	* sysdeps/unix/sysv/linux/syscall_cancel.c: New file.
    	* sysdeps/unix/sysv/linux/sysdep.h (SYSCALL_CANCEL_RET): New macro.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=590b6e0aa785db2164870707244d4ae16a711277

commit 590b6e0aa785db2164870707244d4ae16a711277
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Thu May 10 17:24:56 2018 -0300

    nptl: Cleanup cancellation macros
    
    This patch wraps all uses of *_{enable,disable}_asynccancel and
    and *_CANCEL_{ASYNC,RESET} in either already provided macros
    (lll_futex_timed_wait_cancel) or creates new ones if the
    functionality is not provided (SYSCALL_CANCEL_NCS, lll_futex_wait_cancel,
    and lll_futex_timed_wait_cancel).
    
    Also for some generic implementations, the direct call of the macros
    are removed since the underlying symbols are suppose to provide
    cancellation support.
    
    This is a priliminary patch intended to simplify the work required
    for BZ#12683 fix.  It is a refactor change, no semantic changes are
    expected.
    
    Checked on x86_64-linux-gnu and i686-linux-gnu.
    
    	* nptl/pthread_join_common.c (__pthread_timedjoin_ex): Use
    	lll_wait_tid with timeout.
    	* nptl/sem_wait.c (__old_sem_wait): Use lll_futex_wait_cancel.
    	* sysdeps/nptl/aio_misc.h (AIO_MISC_WAIT): Use
    	futex_reltimed_wait_cancelable for cancelabla mode.
    	* sysdeps/nptl/gai_misc.h (GAI_MISC_WAIT): Likewise.
    	* sysdeps/posix/open64.c (__libc_open64): Do not call cancelation
    	macros.
    	* sysdeps/posix/sigwait.c (__sigwait): Likewise.
    	* sysdeps/posix/waitid.c (__sigwait): Likewise.
    	* sysdeps/unix/sysdep.h (__SYSCALL_CANCEL_CALL,
    	SYSCALL_CANCEL_NCS): New macro.
    	* sysdeps/nptl/lowlevellock.h (lll_wait_tid): Add timeout argument.
    	(lll_timedwait_tid): Remove macro.
    	* sysdeps/unix/sysv/linux/i386/lowlevellock.h (lll_wait_tid):
    	Likewise.
    	(lll_timedwait_tid): Likewise.
    	* sysdeps/unix/sysv/linux/sparc/lowlevellock.h (lll_wait_tid):
    	Likewise.
    	(lll_timedwait_tid): Likewise.
    	* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (lll_wait_tid):
    	Likewise.
    	(lll_timedwait_tid): Likewise.
    	* sysdeps/unix/sysv/linux/clock_nanosleep.c (__clock_nanosleep):
    	Use SYSCALL_CANCEL_NCS.
    	* sysdeps/unix/sysv/linux/futex-internal.h
    	(futex_reltimed_wait_cancelable): Use LIBC_CANCEL_{ASYNC,RESET}
    	instead of __pthread_{enable,disable}_asynccancel.
    	* sysdeps/unix/sysv/linux/lowlevellock-futex.h
    	(lll_futex_wait_cancel): New macro.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=323ab50e88231adaf93d3bdf42b15612f2795ce9

commit 323ab50e88231adaf93d3bdf42b15612f2795ce9
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Wed May 9 10:39:49 2018 -0300

    i386: Remove bogus THREAD_ATOMIC_* macros
    
    The x86 defines optimized THREAD_ATOMIC_* macros where reference always
    the current thread instead of the one indicated by input 'descr' argument.
    It work as long the input is the self thread pointer, however it generates
    wrong code is the semantic is to set a bit atomicialy from another thread.
    
    This is not an issue for current GLIBC usage, however the new cancellation
    code expects that some synchronization code to atomically set bits from
    different threads.
    
    If some usage indeed proves to be a hotspot we can add an extra macro
    with a more descriptive name (THREAD_ATOMIC_BIT_SET_SELF for instance)
    where i386 might optimize it.
    
    Checked on i686-linux-gnu.
    
    	* sysdeps/i686/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL,
    	THREAD_ATOMIC_AND, THREAD_ATOMIC_BIT_SET): Remove macros.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=b6838cd772c01dca2cf49217d4af8bcfdbe8eeb8

commit b6838cd772c01dca2cf49217d4af8bcfdbe8eeb8
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Wed May 9 10:32:25 2018 -0300

    x86: Remove wrong THREAD_ATOMIC_* macros
    
    The x86 defines optimized THREAD_ATOMIC_* macros where reference always
    the current thread instead of the one indicated by input 'descr' argument.
    It work as long the input is the self thread pointer, however it generates
    wrong code is the semantic is to set a bit atomicialy from another thread.
    
    This is not an issue for current GLIBC usage, however the new cancellation
    code expects that some synchronization code to atomically set bits from
    different threads.
    
    The generic code generates an additional load to reference to TLS segment,
    for instance the code:
    
      THREAD_ATOMIC_BIT_SET (THREAD_SELF, cancelhandling, CANCELED_BIT);
    
    Compiles to:
    
      lock;orl $4, %fs:776
    
    Where with patch changes it now compiles to:
    
      mov %fs:16,%rax
      lock;orl $4, 776(%rax)
    
    If some usage indeed proves to be a hotspot we can add an extra macro
    with a more descriptive name (THREAD_ATOMIC_BIT_SET_SELF for instance)
    where x86_64 might optimize it.
    
    Checked on x86_64-linux-gnu.
    
    	* sysdeps/x86_64/nptl/tls.h (THREAD_ATOMIC_CMPXCHG_VAL,
    	THREAD_ATOMIC_AND, THREAD_ATOMIC_BIT_SET): Remove macros.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=42794ba6346307769c5836e51c0b85b988b00732

commit 42794ba6346307769c5836e51c0b85b988b00732
Author: Adhemerval Zanella <adhemerval.zanella@linaro.com>
Date:   Mon Sep 21 15:55:58 2015 -0700

    nptl: Fix testcases for new pthread cancellation mechanism
    
    With upcoming fix for BZ#12683, pthread cancellation does not act for:
    
      1. If syscall is blocked but with some side effects already having
         taken place (e.g. a partial read or write).
      2. After the syscall has returned.
    
    The main change is due the fact programs need to act in syscalls with
    side-effects (for instance, to avoid leak of allocated resources or
    handle partial read/write).
    
    This patch changes the NPTL testcase that assumes the old behavior and
    also remove the tst-cancel-wrappers.sh test (which checks for symbols
    that will not exist anymore).
    
    It also changes how to call the read syscall on tst-backtrace{5,6}
    to use syscall instead of read cancelable syscall to avoid need to
    handle the cancelable bridge function calls.
    
    Checked on i686-linux-gnu, x86_64-linux-gnu, x86_64-linux-gnux32,
    aarch64-linux-gnu, arm-linux-gnueabihf, powerpc64le-linux-gnu,
    powerpc-linux-gnu, sparcv9-linux-gnu, and sparc64-linux-gnu.
    
    	* debug/tst-backtrace5.c (handle_signal): Check for syscall
    	instead of read.
    	(fn): Issue the read syscall instead of call the cancellable
    	syscall.
    	* nptl/tst-cancel4.c (tf_write): Handle cancelled syscall with
    	side-effects.
    	(tf_send): Likewise.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=00278aa8a3fffa4cb1d9a7b882b84e512aa09ae9

commit 00278aa8a3fffa4cb1d9a7b882b84e512aa09ae9
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Tue May 8 16:28:18 2018 -0300

    nptl: Remove tst-cancel-wrappers test and related macros
    
    With upcoming BZ#12683 fix, syscall cancellation is not more handled
    by {libc,pthread,librt}_{enable,disable}_asynccancel symbols.  This renders
    both LIBC_CANCEL_HANDLED and empty declaration and tst-cancel-wrappers.sh
    unrequired.  This patch removes both the macro and the nptl test.
    
    Checked on x86_64-linux-gnu.
    
    	* io/creat.c (LIBC_CANCEL_HANDLED): Remove macro.
    	* io/ppoll.c (LIBC_CANCEL_HANDLED): Likewise.
    	* misc/pselect.c (LIBC_CANCEL_HANDLED): Likewise.
    	* nptl/pthreadP.h (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/generic/sysdep-cancel.h (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/posix/pause.c (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/posix/sigpause.c (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/creat.c (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/creat64.c (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/sigwait.c (LIBC_CANCEL_HANDLED): Likewise.
    	* sysdeps/unix/sysv/linux/sigwaitinfo.c (LIBC_CANCEL_HANDLED):
    	Likewise.
    	* nptl/Makefile [$(run-built-tests) = yes] (tests-special): Remove
    	tst-cancel-wrappers.sh.
    	(generated): Remove tst-cancel-wrappers.out.
    	(tst-cancel-wrappers.out): Remove rule.
    	* nptl/tst-cancel-wrappers.sh: Remove file.

https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=d6ed8a26a2b63f28721c4adfca00de8da83238ff

commit d6ed8a26a2b63f28721c4adfca00de8da83238ff
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date:   Mon Feb 26 17:52:02 2018 -0300

    powerpc: Add CFI information on indirect syscall
    
    This patch the required CFI information on powerpc indirect syscall
    so backtrace works correctly on signal handler.
    
    Checked on powerpc-linux-gnu and powerpc64le-linux-gnu.
    
    	* sysdeps/unix/sysv/linux/powerpc/syscall.S (syscall): Add CFI
    	information.

-----------------------------------------------------------------------