In ptrace requests that can also deliver a signal to the affected process, the
request itself can be ignored if the signal delivery fails. E.g., in
kerne/ptrace.c, the code for PTRACE_DETACH is:
ret = ptrace_induce_signal(child, engine, data);
ret = ptrace_detach(child, engine);
Similar code exists in other places such as that for PTRACE_CONT.
This code is in linux-2.6.17 with the utrace patches applied.
The behaviour can be exercised by, e.g., calling ptrace(PTRACE_DETACH, pid,
NULL, sig) where sig either is an invalid signal, or some internal conditions in
ptrace.c aren't met.
Created attachment 1316 [details]
Testcase demonstrating the failure.
This t/c does two ptrace(PTRACE_DETACH...) operations. The first tries to
deliver an invalid signal--999--thereby failing (rc == -1) and inhibiting the
intended detach. The second delivers a valid SIGKILL and succeeds (rc == 0).
vanilla kernel's ptrace_detach also fails for invalid signal
what is the errno in the real failure case?
This is really "NOTABUG" but there's no option under "Resolve bug" for that.