Re: proposal: Thread Properties API

[Rich Felker <dalias at libc dot org>]

On Fri, Apr 25, 2014 at 02:58:39PM +0400, Konstantin Serebryany wrote:
> On Thu, Apr 24, 2014 at 6:45 PM, Torvald Riegel <triegel@redhat.com> wrote:
> > On Thu, 2014-04-24 at 13:38 +0400, Konstantin Serebryany wrote:
> >> ================== Callback for thread exit  ==================
> >> Tools such as ASAN, MSAN and LSAN need to perform various cleanup
> >> actions just before a thread exits.
> >>
> >> Currently these tools use an ugly hack to get notified about thread's
> >> destruction: call pthread_setspecific recursively
> >> PTHREAD_DESTRUCTOR_ITERATIONS times; when the
> >> PTHREAD_DESTRUCTOR_ITERATIONS-th call is made we consider the thread
> >> as dead. A cleaner interface would be much appreciated (not sure if it
> >> easy to do or at all possible)
> >
> > GCC's libitm uses a similar hack, but checks whether transactions have
> > been used since the last round of pthreads TLS destruction.
> >
> >> // Register callback to be called right before the thread is totally destroyed.
> >> // The callbacks are chained, they are called in the order opposite to
> >> the order they were registered.
> >
> > I think we need a more detailed contract here.  What does "totally
> > destroyed" mean?
> 
> That's indeed tricky to define.
> There is certainly a point of time when thread is still alive (e.g.
> the 4-th tsd destructor),
> and then there is a another point when it is dead-dead-dead (e.g. when
> it's stack is unmaped or reused by another thread).
> Suggestions?

My definition of "totally destroyed" is a point at which the existence
of the thread is no longer "observable". Of course then the latter
needs a definition; my intent is that "observable" would definitely
count any of the standard interfaces, and definitely would not count
things like external debuggers. But the interfaces being proposed are
themselves problematic in that they provide a way to observe the
existence or nonexistence of a thread. Any time you have two or more
clients to these interfaces, at least one of them is going to see a
wrong result: the thread cannot be "totally destroyed" yet when one
client gets the notification, because another client will later
observe it as not-yet-destroyed. That's why the whole concept of these
proposed interfaces bothers me...

> > What's the requirements on the destruction code
> > running, especially regarding which other components can be used during
> > destruction.
> 
> No libc calls in the destruction code.

That's a fairly poorly-specified requirement. Does this mean you have
to write the code in asm, since the compiler might generate calls to
memset, etc.? In any case how do you intend to get the data anywhere
useful without libc calls, since you can't do any memory
synchronization? Are __sync builtins or C11 atomics a requirement for
using these interfaces?

It's also insufficient. For example I think you also want to require
that TLS isn't accessed, right? And stack protector will access TLS
implicitly (although its access is probably harmless).

> >  For example, how would ASAN/MSAN/LSAN interact with libitm
> > if it would use this feature?
> Never thought about libitm<=>*san interaction. Do you expect any problem there?

I expect problems anywhere the above contract of "totally destroyed"
is violated, which would happen whenever there's more than one client
interacting with these interfaces, as described above.

> > I would guess that we'd need some combination of "stick to limited set
> > of features during destruction" and possibly a multi-round destruction
> > to get the dependencies sorted (otherwise, we'd need the registration
> > order to perfectly match the logical dependencies during destruction).
> >
> >> // The callbacks must be registered only before any threads were
> >> created, at most 8 callbacks can be registered.
> >
> > Why did you include the former constraint?  Wouldn't this prevent some
> > uses of this feature?
> > Why at most 8 callbacks?  Arbitrary choice?
> 
> 8 is as good a number as 10 or 16. It has to be small constant to
> simplify implementation.

1 is probably the only safe number, but I don't think that will make
anyone happy...

> >> // No signals may arrive during the calls to these callbacks;
> >> immediately after the last of these calls the thread is dead.
> >> void __libc_register_thread_exit_callback(void (*cb)());
> >
> > What happens to those signals?  Or is it a requirement on the
> > program/... that they shouldn't get any?
> 
> Can't we simply ignore all signals in the thread after some point
> (like pthread_sigmask)?

Signals must be blocked before a thread can be considered "totally
destroyed". Otherwise there is a trivial way its existence can be
observed by a fully conforming program. This must be done by the
implementation before any of the callbacks are called. It probably
needs to be done anyway, even without callbacks; I suspect there's a
race condition where dynamic TLS might be accessed "too late" in the
thread termination if a signal arrives late.

Rich