Re: Thread-, Signal- and Cancellation-safety documentation

[Rich Felker <dalias at aerifal dot cx>]

On Tue, Mar 26, 2013 at 01:49:20AM -0300, Alexandre Oliva wrote:
> I've (not so) recently started working on documenting these safety
> properties of glibc functions.  I believe now have enough of a patchset
> to get the discussion on form and content started.

Wow, nice project.

> One of the stumbling blocks I ran into was that setlocale (unlike
> uselocale) may change the global locale object and, even though the
> modification of this object is guarded by a rwlock, most uses are not.

I know your work is just documenting existing practice, but unless
there's a clause in POSIX (and now, C11 too, since C11 has threads)
which allows this, I think the underlying issue should be fixed. The
cleanest fix I can envision would be atomically replacing the pointer
to the locale object once the new one is prepared. This would
eliminate the need for readers to lock anything. If there's a separate
pointer for each locale category, readers that depend on more than one
category would still need to use a lock to ensure that the entire
update is seen atomically (but most readers only use one category).

> As a result, many functions were marked as unsafe, with âglocaleâ as the
> reason.  More recently, talking to Carlos about the consequences of this
> and debating how best to approach the need for some functions to be
> MT-Safe in spite of setlocale, we decided to document setlocale itself
> as unsafe, so that calling it after starting threads was documented as
> problematic.  Then, we could document the fast-path functions as MT-Safe
> even though they don't bother with locking.

Per POSIX, setlocale is thread-safe.

> A similar âsolutionâ was used for fsetlocking: since it may disable
> internal locking, it would render several functions MT-Unsafe (and also
> cause them to release locks they didn't take, or take locks and never
> release them).  By declaring fsetlocking itself MT-Unsafe, we avoid
> these problems (since then calling it after starting threads invokes
> undefined behavior), without making every function that optionally takes
> care of internal locking MT-Unsafe.  It would be possible, however, to
> make fsetlocking MT-Safer :-) by changing it hold the stream's recursive
> lock regardless of internal-locking status (to make sure all threads
> that are using the stream complete the operation underway, be it guarded
> by internal locking, be it explicit external locking--use without either
> is undefined).  This wouldn't make it AS-Safe, which would require all
> functions that optionally take the lock to load the status only once
> into an automatic variable, and use it to decide whether to take and
> release the lock, or neither, rather than re-reading the setting that
> may have been modified in a signal handler.  This is fixable but
> cumbersome and possibly not worth it, so I didn't implement this
> change.

I'm a bit confused here. With or without fsetlocking having been used,
all of stdio is thread-safe as long as the same FILE is not acted upon
by more than one thread at a time. fsetlocking pertains to whether the
_data object_ (the FILE) is safe to operate upon from multiple
threads, not whether the stdio functions are thread-safe. Do you have
a way to represent this distinction?

> Another issue was how to enable threads and signal handlers to safely
> call fclose, given that it releases the memory area that contains the
> lock object that other threads might need for synchronization.

This is called "self-synchronized destruction" and the same issue
applies to mutexes, semaphores, barriers, etc. -- the act of obtaining
the lock when you can prove that you're the last to obtain it entitles
you to destroy it and free the memory. Unfortunately glibc has a
number of bugs in this area.

For fclose, self-synchronized destruction is really only possible in
two cases:

1. You can prove that, prior to the call to fclose, all other threads
that may operate on the FILE have finished doing so (and will not
attempt to do so again), OR

2. At the time of fclose, one other thread presently holds the lock on
the FILE, and once that lock is released, it (or any other thread)
will not attempt to operate on the FILE again. Normally this is only
provable under the use of flockfile, but it's also provable in a few
cases with pipes, etc. where the blocking semantics allow you to prove
that a thread holds the file lock. See http://ewontfix.com/8/

> therefore well-defined and safe.  In the end, fclose turned out to be
> AS-Unsafe for other reasons (a non-recursive gconv mutex that could
> self-deadlock), but the rationale, if agreed by the community as sound,
> may be reused in other settings.

Doesn't the freeing make it unsafe? Or does glibc use mmap/munmap for
FILEs?

> As for AC safety, a number of functions take locks in ways that suffice
> to get the locks released upon synchronous cancellation.  Unfortunately,
> that's not enough to ensure AC-safety of the locking: it is possible for
> the cancellation to take place before the lock is taken, and then the
> cleanup would attempt to released the lock anyway.  I referred to this
> problem in the safety notes as lockleak.  Other leaks, such as memory

It's basically impossible to be AC safe without simply disabling async
cancellation before doing your work then reenabling it afterwards.

> and file descriptor leaks, were not regarded as safety issues, but they
> were noted as well.

Note that glibc has a serious file descriptor leak issue even with
synchronous cancellation:

http://sourceware.org/bugzilla/show_bug.cgi?id=12683

Rich