Re: Thread-, Signal- and Cancellation-safety documentation

[Torvald Riegel <triegel at redhat dot com>]

On Sun, 2013-06-02 at 10:31 -0400, Rich Felker wrote:
> On Sun, Jun 02, 2013 at 02:06:36AM -0300, Alexandre Oliva wrote:
> > On Jun  1, 2013, Alexandre Oliva <aoliva@redhat.com> wrote:
> > 
> > > What must not happen is for mem synch primitives, called by users, to
> > > fail to propagate effects of posix API calls executed locally to be
> > > published to global memory, and make effects of API calls executed in
> > > other threads before their latest mem synch primitives visible to the
> > > local thread.  Other than that, all bets are off.
> > 
> > Now, bringing that back to the topic that matters to the project I'm on,
> > and trying to show that the guarantees you're looking for are not
> > given by posix, consider this:
> > 
> > char a, b; // globals
> > int pipefd[2]; // the two ends of a pipe internal to this process
> > 
> > thread1 {
> >   a = 42;
> >   b = 6 * 9;
> >   write (pipefd[1], &b, 1);
> > }
> > 
> > thread2 {
> >   read (pipefd[0], &a, 1);
> >   printf ("%i\n", a);
> > }
> > 
> > 
> > Now, obviously there's a happens-before relationship between the write
> > and the read, right?  However, since neither read nor write are memory
> > synchronization operations, nothing in posix guarantees that the write
> > to a in thread1 won't prevail over the write to a implied by the read
> > call in thread2: in the absence of a memory sync operation, it is a data
> > race, that invokes undefined behavior, even when there's an obvious
> > happens-before.
> 
> There is no obvious happens-before in your code. The _compiler_ is
> free to reorder the store to a with respect to the write call assuming
> it's aware that write is a standard POSIX function that could not
> possibly examine a.

That's not quite true because it's not related to examining a.  The
compiler can reorder if it can assume that write will not have any
synchronization with release or stronger memory orders.

> You'd have to work harder to make a true
> happens-before, but I'm satisfied that it's possible to do so even
> without a good example.
> 
> > Now, both read and write are thread safe, so it follows that being
> > thread safe doesn't offer ordering guarantees, not even when a
> > happens-before is present!
> > 
> > How could this be possible?  Well, precisely because the internal
> > implementation of the read and write calls could use internal mechanisms
> > that, in spite of flushing the data in the write buffer all the way to
> > the read buffer in another thread, does not guarantee that any other
> > data is flushed.
> > 
> > Now, using streams rather than file descriptors would not make any
> > difference as far as ordering guarantees are concerned: in spite of the
> > obvious happens-before, no memory synchronization is guaranteed.
> > Indeed, nothing in posix precludes the FILE* to be fake pointers used
> > just as identifiers, or pointers to memory accessible only to the
> > kernel, as long as the stream manipulating interfaces behave as
> > specified; they could all be atomic system calls, or they could use any
> > form of magic (or sufficiently advanced technology ;-) to implement
> > behavior that meets the specification while ensuring consistency of the
> > internal data structures even in the presence of concurrent calls.
> > 
> > Even the availability of explicit stream locking (flockfile) does not
> > bring memory synchronization with it, so in spite of mutual exclusion,
> > ordering is not guaranteed.  I don't see any requirement that would
> > render non-compliant an implementation of stream write operations that,
> > given multiple writes within a flockfile/funlockfile pair, queued them
> > up in memory local to the thread, getting them all ordered and written
> > out at subsequent explicit mem sync calls, or implicitly.
> > 
> > Any evidence that this is not so?
> 
> I agree with your analysis. I'm not sure whether this is intentional
> or an oversight. I suspect many historical applications perform
> synchronization of shared memory maps with pipes or other
> filesystem-based methods, and I also suspect that all historical
> implementations have full memory barriers, not by choice but because
> they're inevitable, in the kernelspace part of any function capable of
> performing inter-process communication.

You wouldn't even need a full memory barrier somewhere in the
implementation.  Even just using a normal lock to synchronize access to
the buffer backing the pipe (so, all intra-process sync) would give you
the happens-before orders that lead to consistency with the ordering
behavior implied by the functions' results.

> So it may be preferable to
> amend the standard to make such applications conforming,

Right.  If a likely implementation would give the stronger guarantees
anyway, it's not beneficial to require programmers to enforce the
stronger guarantees a second time.  And a lock-based implementation
isn't really that uncommon...

> though I'm
> not even sure _how_ that could be done without imposing overly-strict
> synchronization requirements or very complex language...

You would need a base memory model, but I believe we need that anyway.
With that in place, we could say something like: If read() reads any
value from the pipe produces by another call to write(), then the read
synchronizes with the write.  (synchronizes with as being defined in
C11/C++11 -- this does not establish a total order).


Torvald