The unreliability of AF_UNIX datagram sockets

[Corinna Vinschen]

On Apr 29 13:05, Corinna Vinschen wrote:
> On Apr 27 11:47, Ken Brown wrote:
> > This is a follow-up to
> > 
> >  https://cygwin.com/pipermail/cygwin/2021-April/248383.html
> > 
> > I'm attaching a test case slightly simpler than the one posted by the OP in
> > that thread.  This is a client/server scenario, with non-blocking AF_UNIX
> > datagram sockets.  The client writes COUNT messages while the server is
> > playing with his toes.  Then the server reads the messages.
> > 
> > If COUNT is too big, the expectation is that the client's sendto call will
> > eventually return EAGAIN.  This is what happens on Linux.  On Cygwin,
> > however, there is never a sendto error; the program ends when recv fails
> > with EAGAIN, indicating that some messages were dropped.
> > 
> > I think what's happening is that WSASendTo is silently dropping messages
> > without returning an error.  I guess this is acceptable because of the
> > documented unreliability of AF_INET datagram sockets.  But AF_UNIX datagram
> > sockets are supposed to be reliable.
> > 
> > I can't think of anything that Cygwin can do about this (but I would love to
> > be proven wrong).  My real reason for raising the issue is that, as we
> > recently discussed in a different thread, maybe it's time for Cygwin to
> > start using native Windows AF_UNIX sockets.  But then we would still have to
> > come up with our own implementation of AF_UNIX datagram sockets, and it
> > seems that we can't simply use the current implementation.  AFAICT, Mark's
> > suggestion of using message queues is the best idea so far.
> > 
> > I'm willing to start working on the switch to native AF_UNIX sockets.  (I'm
> > frankly getting bored with working on the pipe implementation, and this
>           ^^^^^^^^^^^^^
> I not really surprised, Windows pipe semantics are annoying.
> 
> > doesn't really seem like it has much of a future.)  But I'd like to be
> > confident that there's a good solution to the datagram problem before I
> > invest too much time in this.
> 
> Summary of our short discussion on IRC:
> 
> - Switching to SOCK_STREAM under the hood adds the necessary reliabilty
>   but breaks DGRAM message boundaries.
> 
> - There appears to be no way in Winsock to handle send buffer overflow
>   gracefully so that user space knows that messages have been discarded.
>   Strange enoug there's a SIO_ENABLE_CIRCULAR_QUEUEING ioctl, but that
>   just makes things worse, by dropping older messages in favor of the
>   newer ones :-P
> 
> I think it should be possible to switch to STREAM sockets to emulate
> DGRAM semantics.  Our advantage is that this is all local.  For all
> practical purposes there's no chance data gets really lost.  Windows has
> an almost indefinite send buffer.
> 
> If you look at the STREAM as a kind of tunneling layer for getting DGRAM
> messages over the (local) line, the DGRAM content could simply be
> encapsulated in a tunnel packet or frame, basically the same way the
> new, boring AF_UNIX code does it.  A DGRAM message encapsulated in a
> STREAM message always has a header which at least contains the length of
> the actual DGRAM message.  So when the peer reads from the socket, it
> always only reads the header until it's complete.  Then it knows how
> much payload is expected and then it reads until the payload has been
> received.

Oh, btw., given this being local, and given that we always send a
defined packet length, with DGRAM max reliable packet size much smaller
than max STREAM packet size, there's almost no chance that the peer gets
an incomplete packet.  Unless, of course, user space requested a smaller
packet size than the sender sent.  In that case the remainder of the
packet is lost, but that's business as usual (MSG_TRUNC).  Obviously the
recv call has to read the entire packet and just discard the rest.


Corinna