Problems with the (new) implementation of AF_UNIX datagram sockets

[Ken Brown]

On 4/15/2021 7:49 AM, Corinna Vinschen wrote:
> Hi Ken,
> 
> On Apr 14 12:15, Ken Brown wrote:
>> Hi Corinna,
>>
>> This is a follow-up to
>>
>>    https://cygwin.com/pipermail/cygwin/2021-April/248284.html
>>
>> I don't know if you've been following that thread, but two serious problems
>> with datagram sockets (on the topic/af_unix branch) have shown up.
>>
>> 1. Writing will block until a connection to the peer's pipe can be made.  In
>> particular, if there are two consecutive writes with the same peer, the
>> second one will block until the peer reads the first message.  This happens
>> because the peer's pipe is not available for the second connection until the
>> peer disconnects the first connection.  This is currently done in recvmsg,
>> and I don't see a straightforward way to do it anywhere else.
> 
> I'm a bit puzzeled.  The idea for datagrams was to call open/send/close
> in each invocation of sendmsg.  Therefore the pipe should become
> available as soon as the other peer has sent it's data block.  The time
> a sendmsg has to wait for the pipe being available should be quite short!

Unfortunately, the pipe isn't available until the server disconnects.  I 
observed this in practice, and it's also documented at

https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-disconnectnamedpipe

"The server process must call DisconnectNamedPipe to disconnect a pipe handle 
from its previous client before the handle can be connected to another client by 
using the ConnectNamedPipe function."

>> 2. There's no way for select to test whether a datagram socket is ready for
>> writing.  That's because we can't know whether a connection to a
>> hypothetical peer's pipe will be possible.  According to Stevens, the issue
>> *should* be whether there's space in the socket's send buffer.  But our
>> sockets don't have a send buffer until they connect to a pipe.
> 
> Even then, there's no guarantee a send will succeed, given that
> select/send are not running atomically.  However, we *could* for a start
> always return success from select for this scenario.  If we have a
> nonblocking socket, it should fail opening the pipe and return EGAIN,
> which is perfectly fine.  If we have a blocking socket, it could block
> on send, which is perfectly valid, too, because of the non-atomicity.
> 
> Or am I missing something?

No, I was missing the non-atomicity.  So maybe that's OK.

>> I think the solution to both problems is for Cygwin to maintain a send
>> buffer for datagram sockets.  Does that seem right, or do you have another
>> idea?
> 
> In theory the send buffer should be a shared buffer between all peers,
> so this could be constructed as a shared ring buffer, accessible from
> af_unix_shmem_t.  But then again, this introduces a security problem,
> so that's not a good idea.  So, process-local buffers.
> 
> But you also have the problem how to empty the buffer.  Do you start a
> new thread which checks if the pipe is getting available and if so,
> sends the buffer content?  In which process?  And what do you do if
> there's still data in the send buffer when the process exits?  This is
> annoyingly complicated and error-prone.

Agreed.

> Another idea might be to implement send/recv on a DGRAM socket a bit
> like accept.  Rather than creating a single_instance socket, we create a
> max_instance socket as for STREAM socket listeners.  The server side
> accepts the connection at recv and immediately opens another pipe
> instance, so we always have at least one dangling instance for the next
> peer.

I thought about that, but you would still have the problem (as in 1 above) that 
the pipe instance isn't available until recv is called.

> 
> Corinna
> 
> 
> P.S.: Idle musings...
> 
> The only other implementation of AF_UNIX sockets using named pipes on
> Windows I know of (U/WIN) implements the gory details as part of their
> priviledged server process, i. e., their equivalent of cygserver.  The
> difference is that the entire system is based on this server process, so
> the U/WIN processes don't run at all if that service isn't running,
> quite unlike Cygwin.  Requiring a server running just to allow AF_UNIX
> sockets to work seems a bit off for us...
> 
> Having said that, maybe the idea to implement AF_UNIX sockets as named
> pipes is... outdated?  Roughly 90% of our users are running a W10
> version supporting AF_UNIX sockets natively (albeit missing native
> SOCK_DGRAM support).  Perhaps it's time to switch...?

Maybe so.

Ken