unix domain socket with shared memory ?

Ralf Habacker Ralf.Habacker@freenet.de
Tue Feb 12 05:10:00 GMT 2002


http://i30www.ira.uka.de/courses/winter00-01/Prosem_LinuxInternals/4.4ipc.ppt

> > Hi all,
> >
> > cfg has told me about the current process of cygwin daemon
> implementation with ipc support.
> >
> > I initial have heard last year, that this work would be started, but
> because of so much other work I have lost the
> > contact to the ongoing process.
>
> It's currently paused, waiting for the development team to persuade
> Chris that the _core_ (not the IPC part) is ready for (finaly filenaming
> and then) inclusion.
>
> > Now I was looking into the ongoing work and it seems to me in a mostly
> read state, isn't it. I like to say: Great
> > work to all who have worked on it. :-)
>
> Thank you. The main folk are Egor Duda who created an inital
> proof-of-concept for solving a security issue, most of the developers
> list bouncing round ideas, and then I (*bow*) took Egors proof of
> concept and created the generic daemon, and the win9x support that it
> now has. Gary R. Van Sickle has also spent some time recently being a
> guinea pig, which I appreciate :].
>
> > The reason why I'm writing this is that I have recognized some
> performance issues with unix domain sockets, which
> ...
>
> > Because the cygwin-daemon branch provides the long missed ipc support,
> the way for for speeding up unix domain
> > sockets with a shared memory implementation may be free. (I not be not
> first one, who tells about this, because I
> > have seen some guys before talking about this possibility)
>
> Hmm, the cygwin-daemon IPC implementation may allow this, but so does
> the existing mmap implementation. So I think a better approach may be
> via mmap (more mature code) with the daemon providing
> cross-process-persistence (if needed).
>
> > One open topic for me is how to handle forking. I imagine, that
> because the file handles are duplicated, the shared
> > memory has to be duplicatd too, but because the socket file name is
> the same, does it use the same shared memory
> > area as the parent or not ???
>
> Forking is easy to implement (see the fixup_after_fork functions) but
> shouldn't be needed if implemented on top of mmap or IPC memory regions.
> What maybe an issue is the logic for how it should behave.
>
> If you write, who reads? The parent or the child? The first reader? Once
> this question is answered, you will find making fork() work straight
> forward. Advice can be offered at that point if needed.
>

How is it implemented in the current implementation ?

> > My intention whis this thread is to make sure, that this strategy is a
> possible way and I'm willing to spent some t
> > ime to get this running, although I think I'm not be able to handle
> this whole task alone.
>
> Yes, it's possible. Is the daemon needed? Maybe not, maybe. You will get
> assistance in doing this.
>
So this is my current strategy for the beginning.

The first task seems to me to reorganice net.cc, so that all AF_INET/AF_LOCAL specific functions are located into
methods of fhandler_socket. After doing this net.cc contains only wrappers for the above mentioned functions.

See a snippet of my current fhandler.h

class fhandler_socket
....
#ifdef NEW_NET
  // a FIXME means, this functions isn't moved into class fhandler_socket yet
  // a CHECK means, this functions is implemented and has to be check for valid functionality

  // FIXME: move from net.cc to fhandler_socker.cc
  int socketpair (int family, int type, int protocol, int *sb);
  // FIXME: move from net.cc to fhandler_socker.cc
  int socket (int af, int type, int protocol);
  // CHECK:
  int bind (const struct sockaddr *my_addr, int addrlen);
  // CHECK:
  int listen (int backlog);
  // CHECK:
  int accept (struct sockaddr *peer, int *len);
  // CHECK:
  int recv (const void *buf, int len, unsigned int flags);
  // CHECK:
  int send (const void *buf, int len, unsigned int flags);
  // CHECK:
  int connect (const struct sockaddr *name, int namelen);
  // CHECK:
  int shutdown (int how);
  // CHECK:
  int getsockname (struct sockaddr *addr, int *namelen);
  // CHECK:
  int getpeername (struct sockaddr *name, int *len);
#endif

This seems to me as a general optimation, which should be incorporated also in the main cygwin branch, because it
would simplifies the process of integrating other socket implementations. Any comments to this ?

The second task is to fork a new fhandler_local class from fhandler_socket, change the above mentioned methods into
virtual methods, which contains all AF_LOCAL specific stuff.

Perhaps is is better to create a base fhandler_socket class with virtual functions and than derivate
fhandler_socket_tcp (which is currently named fhandler_socket) and fhandler_socket_local from this.

net.cc and other files should only use the basic socket class "fhandler_socket", so that the right virtual
functions are used.

This class has be added to the fhandler instance creation dispatcher in path.cc.
I have also seen some FH_SOCKET related stuff in dtable.cc:dtable::release() and net.cc:fdsock(). In fdsock, the
depending on the protocol familiy, the propper objects should be created from fhandler_socket_tcp for tcp sockets
and fhandler_socket_local for unix domain sockets.

If the current fhandler_socket.cc is copied into fhandler_socket_tcp.cc and fhandler_socket_local.cc in this step,
the cygwin1.dll should work as before, because the changes has no functionality changed only source layout.

This step could be also done in the main cygwin tree.

The third step seems to me to implement the new af_local stuff in fhandler_socket_local.cc, which should be
discussed. A sample implementation could be found unter
http://i30www.ira.uka.de/courses/winter00-01/Prosem_LinuxInternals/4.4ipc.ppt, but currently I don't know if there
are some copyrights on this.

For step one a patch is appended (about 95% complete, socket and socketpair are missing, because they don't use an
existing fhandler_socket, instead they create such, so integrating it into fhandler_socket seems not right for me).
The patch was created from the winsup dir assuming that the cygwin-daemon source is located into cygwin-daemon.
Additional it uses a MACRO named NEW_NET, which enables the moved code. If this macro is unset the present
implementation is used. (see Makefile.in)
Because I have checked out the winsup/cygwin tree and the winsup/cygwin-daemon tree in one directory, a patched
winsup/Makefile.common-daemon is needed.
This isn't a ready-for-check-in-patch, so I don't have added a ChangeLog. Also there are no functionality changes,
so I think no copyright assignment is nessessary.
I see this patch as as discussion base. Any comments ?
I have verified propper operation with a sample unix socket client/server from
http://www.ecst.csuchico.edu/~beej/guide/ipc/usock.html. I know, that this does not check all patched methods, so
it would be nice if somebody else could add some tests.

Note: There may be some need to add some sigframe lines, but I'm not sure, where this should be placed.

Another topic which would be nesessary to observe seems to me be the comunication latencies, which the benchmark
shows.
Currently I don't know how much the influence on unix domain socket performance is, but what I see is that for pies
and AF_UNIX the latency are about 700 times respective 370 times more thant for linux with the same hardware.
Tcp shows similar results. UDP has a latency of more than 100 and tcp about 2570 times.

*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                        ctxsw       UNIX         UDP         TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
BRAMSCHE  CYGWIN_NT-5.0 4970. 14.7K 14.K 107.1       13.5K       550K
BRAMSCHE   Linux 2.2.18 3.290  19.2 39.6  76.8  85.3 128.0 238.0 214.
                               ^^^^ ^^^^             ^^^^        ^^^^
Currently it seems to me, that this latencies are caused by the windows and/or cygwin context/task switching
implementation.

The reason why I suggess this are

1. context switching seems to be a general optimization topic if you see in the results shown below.

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host                 OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
                        ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
--------- ------------- ----- ------ ------ ------ ------ ------- -------
BRAMSCHE  CYGWIN_NT-5.0 4970. 5959.5 5933.2 3177.0 8706.5  4218.8  6200.8
BRAMSCHE   Linux 2.2.18 3.530   12.6   25.1   19.8  161.3    31.4   162.6

2. UDP transfer has much less latencies, so this says it is possible to avoid latencies, but I haven't found the
difference yet.

Any comments ?

Ralf








-------------- next part --------------
A non-text attachment was scrubbed...
Name: cygwin-daemon_NEW_NET_01.patch
Type: application/octet-stream
Size: 17968 bytes
Desc: not available
URL: <http://cygwin.com/pipermail/cygwin/attachments/20020212/dcd56b92/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Makefile.common-daemon
Type: application/octet-stream
Size: 5005 bytes
Desc: not available
URL: <http://cygwin.com/pipermail/cygwin/attachments/20020212/dcd56b92/attachment-0001.obj>
-------------- next part --------------
--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/


More information about the Cygwin mailing list