Error piping data through a cygwin command using rsh (write system call fails)

[Johnny B. Goode]

We're using a windows 2000 based server running cygwin as a development 
environment for porting our AIX-based software to the Windows platform.

When we extract the compiled programs, we connect to the cygwin server using 
rsh, passing a file list as input, returning a gzip'ed CPIO archive as 
output using a command like:

	cat filelist | rsh cygserver -l logname "cd /directory ; cpio -oc | gzip" > 
output.cpio.gz

This is executed from an AIX server (running AIX 5.3), effectively packing 
files from the filelist in a CGZ-archive that ends up back on the AIX 
server. The problem is, however, that gzip on the cygwin server terminates 
prematurely with the error "gzip: stdout: Invalid argument".

The problem has been reproduced on two cygwin servers, based on Windows 2000 
and Windows XP Proffessional. It can be reproduced this way:


1. Set up inetd on the Cygwin server:

cyg$ cygrunsrv -I inetd -d "CYGWIN inetd" -p /usr/sbin/inetd -a -d -e 
CYGWIN="tty ntsec"
cyg$ cygrunsrv -S inetd

2. Still on the cygwin server, allow rsh from the alternate (AIX-)server (if 
the server is called aixserver and the user is called aixuser):

cyg$ vi ~/.rhosts
aixserver	aixuser

3. On the AIX server, get a filelist of /bin from the cygwin server 
(assuming the cygwin server has hostname cygserver on the AIX server and the 
cygwin/windows account is named cyguser):

aix$ rsh cygserver -l cyguser "find /bin -print" > filelist

Validate that filelist contains a list of /bin file names.

4. On the AIX-server, reproduce the error by issuing the following command:

aix$ cat filelist | rsh cygserver -l cyguser "cpio -oc 2>/dev/null | gzip" | 
gunzip | cpio -ivtc

(The redirection of the cpio stderr is to avoid "truncating inode number" 
errors.) On my system, this produces the output:

------------------------
40770  11049       0 Jun 08 14:34:58 2005 /bin
100750 11049  117760 Jan 27 12:46:12 2005 /bin/a2p.exe
100750 11049    7870 Dec 02 08:47:03 2003 /bin/aclocal
120777 11049      34 May 27 12:25:11 2005 /bin/aclocal-1.4
120777 11049      35 May 27 12:29:45 2005 /bin/aclocal-1.9

gzip: stdout: Invalid argument

gunzip: stdin: unexpected end of file
100750 11049   67072 Dec 16 19:03:23 2002 /bin/addftinfo.exe
Can't read input
-----------------------

As can be seen, the first few files are returned to the AIX cpio command, 
but at a certain point, the communication fails.

5. Further investigation: To be able to track what happens, I've installed a 
small custom program called "pipeline" in PATH on the cygwin server. 
"pipeline" is a very simple program passing data through itself from stdin 
to stdout, logging encountered write errors. Specifically, it retries a 
write operation if it does not succeed the first time (I am aware that the 
program does not handle partial writes correctly, but it does not need to in 
order to diagnose the error). Source code "pipeline.c" is as follows:

-----------------------
#include <stdio.h>
#include <fcntl.h>
#include <sys/errno.h>

main()
{
    int n, w, t = 0;
    char    buf[65536];
    setmode(0, O_BINARY);
    setmode(1, O_BINARY);
    while ((n=read(0, buf, sizeof(buf))) > 0) {
        w=write(1, buf, n);
        if (w != n && w >= 0) {
            fprintf(stderr, "Woops! @%d: Wrote returned %d, expected to 
write %d, errno %d. Retrying!\n",t, w, n, errno);
            w=write(1, buf, n);
            if (w != n && w >= 0) {
                fprintf(stderr, "Ouch! @ %d: Wrote returned %d, expected to 
write %d, errno %d\n", t, w, n, errno);
            }
        }
        if (w < 0) {
            perror("write");
            exit(1);
        }
        t += w;
    }
    if (n<0) {
        perror("read");
        exit(1);
    }
    return 0;
}
-----------------------

6. With the compiled pipeline program placed somewhere in $PATH on the 
cygwin server, we can run the following command from the AIX server:

aix$ cat filelist | rsh cygserver -l cyguser "cpio -oc 2>/dev/null | gzip | 
pipeline" | gunzip | cpio -ivtc

This time, the complete gzip'ed data stream is returned, all files listed by 
the AIX cpio command. The pipeline program however shows one error line 
during the run:

Woops! @81920: Wrote returned 2236416, expected to write 16384, errno 0. 
Retrying!

Now, this is quite puzzlingÂ… The write call is expected to return number of 
bytes written (here from 0 to 16384, since we try to write 16384 bytes) or 
-1 (which would indicate an error in errno). But write returns 2236416 - 
which indicates some problem in the implementation of the write call. A 
repeated call to write actually writes the data without problems, thus 
letting the complete data stream through.

7. Other informations

I've tried a few other tricks that might say something of the nature of the 
problem:

-----------------
The return value (2236416) seems to be constant; it is produced on both of 
my test servers. The position in the stream where the error occurs varies.

-----------------
If the filelist is first sent to the cygwin server (using rcp or whatever 
method), the following command does not produce the error.

aix$ rsh cygserver -l cyguser "cat filelist | cpio -oc 2>/dev/null | gzip" | 
gunzip | cpio -ivtc

So it seems that data must flow both ways in the socket to produce the 
error.

-----------------
If the file list is served slowly (by a command like the following), the 
error does not show:

aix$ cat filelist | while read file ; do echo $file ; sleep 1 ; done | rsh 
cygserver -l cyguser "cpio -oc 2>/dev/null | gzip | pipeline" | gunzip | 
cpio -ivtc

-----------------
The error will also show when running the rsh command from a Linux server (I 
tried from redhat 8). You may need to skip the "c" options to cpio.

-----------------
I've tried various CYGWIN environment options when setting up the inetd 
daemon - does not change anything.


I hope that someone with a good insight in the cygwin core can help me solve 
this problem. Attached is output of the cygcheck command, as well as the 
pipeline.c source code.

Best regards,

Soeren Hansen

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pipeline.c
URL: <http://cygwin.com/pipermail/cygwin/attachments/20050608/9633cf4c/attachment.c>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cygcheck.out
URL: <http://cygwin.com/pipermail/cygwin/attachments/20050608/9633cf4c/attachment.ksh>
-------------- next part --------------
--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/