[Spam?]Re: problem concating (>>) to a large file

Eric Blake ebb9@byu.net
Tue May 29 15:20:00 GMT 2007

Peter Milne <peter.milne <at> netspeed.com.au> writes:

> Hi Eric
> Were you able to reproduce the problem I encountered with dd?

Yes - and it looks like there is indeed a cygwin bug, unrelated to my newlib 
patch to stdout.

$ cd /tmp
$ dd bs=1 seek=4540030013 if=/dev/zero of=huge count=1
$ ls -l huge blah
-rw-r--r-- 1 eblake Domain Users         15 May 29 08:27 blah
-rw-r--r-- 1 eblake Domain Users 4540030014 May 29 08:21 huge
$ cat foo.c
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>

int main (int argc, char **argv)
   fprintf (stderr, "%lld\n", lseek (1, 0, SEEK_CUR));
   fprintf (stderr, "%x\n", fcntl (1, F_GETFL));
   return 0;
$ ./foo >> blah1
$ ./foo >> huge

Somehow, when the file size is huge, cygwin is not properly propogating that an 
O_APPEND bit (0x9 in the flags printed from F_GETFL) means that the initial 
offset of fd 1 is the end of the file.  (And it would be nice if strace would 
show the initial lseek offset of all inherited fd's when spawning a process).

You ALSO pointed out a bug in cat - it is calling freopen(NULL, "wb", stdout), 
which effectively wipes out whether stdout was previously in append mode; this 
is an upstream bug, but it only affects cygwin and other platforms where "wb" 
makes a difference.  I'll see about rolling out coreutils-6.9-3 that works 
around it, while posting a patch upstream to fix it.  For small files, you got 
lucky - even though the append mode was lost, the file position was already at 
the end of the file, and since cat doesn't seek, the output went to the right 
location.  But for large files, because of the cygwin bug where the file offset 
is wrong, the output is written to the wrong offset, even WITH the newlib 
stdout patch.

Three bugs in three different source repositories via one problem report (one 
in newlib, already fixed; one in cygwin, and one in coreutils)!  That's gotta 
be some sort of record.

Eric Blake
volunteer cygwin coreutils maintainer

Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

More information about the Cygwin mailing list