cygwin 3.3.x: another problem that may be related to pipes

Corinna Vinschen corinna-cygwin@cygwin.com
Mon Nov 15 16:08:02 GMT 2021


On Nov 15 23:50, Takashi Yano wrote:
> On Mon, 15 Nov 2021 14:01:12 +0100
> Corinna Vinschen wrote:
> > [Redirected to cygwin-developers]
> > 
> > On Nov 15 17:18, Takashi Yano via Cygwin wrote:
> > > On Sun, 14 Nov 2021 20:50:59 +0000 (UTC)
> > > bf wrote:
> > > > I've a shell script that processes several hundred xz-compressed text files with a pipeline of the form:
> > > > 
> > > > find . -depth -type f  ... -print0 | \
> > > > xargs -0tI @ xzcat -- @ | \
> > > > iconv --byte-subst='�' --unicode-subst='�' --widechar-subst='�' -f MS-ANSI -t UTF-8 -- | \
> > > > grep ... \
> > > > sed ... | \
> > > > sort ... | \
> > > > sort ...
> > > > 
> > > > . Since updating to cygwin 3.3.x (but not on cygwin 3.2.x) the script consistently fails when decompressing a portion of the files, resulting in lost data, and error messages of the form:
> > > > 
> > > > xzcat: (stdout): Write error: Bad address
> > > 
> > > Thanks for the report.
> > > I could reproduce your problem and found the cause.
> > > 
> > > raw_read()/raw_write() in fhandler_pipe.cc seems to need handling
> > > of STATUS_PENDING in nonblocking mode.
> > > 
> > > I also confirmed the following patch fixes the issue. However, I
> > > am not very sure that this is the right thing.
> > > 
> > > Corinna, Ken, what do you think?
> > 
> > I'm puzzled.  non-blocking pipes return STATUS_PENDING?  What on earth
> > is that supposed to mean on non-blocking (as opposed to asynchronous)
> > pipes?  The problem is that STATUS_PENDING theoretically requires
> > to wait for... something, the pipe handle, perhaps, and then to check
> > io.Status.
> > 
> > Is it possible that nonblocking pipes need to be synchronous, i. e.,
> > setting the FILE_SYNCHRONOUS_IO_NONALERT would fix that?
> 
> Yes. Setting FILE_SYNCHRONOUS_IO_NONALERT for the write pipe seems
> to fix the issue.

That was just a hunch but it doesn't *actually* surprise me...

> Proberbly, this needs the modification for
> raw_write() like the modification I have made for raw_read() in
> topic/pipe branch.

The idea was to do change to synchronous pipes only for 3.4.  I have a
bit of headaches to change the pipes to synchronous as bug fix on the
3.3 branch.  Your first ideas might be better there...

...unless, of course, you and Ken are convinced that this is the best
approach even as 3.3 bugfix.  In that case, just say so and push the
stuff to the 3.3 branch, too, as you see fit.


Thanks,
Corinna


More information about the Cygwin-developers mailing list