RE: Cygwin passes through null writes to other software when redirecting standard input/output (i.e. piping)

["James Johnston" <JamesJ at motionview3d dot com>]

> -----Original Message-----
> Sent: Friday, April 27, 2012 14:38
> Subject: Re: Cygwin passes through null writes to other software when
> redirecting standard input/output (i.e. piping)
> 
> What I don't grok is this:
> 
>   In your example, A and B are both native (== non-Cygwin) applications.
> 
> As I already mentioned in another mail related to the .Net problem, while
> the Cygwin shell running at the time creates the pipe, it doesn't handle read
> and write calls on the pipe.  It only creates the two native child processes and
> passes on the pipe handles.
> 
> So in the end, there is no Cygwin process involved while the native processes
> are doing their reads and writes.  It's all in the hands of these processes.
> There's no third process in the middle which handles the passing of
> information from A to B.
> 
> So, what do you expect Cygwin to do?

Thanks for responding.  I apologize for not fully understanding the problem earlier.  In .NET, if you wanted to pipe the output of one process to another, you'd have to be exactly this third process as far as I can tell - the framework creates a pipe/stream for each process, so you'd need a loop to copy information from one pipe to another.

To help understand how this is all working, I used API Monitor to spy and see how the Windows command prompt sets up its pipe.  It's very simple: it can be simplified down to a call to CreatePipe to make an anonymous pipe.  The two child processes are then spawned, with the pipe reader/writer handles being passed to CreateProcess.  Unused pipe handles are closed in both the parent process and the child processes, so that the processes terminate correctly.  I wrote a straight Win32 C program to spawn my child processes and direct output from one to another's input, using CreatePipe.  The results were correct.

> What you are seeing may be because Cygwin was changed to use
> message-type pipes a couple of revisions ago.  This is not going to
> change.  The change was adopted to fix a problem with Cygwin
> programs and those are obviously our #1 priority.

I will certainly not disagree with you that Cygwin programs need to work 100%!  But surely there has got to be some solution that can achieve this while not widely breaking non-Cygwin programs..!?  I'm actually surprised more people wouldn't  be using Windows programs from Cygwin - to me, that is part of the point of using a system like Cygwin rather than just installing the Linux distro of choice - so I can mix/match both Windows and UNIX software.  For me, both Windows and Cygwin software need to work - not just one or the other.  While you have near-zero interest in running Windows programs from Cygwin - I would bet that does not describe a significant number of users.

So to confirm your hypothesis, I modified my previously-described C program.  It uses CreateNamedPipe followed by CreateFile to get reader/writer handles instead of making an anonymous pipe using CreatePipe.  If I use byte mode, the results are still correct - apparently byte mode pipes won't pass through null writes.  HOWEVER, if I switch to message mode - the results are incorrect - just like the bad results from Cygwin.  Message mode pipes pass through null writes.

So I think for sure, Cygwin's use of message pipes is breaking a lot of Windows software, because of the null writes.  And ALSO additionally perhaps because of this: while reading MSDN today, I came across an interesting snippet that probably indicates a second source of problems I have not explicitly tested for, but would probably find:

"If the specified number of bytes to read is less than the size of the next message, the function reads as much of the message as possible before returning zero (the GetLastError function returns ERROR_MORE_DATA). The remainder of the message can be read using another read operation."

While I did not test for this, I think this would also break a lot of Windows software.  It's conceivable that the sender could write out a lot of data, while the receiver isn't reading anything.  Then, if the reader tries to read but with a small buffer, according to the docs it will get ERROR_MORE_DATA.  Most runtimes donât expect ReadFile to fail with ERROR_MORE_DATA.  For example, the decompiled .NET Framework code I read would throw an exception, as it only gracefully handles ERROR_BROKEN_PIPE.  Probably others like the Visual C++ runtime will treat it as EOF or set the failbit.  If there's interest I expect I could write a test case. (e.g. run "cat" to pipe a gigantic text file to a Windows program that takes a "long" time to process each line).

So I have some questions...

1.  Why on earth did Cygwin start using message pipes?  "adopted to fix a problem" doesn't help... Googling site:Cygwin.com "message pipe" - didn't yield anything either.  I do know older Cygwin versions worked fine, and I had no problem with piping data to/from both Cygwin and non-Cygwin programs.  I am sure it was changed for a good reason, but would someone be willing to offer an explanation for why message pipes are now used?  Do they have problems reading from a byte pipe?  Writing to a byte pipe?  Both?  Under what conditions?

2.  Is it possible that byte pipes could be used to communicate with non-Cygwin programs, while message pipes used to communicate between Cygwin programs?  Since I don't know the answer to the previous question, I have no idea how this question could be answered without causing / moving the breakage somewhere else. 

Here's sample code that can demonstrate both byte and message pipes.  Edit the EXE names in the calls to StartProc() accordingly.

#include <windows.h>
#include <iostream>
using namespace std;
/* error handling left out for brevity */

/* start process with specified input/output handles and return process handle */
HANDLE StartProc(wchar_t *proc, HANDLE in, HANDLE out)
{
	STARTUPINFO si;
	PROCESS_INFORMATION pi;
	memset(&si, 0, sizeof(si));
	memset(&pi, 0, sizeof(pi));

	si.cb = sizeof(si);
	si.dwFlags = STARTF_USESTDHANDLES;
	si.hStdInput = in;
	si.hStdOutput = out;
	si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
	CreateProcess(proc, NULL, NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi);
	CloseHandle(pi.hThread);
	return pi.hProcess;
}

int main()
{
	HANDLE hPipeRead, hPipeWrite;
	SECURITY_ATTRIBUTES secat;
	memset(&secat, 0, sizeof(secat));
	secat.nLength = sizeof(secat);
	secat.bInheritHandle = TRUE;
	/* note: changing to PIPE_TYPE_BYTE will fix the problems */
	hPipeWrite = CreateNamedPipe(L"\\\\.\\pipe\\CoordTestPipe", PIPE_ACCESS_OUTBOUND, PIPE_TYPE_MESSAGE, 1, 0, 0, 0, &secat);
	/* FILE_WRITE_ATTRIBUTES required if you play with SetNamedPipeHandleState */
	hPipeRead = CreateFile(L"\\\\.\\pipe\\CoordTestPipe", GENERIC_READ | FILE_WRITE_ATTRIBUTES, 0, &secat, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
	DWORD mode = PIPE_READMODE_BYTE;
	/*SetNamedPipeHandleState(reader, &mode, NULL, NULL); // this won't fix anything for the poor reader program */

	HANDLE hSender, hReceiver;
	hSender = StartProc(L"Sender.exe", GetStdHandle(STD_INPUT_HANDLE), hPipeWrite);
	hReceiver = StartProc(L"Receiver.exe", hPipeRead, GetStdHandle(STD_OUTPUT_HANDLE));
	/* close pipe handles that aren't needed; processes won't terminate otherwise */
	DuplicateHandle(hSender, hPipeRead, NULL, NULL, 0, FALSE, DUPLICATE_CLOSE_SOURCE);
	DuplicateHandle(hReceiver, hPipeWrite, NULL, NULL, 0, FALSE, DUPLICATE_CLOSE_SOURCE);
	CloseHandle(hPipeRead);
	CloseHandle(hPipeWrite);
	/* wait for child processes to exit */
	WaitForSingleObject(hSender, INFINITE);
	WaitForSingleObject(hReceiver, INFINITE);
	CloseHandle(hSender);
	CloseHandle(hReceiver);
	return 0;
}

> Your complete agreement with each other is not going to have much
> effect.  Cygwin source code is not changed by voting.

OK, fine.  What would you have us do then?  Not provide any seemingly-bothersome input / opinions at all, if the maintainers have no interest in what the users might run into?  I realize you are all very busy people, and that the Cygwin project doesn't have the major backing that larger projects do.  Fixing lower priority bugs might not be possible on your own.   But do you even want to hear feedback and bug reports if they are not accompanied by a perfectly-written patch, or are they too bothersome?  Are the maintainers interested in bug reports, or are they not?  I have already spent a fair bit of time researching this issue, and it's not the first Cygwin issue I've encountered.  I'm going to this trouble because I was hoping that in some way, Cygwin can ultimately be improved - if not by my hands then perhaps by the hands of someone else who might be reading this list and have more ideas or skills!

Write my own patch?  OK, but... I get the strong feeling that: (1) I won't have anyone to ask regarding where in the code I might need to even start looking and would be willing to answer any questions, and any considerations I might need to take into account - I have never actually worked on the Cygwin code before - and there is a lot of it - and I don't want to "cowboy" my way through it and make more problems than I fix! (2) I'm afraid you'd summarily ignore any patch I might come up with, without providing real, helpful feedback on what would need to changed and improved in the patch for it to be accepted.  Reading over the Cygwin contribution guidelines indicates there are an awful lot of things I have to get perfectly right, and frankly I am not sure I am going to get it perfectly right.  I don't want to waste more hours trying to patch this thing if there's no real chance of acceptance.

> Maybe I'm in the minority but I find long-winded impassioned pleas
> rather boring and off-putting.  So, continuing in this vein is not going
> to be productive for you, at least as far as I'm concerned.

Oh good grief... Most of the length in my e-mails have been due to me attempting to do more detailed research than the average person might do.  Would I be better off writing short e-mails that say "Cygwin doesnât work, why?"- without providing more research?

James 


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple