set_open_status

[Ken Brown]

On 2/18/2021 8:07 AM, Corinna Vinschen via Cygwin-developers wrote:
> On Feb 18 10:07, Corinna Vinschen via Cygwin-developers wrote:
>> On Feb 17 16:35, Ken Brown via Cygwin-developers wrote:
>>> Most fhandlers call set_open_status in their 'open' method before a
>>> successful return.  I just noticed that fhandler_fifo::open doesn't do this.
>>> I thought at first that it was an oversight on my part from when I
>>> overhauled the FIFO implementation, but I just checked fhandler_fifo.cc as
>>> of cygwin 2.9.0, and I don't see it there either.
>>>
>>> Is there some reason this would be wrong for FIFOs?
>>
>> I guess this was just an oversight.  It's certainly not fatal, given how
>> open_status is used exclusively by fhandler_base::reset_to_open_binmode()
>> in turn *only* called from setmode(fd, 0).  This is quite a bordercase.
>> Combined with FIFOs supporting only O_BINARY mode anyway...
>>
>> It won't hurt to add the set_open_mode call to FIFOs but it won't change
>> anything, except avoiding to report O_TEXT mode after a call to
>> setmode(fd, 0).
>>
>> Maybe it would be better to change fhandler_base::reset_to_open_binmode
>> instead.  Right now, if open_status hasn't been initialized, it prefers
>> O_TEXT over O_BINARY.  That's unfortunate, because it forces us to call
>> set_open_mode everywhere to make sure the mode is stored correctly for
>> this single, and seldom, invocation of setmode(fd, 0).  It would be nice
>> if set_open_mode would only be necessary on fhandlers supporting O_TEXT
>> at all.
> 
> On second thought, this should do the trick:
> 
> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> index e457e2785e9b..ac9ee7c9e787 100644
> --- a/winsup/cygwin/fhandler.h
> +++ b/winsup/cygwin/fhandler.h
> @@ -274,7 +274,8 @@ class fhandler_base
>     void reset_to_open_binmode ()
>     {
>       set_flags ((get_flags () & ~(O_TEXT | O_BINARY))
> -	       | ((open_status.wbinary || open_status.rbinary)
> +	       | (((open_status.wbinset ? open_status.wbinary : 1)
> +		   || (open_status.rbinset ? open_status.rbinary : 1))
>   		   ? O_BINARY : O_TEXT));
>     }

Yes, this looks right.

Ken