[PATCH v2] Cygwin: console: Add guard for set/unset xterm compatible mode.

[Takashi Yano]

On Tue, 18 Feb 2020 11:43:36 +0100
Corinna Vinschen wrote:
> On Feb 18 18:12, Takashi Yano wrote:
> > - Setting / unsetting xterm compatible mode may cause race issue
> >   between multiple processes. This patch adds guard for that.
> > ---
> >  winsup/cygwin/fhandler.h          |   6 ++
> >  winsup/cygwin/fhandler_console.cc | 125 +++++++++++++++++++++---------
> >  winsup/cygwin/select.cc           |  22 ++----
> >  winsup/cygwin/spawn.cc            |   8 +-
> >  4 files changed, 103 insertions(+), 58 deletions(-)
> 
> The patch looks good to me, but I'm curious...
> 
> Yesterday you wrote that interlocked counting is not a good
> solution due to the 'bash -> cmd -> bash' scenario.  What has
> changed your mind?

Interlocking in this patch is not used in open/close, but
in read()/select() for input, and in write()/close()/exec()
for output. For input, InterlockedIncrement/Decrement() is
used and InterlockedExchage() is used for output.

As for bash->cmd->bash case, first xterm mode is enabled
for output and input in write() and read() in bash, then
xterm mode for input is disabled when it returns from read().
When cmd.exe is executed, xterm mode for output is disabled
in exec(). As a result, cmd.exe is executed under xterm mode
disabled. Next, when bash is executed, xterm mode is re-
enabled in write()/read().

After that, if second bash is exited, xterm mode for input
is disabled when returned from read(), and for output, it
is disabled in close(). Then, cmd.exe is executed under
xterm mode is disabled. If cmd.exe is exited, xterm mode
is re-enabled in write()/read() in bash.

After all, xterm mode for input is enabled/disabled each
read() and select() call. xterm mode for output is enabled
when write() is called and disabled in close() and exec()
for non-cygwin process.

On Mon, 17 Feb 2020 10:00:15 +0100
Corinna Vinschen wrote:
> In terms of this patch, rather than to change the mode on every
> invocation of read/write/select/close, wouldn't it make more sense to
> count the number of mode switches in a shared per-console variable, i.e.

In other words, it is not as suggested above.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>