[PATCH v4 1/1] Cygwin: pty: Fix state management for pseudo console support.

[Takashi Yano]

Hi Corinna,

I have posted several patches for PTY with pseudo console support.
Please apply them in the following order.

[PATCH 1/4] Cygwin: pty: Code cleanup
- Cleanup the code which is commented out by #if 0 regarding pseudo
  console.
- Remove #if 1 for experimental code which seems to be stable.

[PATCH 2/4] Cygwin: pty: Speed up a little hooked Win32 API for pseudo console.
- Some Win32 APIs are hooked in pty code for pseudo console support.
  This causes slow down. This patch improves speed a little.

[PATCH 3/4] Cygwin: pty: Move function hook_api() into hookapi.cc.
- PTY uses Win32 API hook for pseudo console suppot. The function
  hook_api() is used for this purpose and defined in fhandler_tty.cc
  previously. This patch moves it into hookapi.cc.

[PATCH 4/4] Cygwin: pty: Limit API hook to the program linked with the APIs.
- API hook used for pseudo console support causes slow down.
  This patch limits API hook to only program which is linked
  with the corresponding APIs. Normal cygwin program is not
  linked with such APIs (such as WriteFile, etc...) directly,
  therefore, no slow down occurs. However, console access by
  cygwin.dll itself cannot switch the r/w pipe to pseudo console
  side. Therefore, the code to switch it forcely to pseudo
  console side is added to smallprint.cc and strace.cc.

[PATCH v5 1/1] Cygwin: pty: Fix state management for pseudo console support.
- Pseudo console support introduced by commit
  169d65a5774acc76ce3f3feeedcbae7405aa9b57 has some bugs which
  cause mismatch between state variables and real pseudo console
  state regarding console attaching and r/w pipe switching. This
  patch fixes this issue by redesigning the state management.

This hopefully fixes the problem 3 in
https://cygwin.com/ml/cygwin/2019-08/msg00401.html
This also fixes the first problem regarding "Bad file descriptor" error
reported in
https://cygwin.com/ml/cygwin-patches/2019-q3/msg00104.html

[PATCH 1/2] Cygwin: pty: Add a workaround for ^C handling.
- Pseudo console support introduced by commit
  169d65a5774acc76ce3f3feeedcbae7405aa9b57 sometimes cause random
  crash or freeze by pressing ^C while cygwin and non-cygwin
  processes are executed simultaneously in the same pty. This
  patch is a workaround for this issue.

[PATCH 2/2] Cygwin: pty: Disable clear screen on new pty if TERM=dumb or emacs*.
- Pseudo console support introduced by commit
  169d65a5774acc76ce3f3feeedcbae7405aa9b57 shows garbage ^[[H^[[J in
  some of emacs screens. These screens do not handle ANSI escape
  sequences. Therefore, clear screen is disabled on these screens.

This fixes the second problem on emacs reported in
https://cygwin.com/ml/cygwin-patches/2019-q3/msg00104.html

On Tue, 3 Sep 2019 11:16:38 +0200
Corinna Vinschen wrote:
> This is a slowdown of about 15%.  That's quite a lot.  Can't you just
> check the incoming handle against the interesting handles somehow?
> If there's no other way around it, we should at least make sure (in a
> separate patch) that Cygwin calls NtReadFile/NtWriteFile throughout,
> except in tty and console code.

I came up with an idea, and implemented it. As described obove,
Win32 APIs are not hooked any more in normal cygwin process.
Hook is done only if the program is directly linked with corresponding
APIs. However, this strategy does not have the effect for console
access by cygwin1.dll itself. So, to switch r/w pipe to pseudo console
side, I added the code in strace.cc and smallprint.cc.

Could you please have a look?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>