[newlib-cygwin/cygwin-3_6-branch] Cygwin: open: Fix deadlock for opening fifo both side in a process

[Takashi Yano]

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=ffb33791d4c2f9f09488c268ba8233dd02255dff

commit ffb33791d4c2f9f09488c268ba8233dd02255dff
Author: Takashi Yano <takashi.yano@nifty.ne.jp>
Date:   Sat May 17 00:46:41 2025 +0900

    Cygwin: open: Fix deadlock for opening fifo both side in a process
    
    Currently, opening both side of fifo in a process hangs if the
    read side is opened first. The following test case exhibit the
    hang while it works in linux.
    
      #include <unistd.h>
      #include <pthread.h>
      #include <sys/stat.h>
      #include <fcntl.h>
    
      #define fifo1 "/tmp/fifo-test"
    
      void *thr1(void *)
      {
        int fd;
        usleep(100000);
        fd = open(fifo1, O_WRONLY);
        write(fd, "A", 1);
        usleep(100000);
        close(fd);
        return NULL;
      }
    
      int main()
      {
        int fd;
        pthread_t th;
        char c;
        mkfifo(fifo1, 0600);
        pthread_create(&th, NULL, thr1, NULL);
        fd = open(fifo1, O_RDONLY);
        pthread_join(th, NULL);
        read(fd, &c, 1);
        write(1, &c, 1);
        close(fd);
        unlink(fifo1);
        return 0;
      }
    
    The mechanism of hang is as follows. The main thread tries to open
    the fifo for reading, but fhandler_fifo::open blocks until it detects
    that someone is opening the fifo for writing. The other thread wants
    to do that, but it never gets to the point of calling fhandler_fifo::
    open because it is stuck waiting for the lock on cygheap->fdtab.
    
    To fix this, this patch delays the construction of the cygheap_fdnew
    object fd until after fhandler_fifo::open has been called.
    
    Fixes: df63bd490a52 ("* cygheap.h (cygheap_fdmanip): New class: simplifies locking and retrieval of fds from cygheap->fdtab.")
    Reviewd-by: Ken Brown <kbrown@cornell.edu>
    Signed-off-by: Takashi Yano <takashi.yano@nifty.ne.jp>
    (cherry picked from commit cec8a6680ea1fe38f38001b06c34ae355a785209)

Diff:
---
 winsup/cygwin/release/3.6.2 |  3 +++
 winsup/cygwin/syscalls.cc   | 11 ++++++-----
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/winsup/cygwin/release/3.6.2 b/winsup/cygwin/release/3.6.2
index e0c1d705f..915bd0150 100644
--- a/winsup/cygwin/release/3.6.2
+++ b/winsup/cygwin/release/3.6.2
@@ -22,3 +22,6 @@ Fixes:
 
 - Fix conserver-config error.
   https://cygwin.com/pipermail/cygwin/2025-April/258086.html
+
+- Fix deadlock for opening both side of a fifo in a process.
+  Addresses: https://cygwin.com/pipermail/cygwin/2025-May/258138.html
diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index c93bf4c95..d6a2c2d3b 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -1472,11 +1472,6 @@ open (const char *unix_path, int flags, ...)
       mode = va_arg (ap, mode_t);
       va_end (ap);
 
-      cygheap_fdnew fd;
-
-      if (fd < 0)
-	__leave;		/* errno already set */
-
       /* When O_PATH is specified in flags, flag bits other than O_CLOEXEC,
 	 O_DIRECTORY, and O_NOFOLLOW are ignored. */
       if (flags & O_PATH)
@@ -1577,6 +1572,12 @@ open (const char *unix_path, int flags, ...)
       if ((flags & O_TMPFILE) && !fh->pc.isremote ())
 	try_to_bin (fh->pc, fh->get_handle (), DELETE,
 		    FILE_OPEN_FOR_BACKUP_INTENT);
+
+      cygheap_fdnew fd;
+
+      if (fd < 0)
+	__leave;		/* errno already set */
+
       fd = fh;
       if (fd <= 2)
 	set_std_handle (fd);