[newlib-cygwin/main] Cygwin: open: Fix Windows resource leak after fd exhaustion

[Corinna Vinschen]

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=335c25cc2b9208b77f8a2d39634ec3983a6a8e6c

commit 335c25cc2b9208b77f8a2d39634ec3983a6a8e6c
Author:     Igor Podgainoi <Igor.Podgainoi@arm.com>
AuthorDate: Wed Feb 18 09:43:41 2026 +0000
Commit:     Corinna Vinschen <corinna@vinschen.de>
CommitDate: Mon Mar 2 20:39:16 2026 +0100

    Cygwin: open: Fix Windows resource leak after fd exhaustion
    
    In a specific rare case when a Cygwin process runs out of available
    file descriptor numbers (errno set to EMFILE), the underlying Windows
    HANDLE is not being closed. This is partly because currently the given
    file is first opened natively before a new Cygwin file descriptor has
    been assigned - the logic overlooks the fact that it is possible for
    the Windows HANDLE to be valid, but not the internal fd.
    
    Even though the object is explicitly freed from memory later using
    operator delete, the fhandler_disk_file class has no destructor
    defined to mitigate the leak.
    
    This patch introduces a manual call to fh->close() if the assigned fd
    value returned by the operator int &() function in the cygheap_fdnew
    class is less than 0.
    
    Test fixed on AArch64 and x86_64:
    winsup.api/ltp/dup03.exe
    
    Signed-off-by: Igor Podgainoi <Igor.Podgainoi@arm.com>

Diff:
---
 winsup/cygwin/syscalls.cc | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index 1b1ff17b0a5d..7a8e5d4fd56f 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -1576,7 +1576,10 @@ open (const char *unix_path, int flags, ...)
       cygheap_fdnew fd;
 
       if (fd < 0)
-	__leave;		/* errno already set */
+	{
+	  fh->close();
+	  __leave;		/* errno already set */
+	}
 
       fd = fh;
       if (fd <= 2)