[PATCH] Cygwin: ptsname_r: always return an error number on failure

[Ken Brown]

Following Linux, return ENOTTY on a bad file descriptor and also set
errno to ENOTTY.

Previously 0 was returned and errno was set to EBADF.  Returning 0
violates the requirement in
https://man7.org/linux/man-pages/man3/ptsname_r.3.html that an error
number should be returned on failure.  (That man page doesn't specify
setting errno.)

Addresses: https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00245.html
---
 winsup/cygwin/release/3.2.0 | 3 +++
 winsup/cygwin/syscalls.cc   | 5 ++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/winsup/cygwin/release/3.2.0 b/winsup/cygwin/release/3.2.0
index 43725cec2..f748a9bc8 100644
--- a/winsup/cygwin/release/3.2.0
+++ b/winsup/cygwin/release/3.2.0
@@ -52,3 +52,6 @@ Bug Fixes
 - Fix the errno when a path contains .. and the prefix exists but is
   not a directory.
   Addresses: https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00214.html
+
+- Fix the return value when ptsname_r(3) is called with a bad file descriptor
+  Addresses: https://lists.gnu.org/archive/html/bug-gnulib/2021-01/msg00245.html
diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index 4742c6653..18d9e3f88 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -3364,7 +3364,10 @@ ptsname_r (int fd, char *buf, size_t buflen)
 
   cygheap_fdget cfd (fd);
   if (cfd < 0)
-    return 0;
+    {
+      set_errno (ENOTTY);
+      return ENOTTY;
+    }
   return cfd->ptsname_r (buf, buflen);
 }
 
-- 
2.30.0