[newlib-cygwin/cygwin-3_6-branch] mbrtowc: fix handling invalid UTF-8 4 byte sequences if wchar_t == UTF-16

[Corinna Vinschen]

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=4fc629ae7550e011738d386780b12882f1fdd6ce

commit 4fc629ae7550e011738d386780b12882f1fdd6ce
Author:     Corinna Vinschen <corinna@vinschen.de>
AuthorDate: Fri Jun 27 12:32:15 2025 +0200
Commit:     Corinna Vinschen <corinna@vinschen.de>
CommitDate: Fri Jun 27 12:35:55 2025 +0200

    mbrtowc: fix handling invalid UTF-8 4 byte sequences if wchar_t == UTF-16
    
    When commit 28186e81d947 split _mbtowc_r into per-codeset functions, the
    code generating wchar_t from UTF-8 input was slightly rearranged.
    Unfortunately the new code introduced a bug:
    
    On systems with wchar_t being UTF-16, 4 byte sequences have to be
    converted to surrogate pairs.  The low surrogate pair can be fully
    created from the first 3 bytes of the sequence.  However, the surrogates
    should only be created if it's clear that the 4th byte is valid, and the
    entire 4 byte string represents a valid UTF-8 sequence.
    
    The code change in 28186e81d947 neglected just that: In contrast to the
    original code, it now created the low surrogate after having read the
    first 3 bytes of the sequence, without checking validity of the 4th byte.
    
    This patch moves the test sequence to check the 4th byte in front of the
    code generating the low surrogate.  Make sure to return the value 3 (3
    bytes digested) rather than the content of the local variable i, which
    is already set to 4 at this point.
    
    Reported-by: Christian Franke <Christian.Franke@t-online.de>
    Addresses: https://cygwin.com/pipermail/cygwin/2025-June/258358.html
    Fixes: 28186e81d947 ("* libc/ctype/iswalpha.c: Handle all wchar_t as unicode on _MB_CAPABLE systems.")
    Signed-off-by: Corinna Vinschen <corinna@vinschen.de>
    (cherry picked from commit b374973d14ac7969b10ba719feedc709f6971c0d)

Diff:
---
 newlib/libc/stdlib/mbtowc_r.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/newlib/libc/stdlib/mbtowc_r.c b/newlib/libc/stdlib/mbtowc_r.c
index cab8333d70df..6c3bd3d2676e 100644
--- a/newlib/libc/stdlib/mbtowc_r.c
+++ b/newlib/libc/stdlib/mbtowc_r.c
@@ -677,6 +677,21 @@ __utf8_mbtowc (struct _reent *r,
 	state->__count = 3;
       else if (n < (size_t)-1)
 	++n;
+      if (n < 4)
+	return -2;
+      ch = t[i++];
+      if (ch < 0x80 || ch > 0xbf)
+	{
+	  _REENT_ERRNO(r) = EILSEQ;
+	  return -1;
+	}
+      /* Note: Originally we created the low surrogate pair on systems with
+	 wchar_t == UTF-16 *before* checking the 4th byte.  This was utterly
+	 wrong, because this failed to check the last byte for being a valid
+	 value for a complete UTF-8 4 byte sequence.  As a result, calling
+	 functions happily digested the low surrogate and then got an entirely
+	 different character and handled this separately, thus generating
+	 invalid UTF-16 values. */
       if (state->__count == 3 && sizeof(wchar_t) == 2)
 	{
 	  /* On systems which have wchar_t being UTF-16 values, the value
@@ -695,15 +710,7 @@ __utf8_mbtowc (struct _reent *r,
 	    |   (wint_t)((state->__value.__wchb[2] & 0x3f) << 6);
 	  state->__count = 4;
 	  *pwc = 0xd800 | ((tmp - 0x10000) >> 10);
-	  return i;
-	}
-      if (n < 4)
-	return -2;
-      ch = t[i++];
-      if (ch < 0x80 || ch > 0xbf)
-	{
-	  _REENT_ERRNO(r) = EILSEQ;
-	  return -1;
+	  return 3;
 	}
       tmp = (wint_t)((state->__value.__wchb[0] & 0x07) << 18)
 	|   (wint_t)((state->__value.__wchb[1] & 0x3f) << 12)