Summary: | mbrtowc with Big5-HKSCS fails to reset conversion state for conversions that produce two Unicode code points | ||
---|---|---|---|
Product: | glibc | Reporter: | Tom Honermann <tom> |
Component: | locale | Assignee: | Not yet assigned to anyone <unassigned> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | carlos, fw, schwab |
Priority: | P2 | Flags: | fw:
security-
|
Version: | 2.31 | ||
Target Milestone: | 2.32 | ||
Host: | Target: | ||
Build: | Last reconfirmed: | 2020-03-27 00:00:00 |
Description
Tom Honermann
2020-03-27 04:05:02 UTC
Andreas Schwab probably has the most experience with Big5-HKSCS. I'm CC'ing him on this this issue in case he has any thoughts about this. I can confirm that /x88/x62 should map to <U00CA><U0304>. Double checked that against Big5-HKSCS 2016 (which we don't support yet): https://www.ogcio.gov.hk/en/our_work/business/tech_promotion/ccli/terms/doc/e_hkscs_2016.pdf I can confirm the test case behaves as you indicated. In glibc/iconvdata/big5hkscs.c we certainly handle the combining characters: 17843 /* Check for special cases: combining characters. */ \ 17844 if (idx == 195 + 0x22 /* 8862 */) \ 17845 { \ 17846 ch = 0xca; \ 17847 ch2 = 0x304; \ 17848 } \ 17849 else if (idx == 195 + 0x24 /* 8864 */) \ 17850 { \ 17851 ch = 0xca; \ 17852 ch2 = 0x30c; \ 17853 } \ 17854 else if (idx == 195 + 0x63 /* 88a3 */) \ 17855 { \ 17856 ch = 0xea; \ 17857 ch2 = 0x304; \ 17858 } \ 17859 else if (idx == 195 + 0x65 /* 88a5 */) \ 17860 { \ 17861 ch = 0xea; \ 17862 ch2 = 0x30c; \ 17863 } \ 17864 else \ 17865 /* This is illegal. */ \ 17866 STANDARD_FROM_LOOP_ERR_HANDLER (1); \ I haven't debugged this in great detail though. diff --git a/iconvdata/big5hkscs.c b/iconvdata/big5hkscs.c index 01fcfeba76..70f84a5226 100644 --- a/iconvdata/big5hkscs.c +++ b/iconvdata/big5hkscs.c @@ -17898,6 +17898,7 @@ static struct \ put32 (outptr, ch); \ outptr += 4; \ + *statep = 0; \ } #define LOOP_NEED_FLAGS #define EXTRA_LOOP_DECLS , int *statep If we have a pushed back character we should clear statep. I expect we need a test case to cover the case you outline. The quick fix appears to work. GCONV_PATH=/home/carlos/build/glibc/iconvdata ./test 1st wc: 0x00CA 2nd wc: 0x0304 3rd wc: 0x0058 (should be 0x0058) result: 1 (should be 1) I need to dig into this a bit more. Thanks for confirming, Carlos. I had independently arrived at the same quick fix, but you beat me to posting it. I don't know glibc internals well. Something I've been surprised by is that iconv doesn't seem to suffer this same issue. I haven't tried to debug to determine why it is unaffected. (In reply to Carlos O'Donell from comment #1) > I expect we need a test case to cover the case you outline. Agreed. The patch below is what I had added locally as a test. Feel free to use this (I've already signed a glibc contributor form). diff --git a/wcsmbs/Makefile b/wcsmbs/Makefile index f02167fa58..2e58ac36e8 100644 --- a/wcsmbs/Makefile +++ b/wcsmbs/Makefile @@ -58,7 +58,8 @@ include ../Rules ifeq ($(run-built-tests),yes) LOCALES := de_DE.ISO-8859-1 de_DE.UTF-8 en_US.ANSI_X3.4-1968 hr_HR.ISO-8859-2 \ - ja_JP.EUC-JP zh_TW.EUC-TW tr_TR.UTF-8 tr_TR.ISO-8859-9 + ja_JP.EUC-JP zh_TW.EUC-TW tr_TR.UTF-8 tr_TR.ISO-8859-9 \ + zh_HK.BIG5-HKSCS include ../gen-locales.mk $(objpfx)tst-btowc.out: $(gen-locales) diff --git a/wcsmbs/tst-mbrtowc.c b/wcsmbs/tst-mbrtowc.c index 38a431983b..b2b3ff3744 100644 --- a/wcsmbs/tst-mbrtowc.c +++ b/wcsmbs/tst-mbrtowc.c @@ -150,6 +150,43 @@ utf8_test (void) return error; } +static int +big5_hkscs_test (void) +{ + const char *locale = "zh_HK.BIG5-HKSCS"; + const char *mbs; + wchar_t wc; + mbstate_t s; + + if (!setlocale (LC_CTYPE, locale)) + { + fprintf (stderr, "locale '%s' not available!\n", locale); + exit (1); + } + + /* This test exercises a special case conversion of a double byte sequence + to a pair of Unicode code points. */ + wc = 42; /* arbitrary number */ + mbs = "\x88\x62"; /* 0x88 0x62 => U+00CA U+0304 */ + memset (&s, 0, sizeof(s)); + /* Translate the double byte sequence. This call consumes the first two + bytes and writes the first of the two code points. */ + assert (mbrtowc (&wc, mbs, 3, &s) == (size_t) 2); + assert (wc == 0x00CA); + /* Attempt to translate the next sequence. mbrtowc isn't specified to + return (size_t) -3 like mbrtoc16 is in the case where there are multiple + wide chars to write. This call returns 0 despite not writing a null + character. 0 is returned because no new input is consumed; the call + writes the second code point. */ + assert (mbrtowc (&wc, mbs + 2, 1, &s) == (size_t) 0); + assert (wc == 0x0304); + /* The following now consumes and converts the null character. */ + assert (mbrtowc (&wc, mbs + 2, 1, &s) == (size_t) 0); + assert(wc == 0); + + return 0; +} + static int do_test (void) @@ -169,6 +206,8 @@ do_test (void) setlocale (LC_ALL, "ja_JP.EUC-JP"); result |= check_ascii (setlocale (LC_ALL, NULL)); + result |= big5_hkscs_test (); + return result; } Tom, I split the test out specifically for BIG5HKSCS and included all 4 conversions that create 2 wchar_ts. Posted: https://sourceware.org/pipermail/libc-alpha/2020-March/112257.html Fixed in 2.32 Verified fixed for 2.32 as of commit c580e6466d6da8262820cdbad19f32c5546226cf (https://sourceware.org/git/?p=glibc.git;a=commit;h=c580e6466d6da8262820cdbad19f32c5546226cf) Please leave the status as RESOLVED. There is some automation that depends on it. |