[RFC] iconvdata/, localedata/: Fix TSCII and document tests.

Carlos O'Donell carlos@redhat.com
Wed Jun 24 17:01:23 GMT 2020


On 6/24/20 10:24 AM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> We do not support what is written in the TSCII character map today,
>> and the character map is completely broken and will not compile.
> 
> But will a compiled TSCII locale still work, even if the charmap file is
> not parsed correctly by localdef?  As long as the locale definition does
> reference any of the problematic characters?

Yes, a TSCII locale, say C.TSCII will still work, but you get errors
(though those errors don't actually get propagated because we use error_at_line
for linereader errors instead of record_error_at_line which propagates these
errors, and that's a bug in linereader.h lr_error()).

Such a locale compilation generates the following warnings:

$ localedef -f TSCII -i ./localedata/locales/C /home/carlos/tmp/C.TSCII
/usr/share/i18n/charmaps/TSCII:139: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:141: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:142: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:144: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:145: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:148: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:149: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:150: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:151: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:152: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:153: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:154: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:155: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:156: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:157: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:170: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:171: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:172: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:173: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:183: too many bytes in character encoding
/usr/share/i18n/charmaps/TSCII:184: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:185: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:186: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:187: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:188: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:189: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:190: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:191: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:192: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:193: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:194: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:195: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:196: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:197: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:198: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:199: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:200: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:201: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:202: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:203: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:204: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:205: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:206: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:207: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:208: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:209: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:210: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:211: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:212: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:213: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:214: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:215: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:216: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:217: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:218: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:219: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:221: too many bytes in character encoding
/usr/share/i18n/charmaps/TSCII:222: too many bytes in character encoding
/usr/share/i18n/charmaps/TSCII:223: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:224: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:225: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:226: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:227: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:228: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:229: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:230: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:231: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:232: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:233: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:234: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:235: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:236: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:237: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:238: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:239: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:240: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:241: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:242: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:243: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:244: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:245: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:246: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:247: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:248: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:249: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:250: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:251: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:252: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:253: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:254: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:255: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:256: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:257: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:258: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:259: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:260: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:261: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:262: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:263: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:264: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:265: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:266: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:267: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:268: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:269: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:270: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:271: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:272: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:273: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:274: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:275: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:276: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:278: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:279: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:280: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:281: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:282: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:283: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:284: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:285: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:286: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:287: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:288: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:289: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:290: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:291: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:292: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:293: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:294: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:295: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:329: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:330: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:331: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:332: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:333: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:334: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:335: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:336: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:337: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:338: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:339: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:340: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:341: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:342: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:343: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:344: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:345: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:346: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:347: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:348: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:349: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:350: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:351: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:352: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:353: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:354: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:355: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:356: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:357: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:358: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:359: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:360: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:361: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:362: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:363: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:364: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:365: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:366: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:367: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:368: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:369: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:370: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:371: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:372: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:373: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:374: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:375: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:376: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:377: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:378: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:379: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:380: syntax error in CHARMAP definition: invalid encoding given
/usr/share/i18n/charmaps/TSCII:385: unknown character `U00000B82'
/usr/share/i18n/charmaps/TSCII:387: unknown character `U00000BCD'

My patch fixes 179 invalid entries.

Above you can see 184 errors, 2 of which are from invalid WIDTH specifiers (the two at the end),
and 3 are from invalid encodings (mb_cur_max is only 1, and should be 3), leaving 179 invalid entries
all of which are now fixed.

The charmap definition is erroneous and those entries are being ignored, but since the C locale
doesn't use any of those entries it is overall "safe" from that perspective.

Note: The unknown character come from WIDTH entries for characters not defined in the charmap.

Note: There *should* be "too many bytes in character encoding" errors for many more entries, but
      the charmap.c processing is restricted to those entries that successfully parsed (don't
      have multiple 'from' code points which are invalid).

With the fixed TSCII we get:
$ localedef -f /home/carlos/src/glibc/localedata/charmaps/TSCII -i ./localedata/locales/C /home/carlos/tmp/C.TSCII
$ echo $?
0

We've fixed all the lines with invalid format and instead turned them into comments regarding
what the charmap *should* have.

Compiling C.TSCII before and after the TSCII changes yields identical binary locales with the
exception of LC_CTYPE which is likely different because maybe we pikcup an extra entry via
i18n_ctype's extended list of LC_CTYPE information. I haven't reviewed the exact details.

Does that answer your question?

-- 
Cheers,
Carlos.



More information about the Libc-alpha mailing list