localedata linting revised again

[Zack Weinberg]

I've revised my localedata linter to use iconv instead of python's
built-in codecs, and to only complain about strings being
unrepresentable if transliteration doesn't help.

All of the remaining complaints are about strings that aren't NFC
(full list at bottom of this message).  Most, but not all, of these
appear to be LC_COLLATE specifications for decomposed accented
characters, which I would have expected to be handled generically for
all languages (if there is a canonical equivalence between two
codepoint sequences, then it seems intuitively obvious to me that they
should always be treated the same for collation, perhaps with the
actual code points used as a tiebreaker).  But given the contents of
the various files, apparently it isn't, and I think that's a bug.

zw

---

localedata/locales/as_IN:120: string not normalized:
  source: 09B9 09DF
     nfc: 09B9 09AF 09BC
localedata/locales/as_IN:121: string not normalized:
  source: 09A8 09B9 09DF
     nfc: 09A8 09B9 09AF 09BC
localedata/locales/da_DK:144: string not normalized:
  source: 0041 0308
     nfc: 00C4
localedata/locales/da_DK:146: string not normalized:
  source: 0041 030A
     nfc: 00C5
localedata/locales/da_DK:150: string not normalized:
  source: 004F 0308
     nfc: 00D6
localedata/locales/da_DK:154: string not normalized:
  source: 0061 0308
     nfc: 00E4
localedata/locales/da_DK:156: string not normalized:
  source: 0061 030A
     nfc: 00E5
localedata/locales/da_DK:160: string not normalized:
  source: 006F 0308
     nfc: 00F6
localedata/locales/de_DE:50: string not normalized:
  source: 0041 0308
     nfc: 00C4
localedata/locales/de_DE:52: string not normalized:
  source: 004F 0308
     nfc: 00D6
localedata/locales/de_DE:54: string not normalized:
  source: 0055 0308
     nfc: 00DC
localedata/locales/de_DE:56: string not normalized:
  source: 0061 0308
     nfc: 00E4
localedata/locales/de_DE:58: string not normalized:
  source: 006F 0308
     nfc: 00F6
localedata/locales/de_DE:60: string not normalized:
  source: 0075 0308
     nfc: 00FC
localedata/locales/de_DE:64: string not normalized:
  source: 0041 030A
     nfc: 00C5
localedata/locales/de_DE:66: string not normalized:
  source: 0061 030A
     nfc: 00E5
localedata/locales/fa_IR:106: string not normalized:
  source: 0627 0653
     nfc: 0622
localedata/locales/fa_IR:107: string not normalized:
  source: 0627 0654
     nfc: 0623
localedata/locales/fa_IR:108: string not normalized:
  source: 0627 0655
     nfc: 0625
localedata/locales/fa_IR:109: string not normalized:
  source: 0648 0654
     nfc: 0624
localedata/locales/fa_IR:111: string not normalized:
  source: 064A 0654
     nfc: 0626
localedata/locales/hu_HU:460: string not normalized:
  source: 0041 0301
     nfc: 00C1
localedata/locales/hu_HU:461: string not normalized:
  source: 0045 0301
     nfc: 00C9
localedata/locales/hu_HU:462: string not normalized:
  source: 0049 0301
     nfc: 00CD
localedata/locales/hu_HU:463: string not normalized:
  source: 004F 0301
     nfc: 00D3
localedata/locales/hu_HU:464: string not normalized:
  source: 004F 0308
     nfc: 00D6
localedata/locales/hu_HU:465: string not normalized:
  source: 004F 030B
     nfc: 0150
localedata/locales/hu_HU:466: string not normalized:
  source: 0055 0301
     nfc: 00DA
localedata/locales/hu_HU:467: string not normalized:
  source: 0055 0308
     nfc: 00DC
localedata/locales/hu_HU:468: string not normalized:
  source: 0055 030B
     nfc: 0170
localedata/locales/hu_HU:470: string not normalized:
  source: 0061 0301
     nfc: 00E1
localedata/locales/hu_HU:471: string not normalized:
  source: 0065 0301
     nfc: 00E9
localedata/locales/hu_HU:472: string not normalized:
  source: 0069 0301
     nfc: 00ED
localedata/locales/hu_HU:473: string not normalized:
  source: 006F 0301
     nfc: 00F3
localedata/locales/hu_HU:474: string not normalized:
  source: 006F 0308
     nfc: 00F6
localedata/locales/hu_HU:475: string not normalized:
  source: 006F 030B
     nfc: 0151
localedata/locales/hu_HU:476: string not normalized:
  source: 0075 0301
     nfc: 00FA
localedata/locales/hu_HU:477: string not normalized:
  source: 0075 0308
     nfc: 00FC
localedata/locales/hu_HU:478: string not normalized:
  source: 0075 030B
     nfc: 0171
localedata/locales/ig_NG:103: string not normalized:
  source: 0049 0323
     nfc: 1ECA
localedata/locales/ig_NG:104: string not normalized:
  source: 0069 0323
     nfc: 1ECB
localedata/locales/ig_NG:105: string not normalized:
  source: 0049 0323 0301
     nfc: 1ECA 0301
localedata/locales/ig_NG:106: string not normalized:
  source: 0069 0323 0301
     nfc: 1ECB 0301
localedata/locales/ig_NG:107: string not normalized:
  source: 0049 0323 0300
     nfc: 1ECA 0300
localedata/locales/ig_NG:108: string not normalized:
  source: 0069 0323 0300
     nfc: 1ECB 0300
localedata/locales/ig_NG:114: string not normalized:
  source: 004F 0323
     nfc: 1ECC
localedata/locales/ig_NG:115: string not normalized:
  source: 006F 0323
     nfc: 1ECD
localedata/locales/ig_NG:116: string not normalized:
  source: 004F 0323 0301
     nfc: 1ECC 0301
localedata/locales/ig_NG:117: string not normalized:
  source: 006F 0323 0301
     nfc: 1ECD 0301
localedata/locales/ig_NG:118: string not normalized:
  source: 004F 0323 0300
     nfc: 1ECC 0300
localedata/locales/ig_NG:119: string not normalized:
  source: 006F 0323 0300
     nfc: 1ECD 0300
localedata/locales/ig_NG:130: string not normalized:
  source: 0055 0323
     nfc: 1EE4
localedata/locales/ig_NG:131: string not normalized:
  source: 0075 0323
     nfc: 1EE5
localedata/locales/ig_NG:132: string not normalized:
  source: 0055 0323 0301
     nfc: 1EE4 0301
localedata/locales/ig_NG:133: string not normalized:
  source: 0075 0323 0301
     nfc: 1EE5 0301
localedata/locales/ig_NG:134: string not normalized:
  source: 0075 0323 0300
     nfc: 1EE5 0300
localedata/locales/ig_NG:135: string not normalized:
  source: 0055 0323 0300
     nfc: 1EE4 0300
localedata/locales/ig_NG:141: string not normalized:
  source: 004E 0307
     nfc: 1E44
localedata/locales/ig_NG:142: string not normalized:
  source: 006E 0307
     nfc: 1E45
localedata/locales/ig_NG:144: string not normalized:
  source: 0041 0301
     nfc: 00C1
localedata/locales/ig_NG:145: string not normalized:
  source: 0061 0301
     nfc: 00E1
localedata/locales/ig_NG:147: string not normalized:
  source: 0045 0301
     nfc: 00C9
localedata/locales/ig_NG:148: string not normalized:
  source: 0065 0301
     nfc: 00E9
localedata/locales/ig_NG:150: string not normalized:
  source: 0049 0301
     nfc: 00CD
localedata/locales/ig_NG:151: string not normalized:
  source: 0069 0301
     nfc: 00ED
localedata/locales/ig_NG:153: string not normalized:
  source: 004F 0301
     nfc: 00D3
localedata/locales/ig_NG:154: string not normalized:
  source: 006F 0301
     nfc: 00F3
localedata/locales/ig_NG:156: string not normalized:
  source: 0055 0301
     nfc: 00DA
localedata/locales/ig_NG:157: string not normalized:
  source: 0075 0301
     nfc: 00FA
localedata/locales/ig_NG:159: string not normalized:
  source: 0041 0300
     nfc: 00C0
localedata/locales/ig_NG:160: string not normalized:
  source: 0061 0300
     nfc: 00E0
localedata/locales/ig_NG:162: string not normalized:
  source: 0045 0300
     nfc: 00C8
localedata/locales/ig_NG:163: string not normalized:
  source: 0065 0300
     nfc: 00E8
localedata/locales/ig_NG:165: string not normalized:
  source: 0049 0300
     nfc: 00CC
localedata/locales/ig_NG:166: string not normalized:
  source: 0069 0300
     nfc: 00EC
localedata/locales/ig_NG:168: string not normalized:
  source: 004F 0300
     nfc: 00D2
localedata/locales/ig_NG:169: string not normalized:
  source: 006F 0300
     nfc: 00F2
localedata/locales/ig_NG:171: string not normalized:
  source: 0055 0300
     nfc: 00D9
localedata/locales/ig_NG:172: string not normalized:
  source: 0075 0300
     nfc: 00F9
localedata/locales/ik_CA:77: string not normalized:
  source: 004C 0323
     nfc: 1E36
localedata/locales/ik_CA:78: string not normalized:
  source: 006C 0323
     nfc: 1E37
localedata/locales/lb_LU:50: string not normalized:
  source: 0041 0308
     nfc: 00C4
localedata/locales/lb_LU:52: string not normalized:
  source: 004F 0308
     nfc: 00D6
localedata/locales/lb_LU:54: string not normalized:
  source: 0055 0308
     nfc: 00DC
localedata/locales/lb_LU:56: string not normalized:
  source: 0045 0308
     nfc: 00CB
localedata/locales/lb_LU:58: string not normalized:
  source: 0061 0308
     nfc: 00E4
localedata/locales/lb_LU:60: string not normalized:
  source: 006F 0308
     nfc: 00F6
localedata/locales/lb_LU:62: string not normalized:
  source: 0075 0308
     nfc: 00FC
localedata/locales/lb_LU:64: string not normalized:
  source: 0065 0308
     nfc: 00EB
localedata/locales/mni_IN:67: string not normalized:
  source: 09DF 09C1 09AE
     nfc: 09AF 09BC 09C1 09AE
localedata/locales/mni_IN:76: string not normalized:
  source: 09DF 09C1 09AE 09B6 0995 09C8 09B6 09BE
     nfc: 09AF 09BC 09C1 09AE 09B6 0995 09C8 09B6 09BE
localedata/locales/nb_NO:137: string not normalized:
  source: 0041 0308
     nfc: 00C4
localedata/locales/nb_NO:139: string not normalized:
  source: 0041 030A
     nfc: 00C5
localedata/locales/nb_NO:143: string not normalized:
  source: 004F 0308
     nfc: 00D6
localedata/locales/nb_NO:147: string not normalized:
  source: 0061 0308
     nfc: 00E4
localedata/locales/nb_NO:149: string not normalized:
  source: 0061 030A
     nfc: 00E5
localedata/locales/nb_NO:153: string not normalized:
  source: 006F 0308
     nfc: 00F6
localedata/locales/pa_IN:95: string not normalized:
  source: 0A36 0A41 0A71 0A15 0A30
     nfc: 0A38 0A3C 0A41 0A71 0A15 0A30
localedata/locales/pa_IN:96: string not normalized:
  source: 0A36 0A28 0A3F 0A71 0A1A 0A30
     nfc: 0A38 0A3C 0A28 0A3F 0A71 0A1A 0A30
localedata/locales/pa_IN:104: string not normalized:
  source: 0A36 0A41 0A71 0A15 0A30 0A35 0A3E 0A30
     nfc: 0A38 0A3C 0A41 0A71 0A15 0A30 0A35 0A3E 0A30
localedata/locales/pa_IN:105: string not normalized:
  source: 0A36 0A28 0A3F 0A71 0A1A 0A30 0A35 0A3E 0A30
     nfc: 0A38 0A3C 0A28 0A3F 0A71 0A1A 0A30 0A35 0A3E 0A30
localedata/locales/pa_IN:139: string not normalized:
  source: 0A36 0A3E 0A2E
     nfc: 0A38 0A3C 0A3E 0A2E
localedata/locales/sgs_LT:85: string not normalized:
  source: 0070 0065 0074 006E 0069 0304 010D 0117
     nfc: 0070 0065 0074 006E 012B 010D 0117
localedata/locales/sgs_LT:117: string not normalized:
  source: 0074 0227 0304 0070
     nfc: 0074 01E1 0070
localedata/locales/sgs_LT:118: string not normalized:
  source: 006E 0065 0304
     nfc: 006E 0113
localedata/locales/sv_SE:121: string not normalized:
  source: 0041 0308
     nfc: 00C4
localedata/locales/sv_SE:123: string not normalized:
  source: 0041 030A
     nfc: 00C5
localedata/locales/sv_SE:127: string not normalized:
  source: 004F 0308
     nfc: 00D6
localedata/locales/sv_SE:131: string not normalized:
  source: 0061 0308
     nfc: 00E4
localedata/locales/sv_SE:133: string not normalized:
  source: 0061 030A
     nfc: 00E5
localedata/locales/sv_SE:137: string not normalized:
  source: 006F 0308
     nfc: 00F6
localedata/locales/wa_BE:67: string not normalized:
  source: 0041 030A
     nfc: 00C5
localedata/locales/wa_BE:68: string not normalized:
  source: 0061 030A
     nfc: 00E5
localedata/locales/yo_NG:79: string not normalized:
  source: 0045 0323
     nfc: 1EB8
localedata/locales/yo_NG:80: string not normalized:
  source: 0065 0323
     nfc: 1EB9
localedata/locales/yo_NG:85: string not normalized:
  source: 0045 0323 0301
     nfc: 1EB8 0301
localedata/locales/yo_NG:86: string not normalized:
  source: 0065 0323 0301
     nfc: 1EB9 0301
localedata/locales/yo_NG:87: string not normalized:
  source: 0045 0323 0300
     nfc: 1EB8 0300
localedata/locales/yo_NG:88: string not normalized:
  source: 0065 0323 0300
     nfc: 1EB9 0300
localedata/locales/yo_NG:90: string not normalized:
  source: 004F 0323
     nfc: 1ECC
localedata/locales/yo_NG:91: string not normalized:
  source: 006F 0323
     nfc: 1ECD
localedata/locales/yo_NG:96: string not normalized:
  source: 004F 0323 0301
     nfc: 1ECC 0301
localedata/locales/yo_NG:97: string not normalized:
  source: 006F 0323 0301
     nfc: 1ECD 0301
localedata/locales/yo_NG:98: string not normalized:
  source: 004F 0323 0300
     nfc: 1ECC 0300
localedata/locales/yo_NG:99: string not normalized:
  source: 006F 0323 0300
     nfc: 1ECD 0300
localedata/locales/yo_NG:101: string not normalized:
  source: 0053 0323
     nfc: 1E62
localedata/locales/yo_NG:102: string not normalized:
  source: 0073 0323
     nfc: 1E63
localedata/locales/yo_NG:104: string not normalized:
  source: 0041 0301
     nfc: 00C1
localedata/locales/yo_NG:105: string not normalized:
  source: 0061 0301
     nfc: 00E1
localedata/locales/yo_NG:106: string not normalized:
  source: 0045 0301
     nfc: 00C9
localedata/locales/yo_NG:107: string not normalized:
  source: 0065 0301
     nfc: 00E9
localedata/locales/yo_NG:108: string not normalized:
  source: 0049 0301
     nfc: 00CD
localedata/locales/yo_NG:109: string not normalized:
  source: 0069 0301
     nfc: 00ED
localedata/locales/yo_NG:110: string not normalized:
  source: 004D 0301
     nfc: 1E3E
localedata/locales/yo_NG:111: string not normalized:
  source: 006D 0301
     nfc: 1E3F
localedata/locales/yo_NG:112: string not normalized:
  source: 004E 0301
     nfc: 0143
localedata/locales/yo_NG:113: string not normalized:
  source: 006E 0301
     nfc: 0144
localedata/locales/yo_NG:114: string not normalized:
  source: 004F 0301
     nfc: 00D3
localedata/locales/yo_NG:115: string not normalized:
  source: 006F 0301
     nfc: 00F3
localedata/locales/yo_NG:116: string not normalized:
  source: 0055 0301
     nfc: 00DA
localedata/locales/yo_NG:117: string not normalized:
  source: 0075 0301
     nfc: 00FA
localedata/locales/yo_NG:119: string not normalized:
  source: 0041 0300
     nfc: 00C0
localedata/locales/yo_NG:120: string not normalized:
  source: 0061 0300
     nfc: 00E0
localedata/locales/yo_NG:121: string not normalized:
  source: 0045 0300
     nfc: 00C8
localedata/locales/yo_NG:122: string not normalized:
  source: 0065 0300
     nfc: 00E8
localedata/locales/yo_NG:123: string not normalized:
  source: 0049 0300
     nfc: 00CC
localedata/locales/yo_NG:124: string not normalized:
  source: 0069 0300
     nfc: 00EC
localedata/locales/yo_NG:127: string not normalized:
  source: 004E 0300
     nfc: 01F8
localedata/locales/yo_NG:128: string not normalized:
  source: 006E 0300
     nfc: 01F9
localedata/locales/yo_NG:129: string not normalized:
  source: 004F 0300
     nfc: 00D2
localedata/locales/yo_NG:130: string not normalized:
  source: 006F 0300
     nfc: 00F2
localedata/locales/yo_NG:131: string not normalized:
  source: 0055 0300
     nfc: 00D9
localedata/locales/yo_NG:132: string not normalized:
  source: 0075 0300
     nfc: 00F9
localedata/locales/yo_NG:139: string not normalized:
  source: 0041 0302
     nfc: 00C2
localedata/locales/yo_NG:140: string not normalized:
  source: 0061 0302
     nfc: 00E2
localedata/locales/yo_NG:141: string not normalized:
  source: 0045 0302
     nfc: 00CA
localedata/locales/yo_NG:142: string not normalized:
  source: 0065 0302
     nfc: 00EA
localedata/locales/yo_NG:143: string not normalized:
  source: 1EB8 0302
     nfc: 1EC6
localedata/locales/yo_NG:144: string not normalized:
  source: 1EB9 0302
     nfc: 1EC7
localedata/locales/yo_NG:145: string not normalized:
  source: 0049 0302
     nfc: 00CE
localedata/locales/yo_NG:146: string not normalized:
  source: 0069 0302
     nfc: 00EE
localedata/locales/yo_NG:147: string not normalized:
  source: 004F 0302
     nfc: 00D4
localedata/locales/yo_NG:148: string not normalized:
  source: 006F 0302
     nfc: 00F4
localedata/locales/yo_NG:149: string not normalized:
  source: 1ECC 0302
     nfc: 1ED8
localedata/locales/yo_NG:150: string not normalized:
  source: 1ECD 0302
     nfc: 1ED9
localedata/locales/yo_NG:151: string not normalized:
  source: 0055 0302
     nfc: 00DB
localedata/locales/yo_NG:152: string not normalized:
  source: 0075 0302
     nfc: 00FB
localedata/locales/yo_NG:154: string not normalized:
  source: 0041 030C
     nfc: 01CD
localedata/locales/yo_NG:155: string not normalized:
  source: 0061 030C
     nfc: 01CE
localedata/locales/yo_NG:156: string not normalized:
  source: 0045 030C
     nfc: 011A
localedata/locales/yo_NG:157: string not normalized:
  source: 0065 030C
     nfc: 011B
localedata/locales/yo_NG:160: string not normalized:
  source: 0049 030C
     nfc: 01CF
localedata/locales/yo_NG:161: string not normalized:
  source: 0069 030C
     nfc: 01D0
localedata/locales/yo_NG:162: string not normalized:
  source: 004F 030C
     nfc: 01D1
localedata/locales/yo_NG:163: string not normalized:
  source: 006F 030C
     nfc: 01D2
localedata/locales/yo_NG:166: string not normalized:
  source: 0055 030C
     nfc: 01D3
localedata/locales/yo_NG:167: string not normalized:
  source: 0075 030C
     nfc: 01D4
-------------- next part --------------
A non-text attachment was scrubbed...
Name: check-localedef.py
Type: text/x-python
Size: 27700 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20170828/15e0a294/attachment.py>