| Summary: | Tibetan script collation broken (Dzongkha and Tibetan) | ||
|---|---|---|---|
| Product: | glibc | Reporter: | Elie Roux <elie.roux> |
| Component: | localedata | Assignee: | Mike FABIAN <maiku.fabian> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | elie.roux, jeremip11, libc-locales, maiku.fabian |
| Priority: | P2 | Flags: | fweimer:
security-
|
| Version: | 2.24 | ||
| Target Milestone: | 2.28 | ||
| Host: | Target: | ||
| Build: | Last reconfirmed: | ||
| Attachments: |
sorted list for test
icu-collation-test.py glibc/locales/iso14651_t1_common glib/localedata/locales/dz_BT |
||
|
Description
Elie Roux
2017-06-05 10:32:15 UTC
Do you have a complete test file which tests all collation rules for Tibetan? Are the current rules in CLDR correct or not? https://unicode.org/cldr/trac/browser/trunk/common/collation/dz.xml Your ticket http://unicode.org/cldr/trac/ticket/9895 seems to be an update for the rules currently in CLDR, but your update is not yet included, right? Created attachment 10696 [details] sorted list for test This is a correctly sorted test file, made using the latest rules of https://github.com/eroux/tibetan-collation Thanks for your answer! You're right, my proposal to update the CLDR rules hasn't been considered yet. I don't know how much time these things usually take, but if you have any advice on speeding up the process I'd be glad to hear them! I've attached a sorted list of strings, it sorts correctly with the latest rules of https://github.com/eroux/tibetan-collation/ Thank you! I wonder whether there isn’t a contradicton in your rules. https://github.com/eroux/tibetan-collation/blob/master/implementations/Unicode/rules.txt#L7 contains: &གཉ<གཉྫ so གཉ comes *before* གཉྫ as a primary difference. But then https://github.com/eroux/tibetan-collation/blob/master/implementations/Unicode/rules.txt#L30 contains: &ཉ<<ྋྙ<གཉ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ And this causes གཉ to be sorted *after* གཉྫ. (I tested this with icu 57.1 using it via Python3. Hello Fabian, Thanks a lot for your thorough review, that's appreciated! I have to say I don't really understand the second part, why would line 30 causes གཉ to be sorted after གཉྫ ? can you elaborate a little bit? Anyways, a solution I suppose is to remove line 7 and have line 30 be: &ཉ<<ྋྙ<གཉ<གཉྫ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ would that solve the problem? I can elaborate a bit on why lines 4 to 18 are separated from the rest if you want, but I think it will be helpful to understand your point of view a bit better first. Hello Fabian, Thanks a lot for your thorough review, that's appreciated! I have to say I don't really understand the second part, why would line 30 causes གཉ to be sorted after གཉྫ ? can you elaborate a little bit? Anyways, a solution I suppose is to remove line 7 and have line 30 be: &ཉ<<ྋྙ<གཉ<གཉྫ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ would that solve the problem? I can elaborate a bit on why lines 4 to 18 are separated from the rest if you want, but I think it will be helpful to understand your point of view a bit better first. (In reply to Elie Roux from comment #6) > Hello Fabian, > > Thanks a lot for your thorough review, that's appreciated! > > I have to say I don't really understand the second part, why would line 30 > causes གཉ to be sorted after གཉྫ ? can you elaborate a little bit? I am not sure why this happens either. But it seems to happen. I tested like this: First the input test file to be sorted, made very short to contain only the strings in question: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ cat localedata/dz_BT.UTF-8.in.mini གཉ གཉྫ mfabian@taka:/local/mfabian/src/glibc (locales *$%) And I use a very short rule file, first containg only &གཉ<གཉྫ and the other rule commented out: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ cat rules-mini.txt &གཉ<གཉྫ #&ཉ<<ྋྙ<གཉ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ mfabian@taka:/local/mfabian/src/glibc (locales *$%) Now I sort using my small test program ~/bin/icu-collation-test.py (I’ll attach it in the next comment): mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ ~/bin/icu-collation-test.py -r rules-mini.txt -i localedata/dz_BT.UTF-8.in.mini -o /tmp/dz_BT.UTF-8.out mfabian@taka:/local/mfabian/src/glibc (locales *$%) And check the result: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ diff -u /local/mfabian/src/glibc/localedata/dz_BT.UTF-8.in.mini /tmp/dz_BT.UTF-8.out mfabian@taka:/local/mfabian/src/glibc (locales *$%) No difference between input and output, i.e. གཉ is still before གཉྫ in dz_BT.UTF-8.out. Now I remove the comment in front of the second line in rules-mini.txt: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ cat rules-mini.txt &གཉ<གཉྫ &ཉ<<ྋྙ<གཉ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ mfabian@taka:/local/mfabian/src/glibc (locales *$%) And sort again: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ ~/bin/icu-collation-test.py -r rules-mini.txt -i localedata/dz_BT.UTF-8.in.mini -o /tmp/dz_BT.UTF-8.out mfabian@taka:/local/mfabian/src/glibc (locales *$%) Checking the result: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ diff -u /local/mfabian/src/glibc/localedata/dz_BT.UTF-8.in.mini /tmp/dz_BT.UTF-8.out --- /local/mfabian/src/glibc/localedata/dz_BT.UTF-8.in.mini 2018-01-15 15:21:59.357477414 +0100 +++ /tmp/dz_BT.UTF-8.out 2018-01-15 15:26:12.266632745 +0100 @@ -1,2 +1,2 @@ -གཉ གཉྫ +གཉ mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ Now the order is reversed, གཉ comes after གཉྫ. The same happened to me while I was implementing the rules for glibc and test sorting using glibc. I found this very confusing and thought I might have done something wrong implementing the rules in the glibc way. But then I tested with the above small Python3 program using icu and found that it behaves the same way. Created attachment 10749 [details]
icu-collation-test.py
A simple tool to test ICU/CLDR style collation rules.
I have to say I don't really understand why ICU behaves like that... I think we should do two things: - change my rule file so that it contains just one line and fix this oddity - report a bug on ICU (maybe it's not a bug per se, but I can't see any other way to solve this mistery) I'll fix the rule file (possibly today). If you have some time do you think you could report the ICU bug? Thank you! (In reply to Elie Roux from comment #9) > I have to say I don't really understand why ICU behaves like that... I think > we should do two things: > > - change my rule file so that it contains just one line and fix this oddity > - report a bug on ICU (maybe it's not a bug per se, but I can't see any > other way to solve this mistery) The glibc rules I implemented showed the same behaviour, so I doubt this is an ICU bug, I think it is a bug in the rules. > I'll fix the rule file (possibly today). If you have some time do you think > you could report the ICU bug? > > Thank you! The order of the rules does matter. Look at this: A somewhat longer test input: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ cat localedata/dz_BT.UTF-8.in.mini ཉ ྋྙ གཉ གཉྫ mfabian@taka:/local/mfabian/src/glibc (locales *$%) Again the same two lines in the rules file: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ cat rules-mini.txt &གཉ<གཉྫ &ཉ<<ྋྙ<གཉ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ mfabian@taka:/local/mfabian/src/glibc (locales *$%) Testing: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ ~/bin/icu-collation-test.py -r rules-mini.txt -i localedata/dz_BT.UTF-8.in.mini -o /tmp/dz_BT.UTF-8.out mfabian@taka:/local/mfabian/src/glibc (locales *$%) Result looks unexpected, probably not what you want: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ diff -u /local/mfabian/src/glibc/localedata/dz_BT.UTF-8.in.mini /tmp/dz_BT.UTF-8.out --- /local/mfabian/src/glibc/localedata/dz_BT.UTF-8.in.mini 2018-01-15 15:45:48.332377013 +0100 +++ /tmp/dz_BT.UTF-8.out 2018-01-15 15:50:46.357040054 +0100 @@ -1,4 +1,4 @@ +གཉྫ ཉ ྋྙ གཉ -གཉྫ mfabian@taka:/local/mfabian/src/glibc (locales *$%) Now I reverse the order of the two lines in the rules file: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ cat rules-mini.txt &ཉ<<ྋྙ<གཉ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ &གཉ<གཉྫ mfabian@taka:/local/mfabian/src/glibc (locales *$%) Testing again: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ ~/bin/icu-collation-test.py -r rules-mini.txt -i localedata/dz_BT.UTF-8.in.mini -o /tmp/dz_BT.UTF-8.out mfabian@taka:/local/mfabian/src/glibc (locales *$%) No difference, I get the expected order: mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ diff -u /local/mfabian/src/glibc/localedata/dz_BT.UTF-8.in.mini /tmp/dz_BT.UTF-8.out mfabian@taka:/local/mfabian/src/glibc (locales *$%) $ (In reply to Elie Roux from comment #9) > I have to say I don't really understand why ICU behaves like that... I think > we should do two things: > > - change my rule file so that it contains just one line and fix this oddity > - report a bug on ICU (maybe it's not a bug per se, but I can't see any > other way to solve this mistery) > > I'll fix the rule file (possibly today). If you have some time do you think > you could report the ICU bug? > > Thank you! Here are the two lines with rules: &ཉ<<ྋྙ<གཉ<མཉ<རྙ=ཪྙ<སྙ<བརྙ=བཪྙ<བསྙ &གཉ<གཉྫ And here I added code points in [] brackets to understand better what is going on: &ག[0F42]ཉ[0F49]<ག[0F42]ཉ[0F49]ྫ[0FAB] &ཉ[0F49]<<ྋ[0F8B]ྙ[0F99]<ག[0F42]ཉ[0F49]<མ[0F58]ཉ[0F49]<ར[0F62]ྙ[0F99]=ཪ[0F6A]ྙ[0F99]<ས[0F66]ྙ[0F99]<བ[0F56]ར[0F62]ྙ[0F99]=བ[0F56]ཪ[0F6A]ྙ[0F99]<བ[0F56]ས[0F66]ྙ[0F99] So the first line orders U+0F42 U+0F49 U+0FAB after U+0F42 U+0F49. But then the second line reorders U+0F42 U+0F49 after U+0F8B U+0F99. So the reference point after which U+0F42 U+0F49 U+0FAB has been reordered in the first line has been moved to somewhere else by the second line. Moving away that reference point U+0F42 U+0F49 does not move U+0F42 U+0F49 U+0FAB as well to stay behind the reference point U+0F42 U+0F49. I.e. the second line overrides the first. If the order of the lines is reversed, it works, because the line: &ཉ[0F49]<<ྋ[0F8B]ྙ[0F99]<ག[0F42]ཉ[0F49]<མ[0F58]ཉ[0F49]<ར[0F62]ྙ[0F99]=ཪ[0F6A]ྙ[0F99]<ས[0F66]ྙ[0F99]<བ[0F56]ར[0F62]ྙ[0F99]=བ[0F56]ཪ[0F6A]ྙ[0F99]<བ[0F56]ས[0F66]ྙ[0F99] then first reorders U+0F42 U+0F49 somewhere and *after* doing that, the line: &ག[0F42]ཉ[0F49]<ག[0F42]ཉ[0F49]ྫ[0FAB] inserts U+0F42 U+0F49 U+0FAB after the current position of U+0F42 U+0F49. Thanks a lot for the explanation! I'm inspecting that, I'll change this rule and the one line 12, which is in the same case I think. I need to take more time to work on this, probably tonight. A simpler example. A small rules file:
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
$ cat rules-mini.txt
&b<z
&d<b
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
And a simple test file:
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
$ cat /tmp/test-latin.in
a
b
c
d
e
f
z
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
Testing:
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
$ ~/bin/icu-collation-test.py -r rules-mini.txt -i /tmp/test-latin.in -o /tmp/test-latin.out
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
Result:
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
$ cat /tmp/test-latin.out
a
z
c
d
b
e
f
mfabian@taka:/local/mfabian/src/glibc (locales *$%)
$
The first line in the rules file move z after b. Then the second line
movez b after d. But that does not keep the z aber the b, the z is not
reordered together with the b, z stays where it was originally put by
the first line in the rules file. Moving the reference point b
somewhere else does not move the z.
(In reply to Elie Roux from comment #12) > Thanks a lot for the explanation! I'm inspecting that, I'll change this rule > and the one line 12, which is in the same case I think. I need to take more > time to work on this, probably tonight. Thank you! (In reply to Mike FABIAN from comment #14) > (In reply to Elie Roux from comment #12) > > Thanks a lot for the explanation! I'm inspecting that, I'll change this rule > > and the one line 12, which is in the same case I think. I need to take more > > time to work on this, probably tonight. > > Thank you! Ah, and maybe add some more stuff to your test file, apparently your test file did not check the correct order of གཉ and གཉྫ. I've been able to look at things properly and... there is in fact no bug. The rules do exactly what they are supposed to do, and the sorted list are exactly as the reference dictionary. I was wrong when I suggested that &ཉ<<ྋྙ<གཉ<གཉྫ would be correct. It would seem more natural to me, but the dictionary I took as a reference does not say that at all... This is a pretty specific case, and the sorting of transliterated Sanskrit is sort of a grey area, but I think we can keep that. I'm really sorry for the inconvenience. Would that be ok? (In reply to Elie Roux from comment #16) > I've been able to look at things properly and... there is in fact no bug. > The rules do exactly what they are supposed to do, and the sorted list are > exactly as the reference dictionary. I was wrong when I suggested that > > &ཉ<<ྋྙ<གཉ<གཉྫ > > would be correct. It would seem more natural to me, but the dictionary I > took as a reference does not say that at all... This is a pretty specific > case, and the sorting of transliterated Sanskrit is sort of a grey area, > but I think we can keep that. I'm really sorry for the inconvenience. > > Would that be ok? But what is the purpose of line 7: https://github.com/eroux/tibetan-collation/blob/master/implementations/Unicode/rules.txt#L7 &གཉ<གཉྫ if it is overidden later? Should གཉ be sorted before གཉྫ or not? well, things are supposed to be sorted just like in the sorted list attached to this bug report. Now, I agree there is some magic going on here, and it's not totally obvious to me how this works, but it works. Even though it's not clear what line 7 does, it clearly does something, because if you remove it, the tests on the tibetan-collation github repo fails with: expected [གངས་ལྷགས།, གཉྫིར།, གད།] got [གངས་ལྷགས།, གད།, གཉྫིར།] The test corresponds to page 347 of the tshig mdzod chen mo: https://www.tbrc.org/browser/ImageService?work=W29329&igroup=I1KG15042&image=379&first=1&last=1058&fetchimg=yes So line 7 has a purpose and doesn't get completely overwritten, although I agree the magic that takes place is a bit above my head... I suppose that somehow it indicates that གཉྫ should be sorted after the initial value of གཉ, and this get recorded somehow, even though གཉ then takes another value afterwards. I guess it may become less confusing with a bit of an understanding of Tibetan: in Tibetan གཉ absolutely never exists on its own, as it would be main letter ག then suffix ཉ and this can simply never happens (ཉ cannot be a suffix). What may happen are two cases starting with གཉ: 1. གཉྫིར is transliterated Sanskrit, and sort of exceptionally (and quite erratically) behaves as if ཉ was a suffix, and is thus sorted with the main letter ག, and that's what line 7 is trying to sort. What I believe happens is that at the time line 7 is parsed, གཉ is still sorted with the main letter ག, as it would be in the root collation. So this sorts གཉྫ with the main letter ག. Note that if you put the rule at the end of the file, the result is not the same, so I think it's more or less what's happening... 2. གཉར is prefix ག, then main letter ཉ then suffix ར, which is sorted in a totally different way, with the main letter ཉ, as stated by the rule of line 30, far after main letter ག. So this sorts གཉ with the main letter ཉ. That's my understanding of the situation, and I still think the rules are correct... I'm not sure I have made things clearer, if you want more details don't hesitate to ask! (In reply to Elie Roux from comment #18) > well, things are supposed to be sorted just like in the sorted list attached > to this bug report. > > Now, I agree there is some magic going on here, and it's not totally obvious > to me how this works, but it works. > > Even though it's not clear what line 7 does, it clearly does something, > because if you remove it, the tests on the tibetan-collation github repo > fails with: Yes, it does something. You see that also in my simple Latin example in comment#13: https://sourceware.org/bugzilla/show_bug.cgi?id=21547#c13 In &b<z &d<b The rule &b<z does something, although b is reordered again by the next rule. But z is not reordered and stays where it was reordered by the first rule. So one could have written &a<z &d<b to achieve the same effect to get the order a z c d b e f and that would have been much clearer and easier to understand rules. I have adapted the collation rules for many locales during the last weeks because I am updating to a new iso14651_t1_common file. And none of the locales had a collation rule which reordered some character after an anchor character and then reordered that anchor. It is not that this cannot work, it just seems very weird. I agree it's counter-intuitive. I think the reason why the rule is there is because of some Tibetan way of expressing things, let me elaborate: I could replace &གཉ<གཉྫ with &གངས<གཉྫ this would keep all tests passing, but the reason I don't really want to do that is the following: གངས may not be the very last element before གཉྫ, it's very possible that some not-very common suffix combination (in Dzongkha for instance). And, although that would be quite exceptional, a strange suffix combination could happen between གངས and གཉྫ. While basically nothing could happen between the initial version of གཉ and གཉྫ, so the rule as it is is quite safe. Does it make more sense? If these rules are the only ones to use this pattern and you think it's confusing, I'll remove it though, I suppose &གངས<གཉྫ is a good enough replacement. What do you think? Created attachment 10755 [details]
glibc/locales/iso14651_t1_common
Created attachment 10756 [details]
glib/localedata/locales/dz_BT
I finally could translate your collation rules to the glibc LC_COLLATE syntax and make it work,i.e. sort your test file attached to comment#2 correctly using glibc. The updated dz_BT locale file is attached. And one also needs the update for the iso14651_t1_common file which I have attached as well. That's really great, thank you so much! Is it possible to have the exact same collation rules for the bo locale(s)? bo, bo_IN and bo_CN (In reply to Elie Roux from comment #24) > That's really great, thank you so much! Is it possible to have the exact > same collation rules for the bo locale(s)? bo, bo_IN and bo_CN Yes, I did this: $ grep -A3 ^LC_COLLATE dz_* bo_* dz_BT:LC_COLLATE dz_BT-% Using the rules.txt attached to: dz_BT-% http://unicode.org/cldr/trac/ticket/9895 dz_BT-% See also: https://sourceware.org/bugzilla/show_bug.cgi?id=21547 -- bo_CN:LC_COLLATE bo_CN-copy "dz_BT" bo_CN-END LC_COLLATE bo_CN- -- bo_IN:LC_COLLATE bo_IN-copy "bo_CN" bo_IN-END LC_COLLATE bo_IN- So dz_BT implements the rules and bo_CN and bo_IN just copy them. That's perfect, thank you so much, really! Do you have any idea of when this will be publicly released? (the glibc / i18n release cycle is totally unknown to me) (In reply to Elie Roux from comment #26) > That's perfect, thank you so much, really! Do you have any idea of when this > will be publicly released? (the glibc / i18n release cycle is totally > unknown to me) It seems like it has been postponed to glibc 2.28: https://sourceware.org/glibc/wiki/Release/2.27#Desirable_this_release.3F where it says: Bug 14095 - Review / update collation data from Unicode / ISO 14651 Work on this is almost finished, patch will be sent to the mailing list in a few days. Postponed to 2.28 (Bug 14095 is the big collation update which includes the fix for this bug here). Here is the glibc release timeline: https://sourceware.org/glibc/wiki/Glibc%20Timeline So glibc 2.28 is scheduled for 2018-08-01. Sorry that I could not make it fast enough for glibc 2.27. That's perfect, no problem! I was just being curious This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".
The branch, master has been updated
via 874c56d7979858bbb1bb1604c55769ad0ce7a072 (commit)
via 159738548130d5ac4fe6178977e940ed5f8cfdc4 (commit)
via ce6636b06b67d6bb9b3d6927bf2a926b9b7478f5 (commit)
via ac3a3b4b0d561d776b60317d6a926050c8541655 (commit)
via 770cbe147cf33580e05ba6de78993c3070c5c2f8 (commit)
via 0fc355d9a7b3cc9d5e4190ce929e1eb4459ef0ea (commit)
via 43f3893f4b5679cb9eb93300b18f7febd17e5239 (commit)
via df74ef786f9c87ce5404df3b68a91cb9d2c4c26f (commit)
via d5adfbadd47e6836a7ddae54fba9f88e2b3354db (commit)
via 5f5a96109187b4bb4a10b62139ab1c7fe45f7c1d (commit)
via 8a97e9002ffa807b49e1222e5a9d51ce7896f209 (commit)
via bbdd2fba7d36d8f03c919b34f95238d8cf248b47 (commit)
via 1569e551aff088ed48e2694b07045256f3582271 (commit)
via 9479b6d5e08eacce06c6ab60abc9b2f4eb8b71e4 (commit)
from 93d260ddda87a124d3fbb9af400fa154cfd00b4b (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=874c56d7979858bbb1bb1604c55769ad0ce7a072
commit 874c56d7979858bbb1bb1604c55769ad0ce7a072
Author: Mike FABIAN <mfabian@redhat.com>
Date: Thu Dec 21 18:56:52 2017 +0100
Remove the lines from cmn_TW.UTF-8.in which cannot work at the moment.
See this bug https://sourceware.org/bugzilla/show_bug.cgi?id=22898
These lines don’t yet work because of a glibc bug, not because of
problems in the locale data. No matter what sorting rules one uses,
these characters cannot be sorted at all at the moment.
As soon as that bug is fixed, these lines should be added back to the
test file.
* localedata/cmn_TW.UTF-8.in: Remove the lines which cannot
be sorted correctly at the moment because of a bug.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=159738548130d5ac4fe6178977e940ed5f8cfdc4
commit 159738548130d5ac4fe6178977e940ed5f8cfdc4
Author: Mike FABIAN <mfabian@redhat.com>
Date: Mon Dec 11 18:26:22 2017 +0100
Adapt collation in several locales to the new iso14651_t1_common file
[BZ #22550] - es_ES locale (and other es_* locales): collation should
treat ñ as a primary different character, sync the collation
for Spanish with CLDR
[BZ #21547] - Tibetan script collation broken (Dzongkha and Tibetan)
* localedata/Makefile: Add new test files.
* localedata/lv_LV.UTF-8.in: Adapt test file to new collation order.
* localedata/sv_SE.ISO-8859-1.in: Adapt test file to new collation order.
* localedata/uk_UA.UTF-8.in: Adapt test file to new collation order.
* localedata/am_ET.UTF-8.in: New test file.
* localedata/az_AZ.UTF-8.in: Likewise.
* localedata/be_BY.UTF-8.in: Likewise.
* localedata/ber_DZ.UTF-8.in: Likewise.
* localedata/ber_MA.UTF-8.in: Likewise.
* localedata/bg_BG.UTF-8.in: Likewise.
* localedata/br_FR.UTF-8.in: Likewise.
* localedata/cmn_TW.UTF-8.in: Likewise.
* localedata/crh_UA.UTF-8.in: Likewise.
* localedata/csb_PL.UTF-8.in: Likewise.
* localedata/cv_RU.UTF-8.in: Likewise.
* localedata/cy_GB.UTF-8.in: Likewise.
* localedata/dz_BT.UTF-8.in: Likewise.
* localedata/eo.UTF-8.in: Likewise.
* localedata/es_ES.UTF-8.in: Likewise.
* localedata/fa_IR.UTF-8.in: Likewise.
* localedata/fi_FI.UTF-8.in: Likewise.
* localedata/fil_PH.UTF-8.in: Likewise.
* localedata/fur_IT.UTF-8.in: Likewise.
* localedata/gez_ER.UTF-8@abegede.in: Likewise.
* localedata/ha_NG.UTF-8.in: Likewise.
* localedata/ig_NG.UTF-8.in: Likewise.
* localedata/ik_CA.UTF-8.in: Likewise.
* localedata/kk_KZ.UTF-8.in: Likewise.
* localedata/ku_TR.UTF-8.in: Likewise.
* localedata/ky_KG.UTF-8.in: Likewise.
* localedata/ln_CD.UTF-8.in: Likewise.
* localedata/mi_NZ.UTF-8.in: Likewise.
* localedata/ml_IN.UTF-8.in: Likewise.
* localedata/mn_MN.UTF-8.in: Likewise.
* localedata/mr_IN.UTF-8.in: Likewise.
* localedata/mt_MT.UTF-8.in: Likewise.
* localedata/nb_NO.UTF-8.in: Likewise.
* localedata/om_KE.UTF-8.in: Likewise.
* localedata/os_RU.UTF-8.in: Likewise.
* localedata/ps_AF.UTF-8.in: Likewise.
* localedata/ro_RO.UTF-8.in: Likewise.
* localedata/ru_RU.UTF-8.in: Likewise.
* localedata/sc_IT.UTF-8.in: Likewise.
* localedata/se_NO.UTF-8.in: Likewise.
* localedata/sq_AL.UTF-8.in: Likewise.
* localedata/sv_SE.UTF-8.in: Likewise.
* localedata/szl_PL.UTF-8.in: Likewise.
* localedata/tg_TJ.UTF-8.in: Likewise.
* localedata/tk_TM.UTF-8.in: Likewise.
* localedata/tt_RU.UTF-8.in: Likewise.
* localedata/tt_RU.UTF-8@iqtelif.in: Likewise.
* localedata/ug_CN.UTF-8.in: Likewise.
* localedata/uz_UZ.UTF-8.in: Likewise.
* localedata/vi_VN.UTF-8.in: Likewise.
* localedata/yi_US.UTF-8.in: Likewise.
* localedata/yo_NG.UTF-8.in: Likewise.
* localedata/zh_CN.UTF-8.in: Likewise.
* localedata/locales/am_ET: Adapt collation rules to new iso14651_t1_common
file and fix bugs in the collation.
* localedata/locales/az_AZ: Likewise.
* localedata/locales/be_BY: Likewise.
* localedata/locales/ber_DZ: Likewise.
* localedata/locales/ber_MA: Likewise.
* localedata/locales/bg_BG: Likewise.
* localedata/locales/br_FR: Likewise.
* localedata/locales/br_FR@euro: Likewise.
* localedata/locales/ca_ES: Likewise.
* localedata/locales/cns11643_stroke: Likewise.
* localedata/locales/crh_UA: Likewise.
* localedata/locales/cs_CZ: Likewise.
* localedata/locales/csb_PL: Likewise.
* localedata/locales/cv_RU: Likewise.
* localedata/locales/cy_GB: Likewise.
* localedata/locales/da_DK: Likewise.
* localedata/locales/dz_BT: Likewise.
* localedata/locales/en_CA: Likewise.
* localedata/locales/eo: Likewise.
* localedata/locales/es_CU: Likewise.
* localedata/locales/es_EC: Likewise.
* localedata/locales/es_ES: Likewise.
* localedata/locales/es_US: Likewise.
* localedata/locales/et_EE: Likewise.
* localedata/locales/fa_IR: Likewise.
* localedata/locales/fi_FI: Likewise.
* localedata/locales/fil_PH: Likewise.
* localedata/locales/fur_IT: Likewise.
* localedata/locales/gez_ER@abegede: Likewise.
* localedata/locales/ha_NG: Likewise.
* localedata/locales/hr_HR: Likewise.
* localedata/locales/hsb_DE: Likewise.
* localedata/locales/hu_HU: Likewise.
* localedata/locales/ig_NG: Likewise.
* localedata/locales/ik_CA: Likewise.
* localedata/locales/is_IS: Likewise.
* localedata/locales/iso14651_t1_pinyin: Likewise.
* localedata/locales/kk_KZ: Likewise.
* localedata/locales/ku_TR: Likewise.
* localedata/locales/ky_KG: Likewise.
* localedata/locales/ln_CD: Likewise.
* localedata/locales/lt_LT: Likewise.
* localedata/locales/lv_LV: Likewise.
* localedata/locales/mi_NZ: Likewise.
* localedata/locales/ml_IN: Likewise.
* localedata/locales/mn_MN: Likewise.
* localedata/locales/mr_IN: Likewise.
* localedata/locales/mt_MT: Likewise.
* localedata/locales/nb_NO: Likewise.
* localedata/locales/om_KE: Likewise.
* localedata/locales/os_RU: Likewise.
* localedata/locales/pl_PL: Likewise.
* localedata/locales/ps_AF: Likewise.
* localedata/locales/ro_RO: Likewise.
* localedata/locales/ru_RU: Likewise.
* localedata/locales/ru_UA: Likewise.
* localedata/locales/sc_IT: Likewise.
* localedata/locales/se_NO: Likewise.
* localedata/locales/si_LK: Likewise.
* localedata/locales/sq_AL: Likewise.
* localedata/locales/sv_FI: Likewise.
* localedata/locales/sv_FI@euro: Likewise.
* localedata/locales/sv_SE: Likewise.
* localedata/locales/szl_PL: Likewise.
* localedata/locales/tg_TJ: Likewise.
* localedata/locales/ti_ER: Likewise.
* localedata/locales/tk_TM: Likewise.
* localedata/locales/tl_PH: Likewise.
* localedata/locales/tr_TR: Likewise.
* localedata/locales/tt_RU: Likewise.
* localedata/locales/tt_RU@iqtelif: Likewise.
* localedata/locales/ug_CN: Likewise.
* localedata/locales/uk_UA: Likewise.
* localedata/locales/uz_UZ: Likewise.
* localedata/locales/uz_UZ@cyrillic: Likewise.
* localedata/locales/vi_VN: Likewise.
* localedata/locales/yi_US: Likewise.
* localedata/locales/yo_NG: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=ce6636b06b67d6bb9b3d6927bf2a926b9b7478f5
commit ce6636b06b67d6bb9b3d6927bf2a926b9b7478f5
Author: Mike FABIAN <mfabian@redhat.com>
Date: Mon Jan 1 15:33:50 2018 +0100
Improve gen-locales.mk and gen-locale.sh to make test files with @ options work
With out this, adding collation test files like localedata/gez_ER.UTF-8@abegede.in
does not work for locales which contain @ modifiers.
* gen-locales.mk: Make test files which contain @ modifiers in their
name work.
* localedata/gen-locale.sh: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=ac3a3b4b0d561d776b60317d6a926050c8541655
commit ac3a3b4b0d561d776b60317d6a926050c8541655
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 23 17:29:36 2018 +0100
Fix test cases tst-fnmatch and tst-regexloc for the new iso14651_t1_common file.
See:
http://pubs.opengroup.org/onlinepubs/7908799/xbd/re.html
> A range expression represents the set of collating elements that fall
> between two elements in the current collation sequence,
> inclusively. It is expressed as the starting point and the ending
> point separated by a hyphen (-).
>
> Range expressions must not be used in portable applications because
> their behaviour is dependent on the collating sequence. Ranges will be
> treated according to the current collating sequence, and include such
> characters that fall within the range based on that collating
> sequence, regardless of character values. This, however, means that
> the interpretation will differ depending on collating sequence. If,
> for instance, one collating sequence defines ä as a variant of a,
> while another defines it as a letter following z, then the expression
> [ä-z] is valid in the first language and invalid in the second.
Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
The new iso14651_t1_common lists upper case and lower case Latin characters
in a different order than the old one which causes surprising results
for example in the de_DE locale: [a-z] now includes A because A comes
after a in iso14651_t1_common but does not include Z because that comes
after z in iso14651_t1_common.
* posix/tst-fnmatch.input: Fix results for range expressions
for non C locales.
* posix/tst-regexloc.c: Do not use a range expression for
de_DE.ISO-8859-1 locale.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=770cbe147cf33580e05ba6de78993c3070c5c2f8
commit 770cbe147cf33580e05ba6de78993c3070c5c2f8
Author: Mike FABIAN <mfabian@redhat.com>
Date: Fri Dec 15 07:19:45 2017 +0100
Fix posix/bug-regex5.c test case, adapt to iso14651_t1_common upate
This test case tests how many collating elements are defined in
da_DK.ISO-8859-1 locale. The da_DK locale source defines 4:
collating-element <A-A> from "<U0041><U0041>"
collating-element <A-a> from "<U0041><U0061>"
collating-element <a-A> from "<U0061><U0041>"
collating-element <a-a> from "<U0061><U0061>"
The new iso14651_t1_common file defines more collating elements, two
of them are in the ISO-8859-1 range:
collating-element <U004C_00B7> from "<U004C><U00B7>" % decomposition of LATIN CAPITAL LETTER L WITH MIDDLE DOT
collating-element <U006C_00B7> from "<U006C><U00B7>" % decomposition of LATIN SMALL LETTER L WITH MIDDLE DOT
So the total count is now 6 instead of 4.
* posix/bug-regex5.c: Fix test case because with the new
iso14651_t1_common file, the da_DK locale now has 6 collating elements
in the ISO-8859-1 range instead of 4 with the old iso14651_t1_common
file.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=0fc355d9a7b3cc9d5e4190ce929e1eb4459ef0ea
commit 0fc355d9a7b3cc9d5e4190ce929e1eb4459ef0ea
Author: Mike FABIAN <mfabian@redhat.com>
Date: Wed Dec 13 14:39:54 2017 +0100
Collation order of @-. and space has changed in new iso14651_t1_common file, adapt test files
* localedata/da_DK.ISO-8859-1.in: In the new iso14651_t1_common file
downloaded from ISO, the collation order of @-. and space has changed.
Therefore, this test file needed to be adapted.
* localedata/fr_CA.UTF-8.in: Likewise.
* localedata/fr_FR.UTF-8.in: Likewise.
* localedata/uk_UA.UTF-8.in: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=43f3893f4b5679cb9eb93300b18f7febd17e5239
commit 43f3893f4b5679cb9eb93300b18f7febd17e5239
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Dec 12 14:39:34 2017 +0100
Collation order of ȥ has changed in new iso14651_t1_common file, adapt test files
* localedata/cs_CZ.UTF-8.in: adapt this test file to the collation
order of ȥ in the new iso14651_t1_common file.
* localedata/pl_PL.UTF-8.in: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=df74ef786f9c87ce5404df3b68a91cb9d2c4c26f
commit df74ef786f9c87ce5404df3b68a91cb9d2c4c26f
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 15:45:05 2018 +0100
Add sections for various scripts to the iso14651_t1_common file
* localedata/locales/iso14651_t1_common: Add sections for various
scripts to the iso14651_t1_common file.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=d5adfbadd47e6836a7ddae54fba9f88e2b3354db
commit d5adfbadd47e6836a7ddae54fba9f88e2b3354db
Author: Mike FABIAN <mfabian@redhat.com>
Date: Wed Jan 31 06:18:47 2018 +0100
iso14651_t1_common: make the fourth level the codepoint for characters which are ignorable on all 4 levels
Entries for characters which have “IGNORE” on all 4 levels like:
<U0001> IGNORE;IGNORE;IGNORE;IGNORE % START OF HEADING (in ISO 6429)
are changed into:
<U0001> IGNORE;IGNORE;IGNORE;<U0001> % START OF HEADING (in ISO 6429)
i.e. putting the code point of the character into the fourth level
instead of “IGNORE”. Without that change, all such characters
would compare equal which would make a wcscoll test case fail.
It is better to have a clearly defined sort order even for characters
like this so it is good to use the code point as a tie-break.
* localedata/locales/iso14651_t1_common: Use the code point of a
character in the fourth collation level instead of IGNORE for all
entries which have IGNORE on all 4 levels.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=5f5a96109187b4bb4a10b62139ab1c7fe45f7c1d
commit 5f5a96109187b4bb4a10b62139ab1c7fe45f7c1d
Author: Mike FABIAN <mfabian@redhat.com>
Date: Mon Dec 11 20:00:24 2017 +0100
Add convenience symbols like <AFTER-A>, <BEFORE-A> to iso14651_t1_common
* localedata/locales/iso14651_t1_common: Add some convenient collation
symbols like <AFTER-A>, <BEFORE-A> to make tailoring easier using
rules similar to those in CLDR.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=8a97e9002ffa807b49e1222e5a9d51ce7896f209
commit 8a97e9002ffa807b49e1222e5a9d51ce7896f209
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 18:24:47 2018 +0100
Fixing syntax errors after updating the iso14651_t1_common file
* localedata/locales/iso14651_t1_common: The new version of this
file downloaded from ISO contained several syntax errors which
are fixed by this patch.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=bbdd2fba7d36d8f03c919b34f95238d8cf248b47
commit bbdd2fba7d36d8f03c919b34f95238d8cf248b47
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 18:07:39 2018 +0100
iso14651_t1_common: <U\([0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]\)> → <U000\1>
* localedata/locales/iso14651_t1_common: replace all <U.....>
with <U000.....> because glibc understands only 4 digit or 8 digit
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=1569e551aff088ed48e2694b07045256f3582271
commit 1569e551aff088ed48e2694b07045256f3582271
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 18:04:31 2018 +0100
Necessary changes after updating the iso14651_t1_common file
* localedata/locales/iso14651_t1_common: Necessary changes
to make the file downloaded from ISO usable by glibc.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9479b6d5e08eacce06c6ab60abc9b2f4eb8b71e4
commit 9479b6d5e08eacce06c6ab60abc9b2f4eb8b71e4
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 17:59:00 2018 +0100
Update iso14651_t1_common file to ISO14651_2016_TABLE1_en.txt [BZ #14095]
[BZ #14095] - Review / update collation data from Unicode / ISO 14651
File downloaded from:
http://standards.iso.org/iso-iec/14651/ed-4/ISO14651_2016_TABLE1_en.txt
Updating this file alone is not enough, there are problems in the new
file which need to be fixed and the collation rules for many locales
need to be adapted. This is done by the following patches.
This update also fixes the problem that many characters are treated as
identical when sorting because they were not yet in the old
iso14651_t1_common file, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1336308
- Infinite (∞) and empty set (∅) are treated as if they were the same character by sort and uniq
[BZ #14095]
* localedata/locales/iso14651_t1_common: Update file to
latest version from ISO (ISO14651_2016_TABLE1_en.txt).
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 224 +
gen-locales.mk | 4 +-
localedata/Makefile | 185 +-
localedata/am_ET.UTF-8.in | 347 +
localedata/az_AZ.UTF-8.in | 73 +
localedata/be_BY.UTF-8.in | 16 +
localedata/ber_DZ.UTF-8.in | 50 +
localedata/ber_MA.UTF-8.in | 13 +
localedata/bg_BG.UTF-8.in | 57 +
localedata/br_FR.UTF-8.in | 15 +
localedata/cmn_TW.UTF-8.in |75649 ++++++++++++++++++++++++++
localedata/crh_UA.UTF-8.in | 50 +
localedata/cs_CZ.UTF-8.in | 4 +-
localedata/csb_PL.UTF-8.in | 70 +
localedata/cv_RU.UTF-8.in | 45 +
localedata/cy_GB.UTF-8.in | 72 +
localedata/da_DK.ISO-8859-1.in | 4 +-
localedata/dz_BT.UTF-8.in | 789 +
localedata/eo.UTF-8.in | 32 +
localedata/es_ES.UTF-8.in | 46 +
localedata/fa_IR.UTF-8.in | 71 +
localedata/fi_FI.UTF-8.in | 140 +
localedata/fil_PH.UTF-8.in | 16 +
localedata/fr_CA.UTF-8.in | 9 +-
localedata/fr_FR.UTF-8.in | 9 +-
localedata/fur_IT.UTF-8.in | 12 +
localedata/gen-locale.sh | 5 +-
localedata/gez_ER.UTF-8@abegede.in | 365 +
localedata/ha_NG.UTF-8.in | 47 +
localedata/ig_NG.UTF-8.in | 93 +
localedata/ik_CA.UTF-8.in | 60 +
localedata/kk_KZ.UTF-8.in | 40 +
localedata/ku_TR.UTF-8.in | 52 +
localedata/ky_KG.UTF-8.in | 72 +
localedata/ln_CD.UTF-8.in | 18 +
localedata/locales/am_ET | 549 +-
localedata/locales/az_AZ | 201 +-
localedata/locales/be_BY | 41 +-
localedata/locales/ber_DZ | 173 +-
localedata/locales/ber_MA | 42 +-
localedata/locales/bg_BG | 290 +-
localedata/locales/br_FR | 55 +-
localedata/locales/br_FR@euro | 3 +-
localedata/locales/ca_ES | 16 +-
localedata/locales/cns11643_stroke | 9 +-
localedata/locales/crh_UA | 111 +-
localedata/locales/cs_CZ | 69 +-
localedata/locales/csb_PL | 83 +-
localedata/locales/cv_RU | 75 +-
localedata/locales/cy_GB | 242 +-
localedata/locales/da_DK | 116 +-
localedata/locales/dz_BT | 2484 +-
localedata/locales/en_CA | 8 -
localedata/locales/eo | 69 +-
localedata/locales/es_CU | 3 +-
localedata/locales/es_EC | 2 +-
localedata/locales/es_ES | 49 +-
localedata/locales/es_US | 56 +-
localedata/locales/et_EE | 31 +-
localedata/locales/fa_IR | 287 +-
localedata/locales/fi_FI | 175 +-
localedata/locales/fil_PH | 57 +-
localedata/locales/fur_IT | 15 +-
localedata/locales/gez_ER@abegede | 409 +-
localedata/locales/ha_NG | 165 +-
localedata/locales/hr_HR | 84 +-
localedata/locales/hsb_DE | 64 +-
localedata/locales/hu_HU | 298 +-
localedata/locales/ig_NG | 453 +-
localedata/locales/ik_CA | 153 +-
localedata/locales/is_IS | 72 +-
localedata/locales/iso14651_t1_common |94998 +++++++++++++++++++++++++++++----
localedata/locales/iso14651_t1_pinyin | 9 +-
localedata/locales/kk_KZ | 132 +-
localedata/locales/ku_TR | 87 +-
localedata/locales/ky_KG | 59 +-
localedata/locales/ln_CD | 47 +-
localedata/locales/lt_LT | 52 +-
localedata/locales/lv_LV | 67 +-
localedata/locales/mi_NZ | 43 +-
localedata/locales/ml_IN | 158 +-
localedata/locales/mn_MN | 34 +-
localedata/locales/mr_IN | 76 +-
localedata/locales/mt_MT | 144 +-
localedata/locales/nan_TW@latin | 33 +-
localedata/locales/nb_NO | 120 +-
localedata/locales/om_KE | 120 +-
localedata/locales/os_RU | 14 +-
localedata/locales/pl_PL | 66 +-
localedata/locales/ps_AF | 224 +-
localedata/locales/ro_RO | 99 +-
localedata/locales/ru_RU | 24 +-
localedata/locales/ru_UA | 16 +-
localedata/locales/sc_IT | 15 +-
localedata/locales/se_NO | 298 +-
localedata/locales/si_LK | 42 +
localedata/locales/sq_AL | 291 +-
localedata/locales/sv_FI | 2 +-
localedata/locales/sv_FI@euro | 2 +-
localedata/locales/sv_SE | 113 +-
localedata/locales/szl_PL | 86 +-
localedata/locales/tg_TJ | 106 +-
localedata/locales/ti_ER | 2 +
localedata/locales/tk_TM | 399 +-
localedata/locales/tl_PH | 31 +-
localedata/locales/tr_TR | 47 +-
localedata/locales/tt_RU | 244 +-
localedata/locales/tt_RU@iqtelif | 14 +-
localedata/locales/ug_CN | 196 +-
localedata/locales/uk_UA | 487 +-
localedata/locales/uz_UZ | 131 +-
localedata/locales/uz_UZ@cyrillic | 56 +-
localedata/locales/vi_VN | 242 +-
localedata/locales/yi_US | 125 +-
localedata/locales/yo_NG | 365 +-
localedata/lv_LV.UTF-8.in | 6 +-
localedata/mi_NZ.UTF-8.in | 37 +
localedata/ml_IN.UTF-8.in | 25 +
localedata/mn_MN.UTF-8.in | 15 +
localedata/mr_IN.UTF-8.in | 9 +
localedata/mt_MT.UTF-8.in | 39 +
localedata/nan_TW.UTF-8@latin.in | 11 +
localedata/nb_NO.UTF-8.in | 66 +
localedata/om_KE.UTF-8.in | 36 +
localedata/os_RU.UTF-8.in | 9 +
localedata/pl_PL.UTF-8.in | 4 +-
localedata/ps_AF.UTF-8.in | 61 +
localedata/ro_RO.UTF-8.in | 32 +
localedata/ru_RU.UTF-8.in | 15 +
localedata/sc_IT.UTF-8.in | 12 +
localedata/se_NO.UTF-8.in | 144 +
localedata/sq_AL.UTF-8.in | 82 +
localedata/sv_SE.ISO-8859-1.in | 10 +-
localedata/sv_SE.UTF-8.in | 107 +
localedata/szl_PL.UTF-8.in | 49 +
localedata/tg_TJ.UTF-8.in | 105 +
localedata/tk_TM.UTF-8.in | 213 +
localedata/tt_RU.UTF-8.in | 194 +
localedata/tt_RU.UTF-8@iqtelif.in | 53 +
localedata/ug_CN.UTF-8.in | 16 +
localedata/uk_UA.UTF-8.in | 18 +-
localedata/uz_UZ.UTF-8.in | 26 +
localedata/vi_VN.UTF-8.in | 45 +
localedata/yi_US.UTF-8.in | 39 +
localedata/yo_NG.UTF-8.in | 30 +
localedata/zh_CN.UTF-8.in |25498 +++++++++
posix/bug-regex5.c | 4 +-
posix/tst-fnmatch.input | 58 +-
posix/tst-regexloc.c | 4 +-
149 files changed, 197751 insertions(+), 15000 deletions(-)
create mode 100644 localedata/am_ET.UTF-8.in
create mode 100644 localedata/az_AZ.UTF-8.in
create mode 100644 localedata/be_BY.UTF-8.in
create mode 100644 localedata/ber_DZ.UTF-8.in
create mode 100644 localedata/ber_MA.UTF-8.in
create mode 100644 localedata/bg_BG.UTF-8.in
create mode 100644 localedata/br_FR.UTF-8.in
create mode 100644 localedata/cmn_TW.UTF-8.in
create mode 100644 localedata/crh_UA.UTF-8.in
create mode 100644 localedata/csb_PL.UTF-8.in
create mode 100644 localedata/cv_RU.UTF-8.in
create mode 100644 localedata/cy_GB.UTF-8.in
create mode 100644 localedata/dz_BT.UTF-8.in
create mode 100644 localedata/eo.UTF-8.in
create mode 100644 localedata/es_ES.UTF-8.in
create mode 100644 localedata/fa_IR.UTF-8.in
create mode 100644 localedata/fi_FI.UTF-8.in
create mode 100644 localedata/fil_PH.UTF-8.in
create mode 100644 localedata/fur_IT.UTF-8.in
create mode 100644 localedata/gez_ER.UTF-8@abegede.in
create mode 100644 localedata/ha_NG.UTF-8.in
create mode 100644 localedata/ig_NG.UTF-8.in
create mode 100644 localedata/ik_CA.UTF-8.in
create mode 100644 localedata/kk_KZ.UTF-8.in
create mode 100644 localedata/ku_TR.UTF-8.in
create mode 100644 localedata/ky_KG.UTF-8.in
create mode 100644 localedata/ln_CD.UTF-8.in
create mode 100644 localedata/mi_NZ.UTF-8.in
create mode 100644 localedata/ml_IN.UTF-8.in
create mode 100644 localedata/mn_MN.UTF-8.in
create mode 100644 localedata/mr_IN.UTF-8.in
create mode 100644 localedata/mt_MT.UTF-8.in
create mode 100644 localedata/nan_TW.UTF-8@latin.in
create mode 100644 localedata/nb_NO.UTF-8.in
create mode 100644 localedata/om_KE.UTF-8.in
create mode 100644 localedata/os_RU.UTF-8.in
create mode 100644 localedata/ps_AF.UTF-8.in
create mode 100644 localedata/ro_RO.UTF-8.in
create mode 100644 localedata/ru_RU.UTF-8.in
create mode 100644 localedata/sc_IT.UTF-8.in
create mode 100644 localedata/se_NO.UTF-8.in
create mode 100644 localedata/sq_AL.UTF-8.in
create mode 100644 localedata/sv_SE.UTF-8.in
create mode 100644 localedata/szl_PL.UTF-8.in
create mode 100644 localedata/tg_TJ.UTF-8.in
create mode 100644 localedata/tk_TM.UTF-8.in
create mode 100644 localedata/tt_RU.UTF-8.in
create mode 100644 localedata/tt_RU.UTF-8@iqtelif.in
create mode 100644 localedata/ug_CN.UTF-8.in
create mode 100644 localedata/uz_UZ.UTF-8.in
create mode 100644 localedata/vi_VN.UTF-8.in
create mode 100644 localedata/yi_US.UTF-8.in
create mode 100644 localedata/yo_NG.UTF-8.in
create mode 100644 localedata/zh_CN.UTF-8.in
Fixed. (In reply to Elie Roux from comment #28) > That's perfect, no problem! I was just being curious I hope your collation improvements will be accepted by CLDR soon. This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU C Library master sources".
The branch, mfabian/collation-update-2.27 has been created
at 9589174d076327deb7ed816d16b89b0e7470abd6 (commit)
- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=9589174d076327deb7ed816d16b89b0e7470abd6
commit 9589174d076327deb7ed816d16b89b0e7470abd6
Author: Mike FABIAN <mfabian@redhat.com>
Date: Thu Dec 21 18:56:52 2017 +0100
Remove the lines from cmn_TW.UTF-8.in which cannot work at the moment.
See this bug https://sourceware.org/bugzilla/show_bug.cgi?id=22898
These lines don’t yet work because of a glibc bug, not because of
problems in the locale data. No matter what sorting rules one uses,
these characters cannot be sorted at all at the moment.
As soon as that bug is fixed, these lines should be added back to the
test file.
* localedata/cmn_TW.UTF-8.in: Remove the lines which cannot
be sorted correctly at the moment because of a bug.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=e289a7d4c7f2abf09e4a4877b8cadcded7440e55
commit e289a7d4c7f2abf09e4a4877b8cadcded7440e55
Author: Mike FABIAN <mfabian@redhat.com>
Date: Mon Dec 11 18:26:22 2017 +0100
Adapt collation in several locales to the new iso14651_t1_common file
[BZ #22550] - es_ES locale (and other es_* locales): collation should
treat ñ as a primary different character, sync the collation
for Spanish with CLDR
[BZ #21547] - Tibetan script collation broken (Dzongkha and Tibetan)
* localedata/Makefile: Add new test files.
* localedata/lv_LV.UTF-8.in: Adapt test file to new collation order.
* localedata/sv_SE.ISO-8859-1.in: Adapt test file to new collation order.
* localedata/uk_UA.UTF-8.in: Adapt test file to new collation order.
* localedata/am_ET.UTF-8.in: New test file.
* localedata/az_AZ.UTF-8.in: Likewise.
* localedata/be_BY.UTF-8.in: Likewise.
* localedata/ber_DZ.UTF-8.in: Likewise.
* localedata/ber_MA.UTF-8.in: Likewise.
* localedata/bg_BG.UTF-8.in: Likewise.
* localedata/br_FR.UTF-8.in: Likewise.
* localedata/cmn_TW.UTF-8.in: Likewise.
* localedata/crh_UA.UTF-8.in: Likewise.
* localedata/csb_PL.UTF-8.in: Likewise.
* localedata/cv_RU.UTF-8.in: Likewise.
* localedata/cy_GB.UTF-8.in: Likewise.
* localedata/dz_BT.UTF-8.in: Likewise.
* localedata/eo.UTF-8.in: Likewise.
* localedata/es_ES.UTF-8.in: Likewise.
* localedata/fa_IR.UTF-8.in: Likewise.
* localedata/fi_FI.UTF-8.in: Likewise.
* localedata/fil_PH.UTF-8.in: Likewise.
* localedata/fur_IT.UTF-8.in: Likewise.
* localedata/gez_ER.UTF-8@abegede.in: Likewise.
* localedata/ha_NG.UTF-8.in: Likewise.
* localedata/ig_NG.UTF-8.in: Likewise.
* localedata/ik_CA.UTF-8.in: Likewise.
* localedata/kk_KZ.UTF-8.in: Likewise.
* localedata/ku_TR.UTF-8.in: Likewise.
* localedata/ky_KG.UTF-8.in: Likewise.
* localedata/ln_CD.UTF-8.in: Likewise.
* localedata/mi_NZ.UTF-8.in: Likewise.
* localedata/ml_IN.UTF-8.in: Likewise.
* localedata/mn_MN.UTF-8.in: Likewise.
* localedata/mr_IN.UTF-8.in: Likewise.
* localedata/mt_MT.UTF-8.in: Likewise.
* localedata/nb_NO.UTF-8.in: Likewise.
* localedata/om_KE.UTF-8.in: Likewise.
* localedata/os_RU.UTF-8.in: Likewise.
* localedata/ps_AF.UTF-8.in: Likewise.
* localedata/ro_RO.UTF-8.in: Likewise.
* localedata/ru_RU.UTF-8.in: Likewise.
* localedata/sc_IT.UTF-8.in: Likewise.
* localedata/se_NO.UTF-8.in: Likewise.
* localedata/sq_AL.UTF-8.in: Likewise.
* localedata/sv_SE.UTF-8.in: Likewise.
* localedata/szl_PL.UTF-8.in: Likewise.
* localedata/tg_TJ.UTF-8.in: Likewise.
* localedata/tk_TM.UTF-8.in: Likewise.
* localedata/tt_RU.UTF-8.in: Likewise.
* localedata/tt_RU.UTF-8@iqtelif.in: Likewise.
* localedata/ug_CN.UTF-8.in: Likewise.
* localedata/uz_UZ.UTF-8.in: Likewise.
* localedata/vi_VN.UTF-8.in: Likewise.
* localedata/yi_US.UTF-8.in: Likewise.
* localedata/yo_NG.UTF-8.in: Likewise.
* localedata/zh_CN.UTF-8.in: Likewise.
* localedata/locales/am_ET: Adapt collation rules to new iso14651_t1_common
file and fix bugs in the collation.
* localedata/locales/az_AZ: Likewise.
* localedata/locales/be_BY: Likewise.
* localedata/locales/ber_DZ: Likewise.
* localedata/locales/ber_MA: Likewise.
* localedata/locales/bg_BG: Likewise.
* localedata/locales/br_FR: Likewise.
* localedata/locales/br_FR@euro: Likewise.
* localedata/locales/ca_ES: Likewise.
* localedata/locales/cns11643_stroke: Likewise.
* localedata/locales/crh_UA: Likewise.
* localedata/locales/cs_CZ: Likewise.
* localedata/locales/csb_PL: Likewise.
* localedata/locales/cv_RU: Likewise.
* localedata/locales/cy_GB: Likewise.
* localedata/locales/da_DK: Likewise.
* localedata/locales/dz_BT: Likewise.
* localedata/locales/en_CA: Likewise.
* localedata/locales/eo: Likewise.
* localedata/locales/es_CU: Likewise.
* localedata/locales/es_EC: Likewise.
* localedata/locales/es_ES: Likewise.
* localedata/locales/es_US: Likewise.
* localedata/locales/et_EE: Likewise.
* localedata/locales/fa_IR: Likewise.
* localedata/locales/fi_FI: Likewise.
* localedata/locales/fil_PH: Likewise.
* localedata/locales/fur_IT: Likewise.
* localedata/locales/gez_ER@abegede: Likewise.
* localedata/locales/ha_NG: Likewise.
* localedata/locales/hr_HR: Likewise.
* localedata/locales/hsb_DE: Likewise.
* localedata/locales/hu_HU: Likewise.
* localedata/locales/ig_NG: Likewise.
* localedata/locales/ik_CA: Likewise.
* localedata/locales/is_IS: Likewise.
* localedata/locales/iso14651_t1_pinyin: Likewise.
* localedata/locales/kk_KZ: Likewise.
* localedata/locales/ku_TR: Likewise.
* localedata/locales/ky_KG: Likewise.
* localedata/locales/ln_CD: Likewise.
* localedata/locales/lt_LT: Likewise.
* localedata/locales/lv_LV: Likewise.
* localedata/locales/mi_NZ: Likewise.
* localedata/locales/ml_IN: Likewise.
* localedata/locales/mn_MN: Likewise.
* localedata/locales/mr_IN: Likewise.
* localedata/locales/mt_MT: Likewise.
* localedata/locales/nb_NO: Likewise.
* localedata/locales/om_KE: Likewise.
* localedata/locales/os_RU: Likewise.
* localedata/locales/pl_PL: Likewise.
* localedata/locales/ps_AF: Likewise.
* localedata/locales/ro_RO: Likewise.
* localedata/locales/ru_RU: Likewise.
* localedata/locales/ru_UA: Likewise.
* localedata/locales/sc_IT: Likewise.
* localedata/locales/se_NO: Likewise.
* localedata/locales/si_LK: Likewise.
* localedata/locales/sq_AL: Likewise.
* localedata/locales/sv_FI: Likewise.
* localedata/locales/sv_FI@euro: Likewise.
* localedata/locales/sv_SE: Likewise.
* localedata/locales/szl_PL: Likewise.
* localedata/locales/tg_TJ: Likewise.
* localedata/locales/ti_ER: Likewise.
* localedata/locales/tk_TM: Likewise.
* localedata/locales/tl_PH: Likewise.
* localedata/locales/tr_TR: Likewise.
* localedata/locales/tt_RU: Likewise.
* localedata/locales/tt_RU@iqtelif: Likewise.
* localedata/locales/ug_CN: Likewise.
* localedata/locales/uk_UA: Likewise.
* localedata/locales/uz_UZ: Likewise.
* localedata/locales/uz_UZ@cyrillic: Likewise.
* localedata/locales/vi_VN: Likewise.
* localedata/locales/yi_US: Likewise.
* localedata/locales/yo_NG: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=242596394db9dad6147bb2b7bcb53d8a7610e1d0
commit 242596394db9dad6147bb2b7bcb53d8a7610e1d0
Author: Mike FABIAN <mfabian@redhat.com>
Date: Mon Jan 1 15:33:50 2018 +0100
Improve gen-locales.mk and gen-locale.sh to make test files with @ options work
With out this, adding collation test files like localedata/gez_ER.UTF-8@abegede.in
does not work for locales which contain @ modifiers.
* gen-locales.mk: Make test files which contain @ modifiers in their
name work.
* localedata/gen-locale.sh: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=cc5351f2c0502826f8b4143f3646d44e334ff7b8
commit cc5351f2c0502826f8b4143f3646d44e334ff7b8
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 23 17:29:36 2018 +0100
Fix test cases tst-fnmatch and tst-regexloc for the new iso14651_t1_common file.
See:
http://pubs.opengroup.org/onlinepubs/7908799/xbd/re.html
> A range expression represents the set of collating elements that fall
> between two elements in the current collation sequence,
> inclusively. It is expressed as the starting point and the ending
> point separated by a hyphen (-).
>
> Range expressions must not be used in portable applications because
> their behaviour is dependent on the collating sequence. Ranges will be
> treated according to the current collating sequence, and include such
> characters that fall within the range based on that collating
> sequence, regardless of character values. This, however, means that
> the interpretation will differ depending on collating sequence. If,
> for instance, one collating sequence defines ä as a variant of a,
> while another defines it as a letter following z, then the expression
> [ä-z] is valid in the first language and invalid in the second.
Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
The new iso14651_t1_common lists upper case and lower case Latin characters
in a different order than the old one which causes surprising results
for example in the de_DE locale: [a-z] now includes A because A comes
after a in iso14651_t1_common but does not include Z because that comes
after z in iso14651_t1_common.
* posix/tst-fnmatch.input: Fix results for range expressions
for non C locales.
* posix/tst-regexloc.c: Do not use a range expression for
de_DE.ISO-8859-1 locale.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=ffa8106c727607fb365f2b93649fe3ea182dffe4
commit ffa8106c727607fb365f2b93649fe3ea182dffe4
Author: Mike FABIAN <mfabian@redhat.com>
Date: Fri Dec 15 07:19:45 2017 +0100
Fix posix/bug-regex5.c test case, adapt to iso14651_t1_common upate
This test case tests how many collating elements are defined in
da_DK.ISO-8859-1 locale. The da_DK locale source defines 4:
collating-element <A-A> from "<U0041><U0041>"
collating-element <A-a> from "<U0041><U0061>"
collating-element <a-A> from "<U0061><U0041>"
collating-element <a-a> from "<U0061><U0061>"
The new iso14651_t1_common file defines more collating elements, two
of them are in the ISO-8859-1 range:
collating-element <U004C_00B7> from "<U004C><U00B7>" % decomposition of LATIN CAPITAL LETTER L WITH MIDDLE DOT
collating-element <U006C_00B7> from "<U006C><U00B7>" % decomposition of LATIN SMALL LETTER L WITH MIDDLE DOT
So the total count is now 6 instead of 4.
* posix/bug-regex5.c: Fix test case because with the new
iso14651_t1_common file, the da_DK locale now has 6 collating elements
in the ISO-8859-1 range instead of 4 with the old iso14651_t1_common
file.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=61e613fb97aa619ae4fabac3f106d5fffe15eacb
commit 61e613fb97aa619ae4fabac3f106d5fffe15eacb
Author: Mike FABIAN <mfabian@redhat.com>
Date: Wed Dec 13 14:39:54 2017 +0100
Collation order of @-. and space has changed in new iso14651_t1_common file, adapt test files
* localedata/da_DK.ISO-8859-1.in: In the new iso14651_t1_common file
downloaded from ISO, the collation order of @-. and space has changed.
Therefore, this test file needed to be adapted.
* localedata/fr_CA.UTF-8.in: Likewise.
* localedata/fr_FR.UTF-8.in: Likewise.
* localedata/uk_UA.UTF-8.in: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=059454de60bdb1be9979ee09596c1e9a7e9e6c8b
commit 059454de60bdb1be9979ee09596c1e9a7e9e6c8b
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Dec 12 14:39:34 2017 +0100
Collation order of ȥ has changed in new iso14651_t1_common file, adapt test files
* localedata/cs_CZ.UTF-8.in: adapt this test file to the collation
order of ȥ in the new iso14651_t1_common file.
* localedata/pl_PL.UTF-8.in: Likewise.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=1f4df3bb2ac69f2e1947c2953379a7f19b5f0c35
commit 1f4df3bb2ac69f2e1947c2953379a7f19b5f0c35
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 15:45:05 2018 +0100
Add sections for various scripts to the iso14651_t1_common file
* localedata/locales/iso14651_t1_common: Add sections for various
scripts to the iso14651_t1_common file.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a93fecdcece3e2178834f4b4868b2309b0158753
commit a93fecdcece3e2178834f4b4868b2309b0158753
Author: Mike FABIAN <mfabian@redhat.com>
Date: Wed Jan 31 06:18:47 2018 +0100
iso14651_t1_common: make the fourth level the codepoint for characters which are ignorable on all 4 levels
Entries for characters which have “IGNORE” on all 4 levels like:
<U0001> IGNORE;IGNORE;IGNORE;IGNORE % START OF HEADING (in ISO 6429)
are changed into:
<U0001> IGNORE;IGNORE;IGNORE;<U0001> % START OF HEADING (in ISO 6429)
i.e. putting the code point of the character into the fourth level
instead of “IGNORE”. Without that change, all such characters
would compare equal which would make a wcscoll test case fail.
It is better to have a clearly defined sort order even for characters
like this so it is good to use the code point as a tie-break.
* localedata/locales/iso14651_t1_common: Use the code point of a
character in the fourth collation level instead of IGNORE for all
entries which have IGNORE on all 4 levels.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=3e7089bf28ed1fd77e644bb3ce7405aff7847e61
commit 3e7089bf28ed1fd77e644bb3ce7405aff7847e61
Author: Mike FABIAN <mfabian@redhat.com>
Date: Mon Dec 11 20:00:24 2017 +0100
Add convenience symbols like <AFTER-A>, <BEFORE-A> to iso14651_t1_common
* localedata/locales/iso14651_t1_common: Add some convenient collation
symbols like <AFTER-A>, <BEFORE-A> to make tailoring easier using
rules similar to those in CLDR.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=50a54ba443575e69ffb03aa67d53ccf8b66a4fbd
commit 50a54ba443575e69ffb03aa67d53ccf8b66a4fbd
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 18:24:47 2018 +0100
Fixing syntax errors after updating the iso14651_t1_common file
* localedata/locales/iso14651_t1_common: The new version of this
file downloaded from ISO contained several syntax errors which
are fixed by this patch.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=661ab21c7521ba8e6e8bc7dad897b6cf162e0cd0
commit 661ab21c7521ba8e6e8bc7dad897b6cf162e0cd0
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 18:07:39 2018 +0100
iso14651_t1_common: <U\([0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]\)> → <U000\1>
* localedata/locales/iso14651_t1_common: replace all <U.....>
with <U000.....> because glibc understands only 4 digit or 8 digit
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=06061c30d615b2862ac360f11384092c92022ea7
commit 06061c30d615b2862ac360f11384092c92022ea7
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 18:04:31 2018 +0100
Necessary changes after updating the iso14651_t1_common file
* localedata/locales/iso14651_t1_common: Necessary changes
to make the file downloaded from ISO usable by glibc.
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=bc1d41044c0cf9f0214acdbfd79b6cd11fd1e8c1
commit bc1d41044c0cf9f0214acdbfd79b6cd11fd1e8c1
Author: Mike FABIAN <mfabian@redhat.com>
Date: Tue Jan 30 17:59:00 2018 +0100
Update iso14651_t1_common file to ISO14651_2016_TABLE1_en.txt [BZ #14095]
[BZ #14095] - Review / update collation data from Unicode / ISO 14651
File downloaded from:
http://standards.iso.org/iso-iec/14651/ed-4/ISO14651_2016_TABLE1_en.txt
Updating this file alone is not enough, there are problems in the new
file which need to be fixed and the collation rules for many locales
need to be adapted. This is done by the following patches.
This update also fixes the problem that many characters are treated as
identical when sorting because they were not yet in the old
iso14651_t1_common file, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1336308
- Infinite (∞) and empty set (∅) are treated as if they were the same character by sort and uniq
[BZ #14095]
* localedata/locales/iso14651_t1_common: Update file to
latest version from ISO (ISO14651_2016_TABLE1_en.txt).
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=16e349c550942d274d3193ccedaa88855e3ac690
commit 16e349c550942d274d3193ccedaa88855e3ac690
Author: Mike FABIAN <mfabian@redhat.com>
Date: Fri Mar 2 11:29:24 2018 +0100
Remove --quiet argument when installing locales
Using this argument hides problems. I would like to see when something fails.
* localedata/Makefile: Remove --quiet argument when
installing locales
-----------------------------------------------------------------------
|