Summary: | Review / update transliteration data | ||
---|---|---|---|
Product: | glibc | Reporter: | Marko Myllynen <myllynen> |
Component: | localedata | Assignee: | Not yet assigned to anyone <unassigned> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | carlos, libc-locales, maiku.fabian, pravin.d.s |
Priority: | P2 | Flags: | fweimer:
security-
|
Version: | 2.18 | ||
Target Milestone: | 2.23 | ||
Host: | Target: | ||
Build: | Last reconfirmed: | ||
Attachments: | 0001-Update-the-translit-files-to-Unicode-7.0.0.patch |
Description
Marko Myllynen
2013-10-18 08:04:53 UTC
I made a patch to update most of the translit_* files, see: https://sourceware.org/ml/libc-alpha/2015-04/msg00361.html Joseph S. Myers answered on libc-alpha: Joseph S. Myers> This is [BZ #16061] (I don't know if it fully Joseph S. Myers> addresses everything discussed in that bug, but it's Joseph S. Myers> at least relevant to that bug so the bug number Joseph S. Myers> should be mentioned in the ChangeLog entry, whether Joseph S. Myers> or not it's also added to NEWS and the bug closed Joseph S. Myers> once the patch is in). I don't know if it's relevant Joseph S. Myers> to any other open transliteration bug. Ah, yes, I’ll add the bug number. (In reply to Marko Myllynen from comment #0) > The localedata/locales/translit_* files are probably, based on comments in > them, at least partially generated from some version of UnicodeData.txt > (based on 93a568 it looks like the last major update has been for Unicode > 3.2 and 17b16e suggests them originally coming from an external > contributor). However, there are some characters missing even from the > Latin-1 Supplement block and in general it doesn't seem possible to update > the files just by using UnicodeData.txt. Some of the rules live in > locale/C-translit.h / locale/C-translit.h.in which also contain local > changes (like 61d5a6 / 2a81ea). C-translit.h.in seems to be manually edited and not generated from Unicode data. > It requires likely a lot of work to understand how the files have been > generated in the first place, how to identify relevant local changes, and > how to automate the process to update them in the future. These files seem to be automatically generated with some manual additions: locales/translit_circle locales/translit_cjk_compat locales/translit_combining locales/translit_compat locales/translit_font locales/translit_fraction my patch updates them automatically from UnicodeData.txt keeping the manual additions whereever they seem to make sense. locales/translit_neutral is apparently manually edited and not generated. locales/translit_cjk_variants is not generated from Unicode data either but from a UniVariants.Z file which can still be found here: http://kanji.zinbun.kyoto-u.ac.jp/~yasuoka/ftp/CJKtable/UniVariants.Z It is from 2002-08-15 and I have no idea how it has been created. So I did not touch /translit_cjk_variants. The following files locales/translit_hangul locales/translit_narrow locales/translit_small locales/translit_wide are automatically generated, but generating them automatically from Unicode 7.0.0 data would just reproduce the files as they are now, there have been no updates. Therefore I didn’t write generator scripts for these. Would generator scripts nevertheless be useful, so that we would notice if a change happens? I think a change in these files is very unlikely though. > Some individual examples of currently missing characters are U+00D8 (Ø) This is already here: translit_neutral:<U00D8> "<U004F><U0045>" (manually edited). And my patch adds it to translit_combining as: +% LATIN CAPITAL LETTER O WITH STROKE +<U00D8> <U004F> (But as a special hack, this does not come from UnicodeData.txt). > and U+0110 (Đ) Adding this seems to make sense as well, I added it to the “special hack” section of my gen_translit_combining.py: special_decompose_dict = { (0x0110,): [0x0044], # Đ → D (0x0111,): [0x0064], # đ → d ... > whereas other characters like U+00C6 (Æ) and U+0141 > (Ł) from their blocks (Latin-1 Supplement and Latin Extended-A, > respectively) are present. Some characters (like U+2033, ″) have > decomposition defined as is in Unicode Yes, this one is in translit_compat: $ grep -B1 U2033 translit_compat % DOUBLE PRIME <U2033> "<U2032><U2032>" (was already there, not added by my patch, it is generated from UnicodeData.txt). > but some characters (like U+00D6, Ö) have decomposition defined in > Unicode but not in glibc. glibc had this already in translit_combining: $ grep -B1 U00D6 translit_combining % LATIN CAPITAL LETTER O WITH DIAERESIS <U00D6> <U004F> (was already there, not added by my patch, it is generated from UnicodeData.txt by decomposing to U+004F U+0308 and then stripping the combining character U+0308). before commit 18a3a9a3 this was in locale/C-translit.h.in but it was apparently removed on purpose by commit 18a3a9a3: -/* <U00D6> LATIN CAPITAL LETTER O WITH DIAERESIS. */ -/* XXX It is not clear whether this is the best transliteration for - all locales. If not, we probably have to take it out completely. */ -"\xd6" "OE" “Ö” is transliterated to “OE” for example in German, but in English one usually transliterates it just as “O”. Therefore, translit_combining transliterates it to “O” by decomposing and stripping the combining character and locales like de_DE add their own transliteration rules: $ grep -A20 translit_start de_DE translit_start include "translit_combining";"" % German umlauts. % LATIN CAPITAL LETTER A WITH DIAERESIS. <U00C4> "<U0041><U0308>";"<U0041><U0045>" % LATIN CAPITAL LETTER O WITH DIAERESIS. <U00D6> "<U004F><U0308>";"<U004F><U0045>" % LATIN CAPITAL LETTER U WITH DIAERESIS. <U00DC> "<U0055><U0308>";"<U0055><U0045>" % LATIN SMALL LETTER A WITH DIAERESIS. <U00E4> "<U0061><U0308>";"<U0061><U0065>" % LATIN SMALL LETTER O WITH DIAERESIS. <U00F6> "<U006F><U0308>";"<U006F><U0065>" % LATIN SMALL LETTER U WITH DIAERESIS. <U00FC> "<U0075><U0308>";"<U0075><U0065>" % Danish. % LATIN CAPITAL LETTER A WITH RING ABOVE. <U00C5> "<U0041><U030A>";"<U0041><U0041>" ... Created attachment 8287 [details]
0001-Update-the-translit-files-to-Unicode-7.0.0.patch
updated patch.
(In reply to Mike FABIAN from comment #2) > (In reply to Marko Myllynen from comment #0) > > C-translit.h.in seems to be manually edited and not generated from > Unicode data. Based on earlier changelog comments it seems that C-translit.h.in was updated manually for Unicode 3.2.0, should it now be updated for Unicode 7.0.0 by some means? As discussed off-list, it seems that there are transliterations defined only in C-translit.h.in (like U20B9, INDIAN RUPEE SIGN) which take effect only with the C/POSIX locale but they are not in any translit_* files, should C-translit.h.in and translit_* files be synced for such cases? Or should C/POSIX perhaps be "pure" without any other rules except those from derived from Unicode while the rest could use locally added rules as well? > These files seem to be automatically generated with some manual additions: > > locales/translit_circle > locales/translit_cjk_compat > locales/translit_combining > locales/translit_compat > locales/translit_font > locales/translit_fraction > > my patch updates them automatically from UnicodeData.txt keeping > the manual additions whereever they seem to make sense. Related to above, I wonder should we make local changes more obvious for example by having translit_combining_unicode included from translit_combining? It would make it much easier for others to see what definitions are coming from Unicode and what definitions are ones provided by glibc. Or alternatively group the generated rules separately inside translit_* files. > is apparently manually edited and not generated. > > locales/translit_cjk_variants > > is not generated from Unicode data either but from a UniVariants.Z > file which can still be found here: > > http://kanji.zinbun.kyoto-u.ac.jp/~yasuoka/ftp/CJKtable/UniVariants.Z > > It is from 2002-08-15 and I have no idea how it has been created. > So I did not touch /translit_cjk_variants. Perhaps we could add a note about its origins to the file. > The following files > > locales/translit_hangul > locales/translit_narrow > locales/translit_small > locales/translit_wide > > are automatically generated, but generating them automatically from > Unicode 7.0.0 data would just reproduce the files as they are now, > there have been no updates. Therefore I didn’t write generator > scripts for these. Would generator scripts nevertheless be useful, > so that we would notice if a change happens? I think a change > in these files is very unlikely though. It indeed sounds unlikely but having a generator available might make things easier 10 or 20 years from now if someone wants to verify the situation then. But I think it's your call, I'm ok either way. > > Some individual examples of currently missing characters are U+00D8 (Ø) > > This is already here: > > translit_neutral:<U00D8> "<U004F><U0045>" Yes, it was added a bit after this report, see bug 15593 and commit f20820. > (manually edited). And my patch adds it to translit_combining as: > > +% LATIN CAPITAL LETTER O WITH STROKE > +<U00D8> <U004F> > > (But as a special hack, this does not come from UnicodeData.txt). Please see the above bug for more discussion on this, not sure is there one right answer which transliteration is the correct one to use here. Also, shouldn't Ø and Æ be handled in the same way? Looking at translit_neutral in more detail, I think it's actually wrong place for letters, it should contain non-letters only and if specific rules are needed for letters like Ø or Æ, those should be added directly in locale files (so the patch discussed in bug 15593 should have not been applied to translit_neutral after all). This would also mean that the special rules in the generator for cases like EM DASH and EN DASH should probably end up to translit_neutral not translit_combining. > > and U+0110 (Đ) > > Adding this seems to make sense as well Perhaps it might be best to start with minimal set of special rules and commit additional ones later (for example, I'd like to see 00D0, 00DE, and 014A with their lowercase counterparts added)? > > but some characters (like U+00D6, Ö) have decomposition defined in > > Unicode but not in glibc. > > glibc had this already in translit_combining: > > (was already there, not added by my patch, it is generated from > UnicodeData.txt by decomposing to U+004F U+0308 and then stripping the > combining character U+0308). Yes, I think what I meant to say was that the decomposition to U+004F U+0308 was missing but as you point out it is defined in some locales where it would be needed. Btw, I wonder should U+00D6 actually decompose to U+004F U+00A8 after U+004F U+0308 in those locales? Thanks. (In reply to Marko Myllynen from comment #4) > (In reply to Mike FABIAN from comment #2) > > (In reply to Marko Myllynen from comment #0) > > > > C-translit.h.in seems to be manually edited and not generated from > > Unicode data. > > Based on earlier changelog comments it seems that C-translit.h.in was > updated manually for Unicode 3.2.0, should it now be updated for Unicode > 7.0.0 by some means? Probably, but how? > > is apparently manually edited and not generated. > > > > locales/translit_cjk_variants > > > > is not generated from Unicode data either but from a UniVariants.Z > > file which can still be found here: > > > > http://kanji.zinbun.kyoto-u.ac.jp/~yasuoka/ftp/CJKtable/UniVariants.Z > > > > It is from 2002-08-15 and I have no idea how it has been created. > > So I did not touch /translit_cjk_variants. > > Perhaps we could add a note about its origins to the file. There is already a note in the comment section of that file. > Also, shouldn't Ø and Æ be handled in the same way? What do you mean by “handled in the same way”? > Looking at translit_neutral in more detail, I think it's actually wrong > place for letters, it should contain non-letters only and if specific rules > are needed for letters like Ø or Æ, those should be added directly in locale > files (so the patch discussed in bug 15593 should have not been applied to > translit_neutral after all). This would also mean that the special rules in > the generator for cases like EM DASH and EN DASH should probably end up to > translit_neutral not translit_combining. My guess is that the purpose of translit_neutral is to contain transliterations which are locale “neutral”, i.e. are the same for all locales. So I see no reason not to include letters. > > > but some characters (like U+00D6, Ö) have decomposition defined in > > > Unicode but not in glibc. > > > > glibc had this already in translit_combining: > > > > (was already there, not added by my patch, it is generated from > > UnicodeData.txt by decomposing to U+004F U+0308 and then stripping the > > combining character U+0308). > > Yes, I think what I meant to say was that the decomposition to U+004F U+0308 > was missing but as you point out it is defined in some locales where it > would be needed. Btw, I wonder should U+00D6 actually decompose to U+004F > U+00A8 after U+004F U+0308 in those locales? Ö -> O¨ Why? Is that a reasonable transliteration? It throws away less information but I think it is common practice to transliterate Ö just as O in English for example. (In reply to Mike FABIAN from comment #5) > (In reply to Marko Myllynen from comment #4) > > (In reply to Mike FABIAN from comment #2) > > > (In reply to Marko Myllynen from comment #0) > > > > > > C-translit.h.in seems to be manually edited and not generated from > > > Unicode data. > > > > Based on earlier changelog comments it seems that C-translit.h.in was > > updated manually for Unicode 3.2.0, should it now be updated for Unicode > > 7.0.0 by some means? > > Probably, but how? Good question - do you see it feasible to use the generator to also produce C-translit.h.in (sans the previous individual additions)? > > Perhaps we could add a note about its origins to the file. > > There is already a note in the comment section of that file. Ah, not sure how I missed that. > > Also, shouldn't Ø and Æ be handled in the same way? > > What do you mean by “handled in the same way”? After applying the patch we would have different kind of rules for Ø (U+00D6) and Æ (U+00C6): locales/translit_combining:<U00D8> <U004F> locales/translit_neutral:<U00D8> "<U004F><U0045>" locales/translit_combining:<U00C6> "<U0041><U0045>" locales/translit_neutral:<U00C6> "<U0041><U0045>" > > Looking at translit_neutral in more detail, I think it's actually wrong > > place for letters, it should contain non-letters only and if specific rules > > are needed for letters like Ø or Æ, those should be added directly in locale > > files (so the patch discussed in bug 15593 should have not been applied to > > translit_neutral after all). This would also mean that the special rules in > > the generator for cases like EM DASH and EN DASH should probably end up to > > translit_neutral not translit_combining. > > My guess is that the purpose of translit_neutral is to contain > transliterations which are locale “neutral”, i.e. are the same for > all locales. So I see no reason not to include letters. Yeah, outright excluding *all* letters might be too harsh for cases where it's clear what the result should be but from the discussion in bug 15593 and the above handling of Ø I got an impression translit_neutral is probably not the right place for it? If a letter is being added to translit_combining by the generator isn't it then better to have it there than in the manually created translit_neutral? I see that i18n includes translit_neutral, not sure does that impose some requirements in any way. > > > > but some characters (like U+00D6, Ö) have decomposition defined in > > > > Unicode but not in glibc. > > > > > > glibc had this already in translit_combining: > > > > > > (was already there, not added by my patch, it is generated from > > > UnicodeData.txt by decomposing to U+004F U+0308 and then stripping the > > > combining character U+0308). > > > > Yes, I think what I meant to say was that the decomposition to U+004F U+0308 > > was missing but as you point out it is defined in some locales where it > > would be needed. Btw, I wonder should U+00D6 actually decompose to U+004F > > U+00A8 after U+004F U+0308 in those locales? > > Ö -> O¨ > > Why? Is that a reasonable transliteration? It throws away less > information but I think it is common practice to transliterate Ö > just as O in English for example. I was merely speculating on this, perhaps we can forget this part. Thanks. This fixes bug 16061. My apologies to Mike Fabian for failing it use --author, the ChangeLog still credits you directly for these changes. commit dd8e8e547647bf7a3f6feb816a848a846feeaf14 Author: Carlos O'Donell <carlos@systemhalted.org> Date: Wed Dec 9 22:27:41 2015 -0500 Update transliteration support to Unicode 7.0.0. The transliteration files are now autogenerated from upstream Unicode data. |