This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PING^8][PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872]


7.06.2019 14:35 Carlos O'Donell <codonell@redhat.com> wrote:
> On 6/7/19 6:52 AM, Rafal Luzynski wrote:
> [...]
> > 
> > uconv implements a smart algorithm to adjust the upper/lower case:
> > 
> > ==================================================================
> > $ echo "Схема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > Skhema
> > 
> > $ echo "Шема" | uconv -f UTF-8 -t ASCII -x Russian-Latin/BGN
> > Shema
> > 
> > $ echo "ШЕМА" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > SHEMA
> > 
> > $ echo "ШЕма" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > SHEma
> > 
> > $ echo "Ш Ема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > SH Yema
> > ==================================================================
> > 
> > Also for them it is easier because they decided that "Х" should be
> > transliterated to "KH" (I think this is the common thing when
> > transliterating to English) while ISO 9 says it should be transliterated
> > to "H" and GOST says it should be "X".  We can't implement this
> > fallback in glibc because the glibc algorithm is very simple.
> 
> *Sigh*
> 
> I should have known you could find enough examples that contradict
> eachother :-)

Sorry about the confusion.  My aim was to demonstrate how uconv adjusts
upper/lower case depending on the context so "Ш" becomes sometimes "SH"
and sometimes "Sh".



7.06.2019 14:59 "Diego (Egor) Kobylkin" <egor@kobylkin.com> wrote:
> [...]
> It's quite simple really - suppose you have a list of pages in an
> wikipedia. 
> 
> For example there are these two entries in Russian:
> 1.Шема
> https://ru.wikipedia.org/w/index.php?title=%D0%A8%D0%B5%D0%BC%D0%B0&redirect=no
> 
> 
> 2.Схема https://ru.wikipedia.org/wiki/%D0%A1%D1%85%D0%B5%D0%BC%D0%B0 
> 
> 
> So you want to scrape wikipedia and them out to files: Шема.txt and
> Схема.txt
> But the target system doesn't support Russian locale and so you must
> transliterate the filenames.

While talking about the filesystem: I think the problem is not
that it does not support Russian locale but that it tries to
handle it and fails at this.  If the filesystem accepted any
byte string as a file name wouldn't it accept a byte string which
constructs correct Cyrillic characters in UTF-8, without any
transliteration?

> If "Ш"->"Sh" and "Сх"->"Sh", both of them will be written into the same
> file "Shema.txt". With no other special handing the first file will be
> overwritten and its data lost.
> 
> If "Ш"->"SH" and "Сх"->"Sh" - there will be two separate files 1.
> SHema.txt 2. Shema.txt . No data loss in this case. 
> [...]

The problem is exclusively in the limitation of glibc itself.
In fact no standard says that "Ш" should be transliterated as "Sh"
(or "SH") and "Х" as "H" (consequently, "Сх" as "Sh").  ISO-9
says that "Ш" should be "Š" and "Х" should be "H" (consequently,
"Сх" should be "Sh" but that would never be confused for "Ш").
GOST 7.79 says that "Ш" should be "SH" (or "Sh") and "Х" should
be "X" (consequently, ""Сх" should be "Sx").  There is no confusion
in any case.  The problem is that we can't express all these rules
in the language of glibc transliterations; the rule:

    Х    "H";"X"

will not work because it would choose a transliteration of "X" only
if "H" was not available in the target charset (which never happens)
while we want it to choose "X" if "Š" is not available.



7.06.2019 23:17 Carlos O'Donell <codonell@redhat.com> wrote:
> [...]
> I also think your point about "technical" is relevant here, nobody
> really wants to read the transliterated results, they want to read
> the original, and providing any hint about the original form has
> value.

It looks like I totally misunderstood the purpose.  I always thought
the aim is to produce a transliteration system for real natural
language texts and to achieve the same output as it would be written
by a human writer.  Which I still think is possible, at least partially
and not necessarily in the current development cycle.  If you guys
want to have only technical hint and want to relax the linguistic
rules then Egor's patches are mostly sufficient.

> In glibc we don't have any framework for an intelligent conversion.
> We would have to write specific code to handle this case and add
> it into the translit code for special handling in this case.

My suggestion was to add such an intelligent conversion.  The rule
should be simple: if a letter is followed by a lowercase it should
be a titlecase (Sh), otherwise it should be uppercase (SH).  But
this may break Egor's requirement to keep them always uppercase.

> I think we should today leave "Ш"->"SH" and "Сх"->"Sh", since it's
> the most conservative position that avoids ambiguity, and then we
> can discuss the aesthetics of this and the other impacts and solutions.
> 
> I appreciate Rafal's position, but I think being conservative here,
> even if it's not as pretty as uconv, is a good guiding idea.

Just to summarize: if you want to apply the relaxed rules, more
technical than linguistic, then I am more willing to accept these
patches.

Regards,

Rafal


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]