Re: [PING^8][PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872]

["Diego (Egor) Kobylkin" <egor at kobylkin dot com>]

On Friday, June 7, 2019 2:35 PM, Carlos O'Donell <codonell@redhat.com> wrote:
> I'd like to hear what Egor has to say about the data loss aspects.

It's quite simple really - suppose you have a list of pages in an wikipedia. 

For example there are these two entries in Russian:
1.Шема https://ru.wikipedia.org/w/index.php?title=%D0%A8%D0%B5%D0%BC%D0%B0&redirect=no 

2.Схема https://ru.wikipedia.org/wiki/%D0%A1%D1%85%D0%B5%D0%BC%D0%B0 


So you want to scrape wikipedia and them out to files: Шема.txt and Схема.txt
But the target system doesn't support Russian locale and so you must transliterate the filenames.
 

If "Ш"->"Sh" and "Сх"->"Sh", both of them will be written into the same file "Shema.txt". With no other special handing the first file will be overwritten and its data lost.

If "Ш"->"SH" and "Сх"->"Sh" - there will be two separate files 1. SHema.txt 2. Shema.txt . No data loss in this case. 


We cant exclude all data loss scenarios but at least shouldn't knowingly let the most basic ones happen just because how SHema looks. Translit is mostly a technical field at least in glibc so the aesthetics would be the last thing I would care about here. 


Anyway I'm all for committing the patch this way or another and opening a new bug should anyone complain about Sh/SH. Until now we had a hard time getting any input from any outsider on this issue. I guess de-facto I am the only end-user that has an opinion on this :-)

Bests,
Egor Kobylkin 


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, June 7, 2019 2:35 PM, Carlos O'Donell <codonell@redhat.com> wrote:

> On 6/7/19 6:52 AM, Rafal Luzynski wrote:
> 

> > 7.06.2019 02:57 Carlos O'Donell codonell@redhat.com wrote:
> > 

> > > On 6/6/19 5:31 PM, Rafal Luzynski wrote:
> > > 

> > > > > > Possible answers (Cyrillic -> Latin Extended -> ASCII):
> > > > > > 

> > > > > > 1.  "Ш" -> "Š" -> "SH"
> > > > > >     e.g.: "Шема" -> "Šema" -> "SHema"
> > > > > >     "Схема" ----------> "Shema"
> > > > > >     

> > > > > > 2.  "Ш" -> "Š" -> "Sh"
> > > > > >     e.g.: "Шема" -> "Šema" -> "Shema"
> > > > > >     "Схема" ----------> "Shema"
> > > > > >     

> > > > > > 

> > > > > > Personally I don't like the answer 1. because "SHema" looks weird
> > > > > > to me. Egor in turn does not like the answer 2. because the output
> > > > > > string becomes ambiguous.
> > > > > > Should we maybe have a smart algorithm which would select the title
> > > > > > case or the upper case of the output characters depending on the
> > > > > > context in the word? Note that it would not resolve the problem of
> > > > > > the output text being ambiguous.
> > > > > 

> > > > > It seems clear that there is no one right/wrong answer but it's a
> > > > > matter
> > > > > of preference, especially the way this currently works. It might be an
> > > > > improvement to output (for instance) SH instead of Sh if all the other
> > > > > letters of a word are upper-case as well but not sure what would help
> > > > > with the result being unambiguous.
> > > > 

> > > > I think you refer to the idea of implementing a smart algorithm which
> > > > would
> > > > adapt the lower/upper case depending on the context but indeed it would
> > > > not resolve the problem of ambiguity.
> > > > So, the smart algorithm aside, what should be the preferred
> > > > transliteration
> > > > rule?
> > > 

> > > I have a weak preference for 1. However, I would change my preference if
> > > someone showed me existing prior implementations that did 1 or 2.
> > 

> > uconv implements a smart algorithm to adjust the upper/lower case:
> > ==================================================================
> > $ echo "Схема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > Skhema
> > $ echo "Шема" | uconv -f UTF-8 -t ASCII -x Russian-Latin/BGN
> > Shema
> > $ echo "ШЕМА" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > SHEMA
> > $ echo "ШЕма" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > SHEma
> > 

> > $ echo "Ш Ема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> > SH Yema
> > 

> > ===================================================================
> > 

> > Also for them it is easier because they decided that "Х" should be
> > transliterated to "KH" (I think this is the common thing when
> > transliterating to English) while ISO 9 says it should be transliterated
> > to "H" and GOST says it should be "X". We can't implement this
> > fallback in glibc because the glibc algorithm is very simple.
> 

> Sigh
> 

> I should have known you could find enough examples that contradict
> eachother :-)
> 

> I'd like to hear what Egor has to say about the data loss aspects.
> 

> -------------------------------------------------------------------------------------------------------------------------------------------------------------
> 

> Cheers,
> Carlos.

Attachment: publickey - egor@kobylkin.com - 0x01FEB4E8.asc
Description: application/pgp-keys

Attachment: signature.asc
Description: OpenPGP digital signature