Re: [PATCH] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] re-submission for 2.29

[Egor Kobylkin <egor at kobylkin dot com>]

The culprits were the "" around the "<U0423><U0301>" (<U00DA>) and
"<U0443><U0301>" (<U00FA>).
It works now with
% CYRILLIC UNDEFINED
<U0423><U0301> <U00DA>;"<U0055><U0060>"
% CYRILLIC UNDEFINED
<U0443><U0301> <U00FA>;"<U0075><U0060>"

The <U0301> is "combining" and obviously it doesn't work if enclosed in
quotes with the letter codepoint. Please let me know if there is another
explanation.

I will now make those changes and generate the patch itself.
Egor

On 09.10.2018 15:18, Egor Kobylkin wrote:
> Hi,
> 
> I have now implemented all the changes requested for translit_cyrillic
> file but started hitting what seems like a bug:
> 
> - If the line <U0425> <U0048>;<U0058> is present in translt_cyrillic the
> locale compilation fails i.e. grep CYRILLIC < $testfile |
> LOCPATH=$workdir/compiled_locales/"$locale"/ LC_ALL="$locale".UTF-8
> iconv -f UTF-8 -t ASCII//TRANSLIT is hanging frozen.
> 
> - If the line <U0425> <U0048>;<U0058> is absent from translit_cyrillic
> everything works, just the transliteration of <U0425> fails as expected
> (? is displayed)
> 
> - If translit_cyrillic contains <U0425> <U0048>;<U0058> as the _only_
> line the transliteration of <U0425> works again (others as ?).
> 
> Would you have any idea into what direction should I look? The new
> translit_cyrillic is attached.
> 
> (<U0425> is % CYRILLIC CAPITAL LETTER HA)
> 
> Best regards,
> Egor
> 
> On 09.10.2018 01:35, Egor Kobylkin wrote:
>> On 09.10.2018 00:23, Rafal Luzynski wrote:
>>> 8.10.2018 14:40 Marko Myllynen <myllynen@redhat.com> wrote:
>>>> Hi,
>>>>
>>>> Thanks for the update. I have few mostly cosmetic comments below,
>>>> hopefully we'll hear from others whether they agree with this direction.
>>>>
>>
>> Yeah, the earlier we have feedback the more productive we are. I'd be
>> happy to get much feedback on this as early as possible. So please
>> everybody concerned please chime in.
>>
>>>
>>>> - No duplicates:
>>>>
>>>> % CYRILLIC SMALL LETTER IE
>>>> <U0435> <U0065>; <U0065>
>>>>
>>>> should become:
>>>>
>>>> % CYRILLIC SMALL LETTER IE
>>>> <U0435> <U0065>
>>>>
>>>> - There are few issues with the definitions:
>>>>
>>>> % CYRILLIC CAPITAL LETTER U
>>>> <U0423> <U0055>; <U0055>
>>>> % CYRILLIC UNDEFINED
>>>> <U0423><U0423> <U00DA>; "<U0055><U0060>"
>>>>
>>>> % CYRILLIC SMALL LETTER U
>>>> <U0443> <U0075>; <U0075>
>>>> % CYRILLIC UNDEFINED
>>>> <U0443><U0443> <U00FA>; "<U0075><U0060>"
>>>
>>> Are the duplicates here because some Cyrillic letters may have multiple
>>> Latin transliterations depending on the context, for example Cyrillic IE
>>> must be transliterated sometimes as "e", sometimes as "ie", sometimes
>>> as "ye" or "je"?  Can we provide rules for groups of characters instead?
>> No, the duplicates are just by design of my line generating logic. I
>> have fixed (removed) them. The varying transcription between
>> languages/locales can not be handled in one file at all as far as I
>> understood.
>>
>>>
>>>> I wonder would it be possible to automate generation of this file so
>>>> that issues like the above could avoided? But perhaps that could be the
>>>> next step once this initial patch lands.
>>
>> I am generating the content part of the translit_cyrillc from the
>> LibreOffice Spreadsheet. Not sure if you had time to view it by now?
>> https://sourceware.org/bugzilla/attachment.cgi?id=11299
>>
>> Anyway I have just fixed the issues identified by Marko above in that
>> spreadsheet. I will do the changes for the below request and then upload
>> the new translit_cyrillic file to the bugzilla.
>>
>>
>>>> - Please add the standard glibc locale header (see the existing
>>>> translit_* files for reference)
>>>> - Consider wrapping the header lines at or around column 70-72
>>>> - Consider describing which characters, character ranges, or blocks are
>>>> supported (perhaps also describe why some of those are not included, see
>>>> e.g. https://en.wikipedia.org/wiki/Cyrillic_script_in_Unicode)
>>>> - Please remove trailing whitespaces and spaces after ;
>>>
>>> Thanks for this, Marko.  While at this, in the ChangeLog and in the commit
>>> message these paths:
>>>
>>> 	* locales/aa_DJ: likewise
>>>
>>> 1. Should be a relative path starting in the root directory of glibc
>> source,
>>>    that is: "* localedata/locales/aa_DJ".
>>> 2. Should be "Likewise." (starting with an uppercase and ending with a
>> dot).
>>
>> will do.
>>
>> Bests,
>> Egor
>>
>