Help needed reviewing Cyrillic -> ASCII transliteration [BZ #2872]

[Rafal Luzynski]

Hi,

Egor provided as many as 11 versions of the patch fixing bug 2872
(Transliteration Cyrillic -> ASCII fails [1]).  I still have trouble
deciding which version is the best because we have some disagreements
about how to implement this.  I need more feedback from more experienced
maintainers.  I am afraid that my reviews so far only made Egor do
more unnecessary work.

Here are some of my questions.  Please note that they don't require any
knowledge of Cyrillic script so everybody are welcome to provide their
opinion.

* Should we take the title of the bug literally and provide the
  transliteration exclusively to plain ASCII or should we support the
  transliteration to extended Latin (with some diacritic characters,
  as per ISO 9 [2]) and support plain ASCII only as a fallback?
* Should we agree for Cyrillic -> extended Latin -> ASCII even if the
  ASCII fallback does not fully conform with any existing standard?
* Should we implement Cyrillic -> plain ASCII as per GOST System B [3]
  and skip extended Latin if it is impossible to handle both for standards
  technical reasons?
* Is the C builtin locale the correct place to put this transliteration?
  If yes, should we think about including the support of other alphabets
  as well (like extended Latin -> plain ASCII, Greek -> Latin, and so on)
  ever in future?
* Should the Cyrillic transliteration work in every locale (possibly with
  few exceptions) or should we require that a locale actually using
  Cyrillic script must be used? (E.g., should it work when ru_RU is not
  installed? should it work if en_US is the only locale installed? Should
  it work when no locale is installed, even en_US?)
* Is it required that transliteration produces unambiguous output which
  means that two different original strings never produce the same result?
  (As a consequence, the reverse transliteration could be possible).

Additionally we have a disagreement about how should we handle the case
when a single original uppercase character transliterates into a digraph
in ASCII.  Should both ASCII characters be uppercase (which is good for
all uppercase strings and also good to emphasize that the original character
was single rather than two separate characters which accidentally
transliterate into two characters making a digraph) or should only the first
ASCII character be uppercase (which is good for the titlecase words which is
common in natural texts)?  An example is "Ш" - should it be "SH" or "Sh"?
Note that "Сх" may also produce "Sh" ("S" + "h" -> "Sh").

We are lucky that some of existing glibc locales already handle
transliteration
from Cyrillic to Latin, for example sr_RS and uk_UA.  Unfortunately, they
follow their national standards rather than ISO or GOST so they cannot
be copied directly to ru_RU or applied universally to all locales.

Also, taking Egor's work into account, can we include this bug into the
list of desirable to be fixed in 2.29?

Regards,

Rafal

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=2872
[2] https://en.wikipedia.org/wiki/ISO_9
[3] https://en.wikipedia.org/wiki/ISO_9#GOST_7.79_System_B