[PATCHv2] Update the localedata/locales/translit_* files to Unicode 7.0.0

[Mike FABIAN]

This is an update to my earlier patches:

https://sourceware.org/ml/libc-alpha/2015-04/msg00361.html

Updates:

    - transliteration rules for da, nb, nn, and sv locales
      added to transliterate for example "ö" to "oe" in these
      locales because the "neutral" transliteration should be
      "ö" to "o" (For example in English, coöperation as used in
      http://www.newyorker.com/humor/borowitz-report/obama-putin-agree-never-to-speak-to-each-other-again
      should be transliterated to "cooperation", not "cooeperation").
      This should fix [BZ #89].

    - lots of stuff added to translit_neutral
    - some more tweaks to the script generating the translit files
      generated from Unicode

I tested the patches on Fedora 22.

Can somebody review this please?

----------------------------------------------------------------------

The attached file updates these translit files to Unicode 7.0.0:

    locales/translit_circle
    locales/translit_cjk_compat
    locales/translit_combining
    locales/translit_compat
    locales/translit_font  
    locales/translit_fraction

it also contains lots of manual updates to

    locales/translit_neutral

now, many of them taken from 

    http://unicode.org/cldr/trac/browser/trunk/common/transforms/Latin-ASCII.xml

It does *not* update these translit files:

    locales/translit_cjk_variants
    locales/translit_hangul
    locales/translit_narrow
    locales/translit_small
    locales/translit_wide

because translit_cjk_variants is apparently not generated from Unicode
data.

The other files, translit_hangul, translit_narrow, translit_small,
translit_wide are generated but they would not change when using Unicode
7.0.0 data, nothing seems to have changed in Unicode affecting these
files. I could add scripts to generate these as well, but they would
just reproduce the current files.  Maybe I should do that nevertheless,
just to be able to see if something changes in future (quite unlikely, I
think).

Some code was duplicated in utf8_gen.py and utf8_compatibility.py,
Alexandre Oliva had already suggested to split this into an extra file.
As the new generator scripts added by this patch needed this code
again I saw that Alexandre was right and did put the reusable code
into an extra file unicode_utils.py.

Not everything in the generated translit_* files could be reproduced
exactly from Unicode data, the were some manual additions in the files
(which were not mentioned in the comments on top of these files,
the “grep” and “sed” expressions mentioned in the comments reproduce
most of the contents of these files but not everything).

Where the manual additions seemed to make sense, I added manual
hacks to the new generator scripts gen_translit_*.py to reproduce
these manual additons as well.

-- 
Mike FABIAN <mfabian@redhat.com>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Remove-duplicate-transliterations-for-U-0152-and-U-0.patch
Type: text/x-patch
Size: 944 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20150615/450550eb/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Addition-and-fixes-for-translit_neutral.patch
Type: text/x-patch
Size: 13696 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20150615/450550eb/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Update-the-translit-files-to-Unicode-7.0.0.patch
Type: text/x-patch
Size: 201160 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20150615/450550eb/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Add-transliteration-rules-for-da-nb-nn-and-sv-locale.patch
Type: text/x-patch
Size: 4067 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20150615/450550eb/attachment-0003.bin>