[Bug localedata/25036] New: Update collation order for Swedish

Wed Sep 25 22:38:00 GMT 2019

https://sourceware.org/bugzilla/show_bug.cgi?id=25036

            Bug ID: 25036
           Summary: Update collation order for Swedish
           Product: glibc
           Version: 2.32
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: sebras at gmail dot com
                CC: libc-locales at sourceware dot org
  Target Milestone: ---

[TL;DR This is about an issue with/proposed patch for Swedish localedata]

Today I tried to compile llpp git HEAD (3371788 of [1]) but one if it's
dependencies, ocaml-4.09 [2] failed to link properly due to undefined
references, see below.
[1] https://github.com/moosotc/llpp.git
[2] https://caml.inria.fr/pub/distrib/ocaml-4.09/ocaml-4.09.0.tar.xz

....
gcc -O2 -fno-strict-aliasing -fwrapv -Wall -fno-tree-vrp -g
-D_FILE_OFFSET_BITS=64 -D_REENTRANT -DCAML_NAME_SPACE 
-DOCAML_STDLIB_DIR='"/usr/local/lib/ocaml"'  -Wl,-E -o ocamlrun prims.o
libcamlrun.a -lm -ldl  -lpthread
/usr/bin/ld: prims.o:(.data.rel+0x0): undefined reference to `caml_'
/usr/bin/ld: prims.o:(.data.rel+0x1d0): undefined reference to `caml_bs'
/usr/bin/ld: prims.o:(.data.rel+0x278): undefined reference to
`caml_convert_ra'
....

In ocaml-4.09/runtime there is a shell script named gen_primitives.sh which as
input takes a set of .c-files implementing the ocaml primitives, and as output
generates a list of symbol names of all the primitives. This list is then used
to generate a .c-file named prims.c, referencing each of the primitives.
prims.c is later compiled and linked with a library compiled from the original
set of .c-files. If the list generated by gen_primitives.sh contains spurious
symbols you will get the link error I encountered above.

gen_primitives.sh uses this sed command to collect the symbols from a .c-file:

sed -n -e "s/^CAMLprim value \([a-z0-9_][a-z0-9_]*\).*/\1/p" <.c-file name
here>

When the script processes ints.c it encounters (among others) these two lines,
which result in the symbols caml_bs and caml_int_compare being listed as
present:

CAMLprim value caml_bswap16(value v)
CAMLprim value caml_int_compare(value v1, value v2)

But that is surprising since the expected result would be caml_bswap16 and
caml_in_compare! I managed to reproduce this independently of the script and
the input files by running:

echo -e 'CAMLprim value caml_bswap16(value v)\nCAMLprim value
caml_int_compare(value v1, value v2)' \
        | sed -n -e "s/^CAMLprim value \([a-z0-9_][a-z0-9_]*\).*/\1/p"

Cutting down the text and regexes used results in a smaller example:

echo -e 'bsw\nint' | sed -n -e "s/^\([a-z0-9_][a-z0-9_]*\).*/\1/p"

which gives the outputs "bs" and "int". Minimizing the test case further:

echo -e 'w\ni' | sed -n -e "s/^\([a-z0-9_][a-z0-9_]*\).*/\1/p"

yields something surprising: only "i" is printed! Next I tested with the
entire alphabet (example cut down for brevity):

echo -e 's\nt\nu\nv\nw\nx\ny\nz' | sed -n -e 's/^[a-z]$/&/p'
echo -e 's\nt\nu\nv\nw\nx\ny\nz' | sed -n -e 's/^[a-z]$/&/p'

The problem only appeared for "w" -- why? At this point the author of llpp 
suggested that it might be due to locale settings. My only locale settings are:

LANG=en_US.UTF-8
LC_COLLATE=sv_SE.UTF-8

Despite me using a predominantly English (US) desktop I do prefer if e.g. music
files whose names contain Swedish characters are sorted according to Swedish
alphabetical order. The same goes for sort and other pieces of software.

The same command with a different collation setting yielded the expected
result:

echo -e 's\nt\nu\nv\nw\nx\ny\nz' | LC_COLLATE=sv_SE.UTF-8 sed -n -e
's/^[a-z]$/&/p'
echo -e 's\nt\nu\nv\nw\nx\ny\nz' | LC_COLLATE= sed -n -e 's/^[a-z]$/&/p'

The same is true also for the prior command:

echo -e 'w\ni' | LC_COLLATE= sed -n -e "s/^\([a-z0-9_][a-z0-9_]*\).*/\1/p"

which now prints "w" and "i". Unsetting LC_COLLATE allowed me to compile ocaml
and
thus llpp successfully which resolved my original issue.

This is not the end of the story though, and does not explain why I'm filing a
bug report with glibc.

I wanted to understand why LC_COLLATE influence the regexes in sed, and why
specifically for "w"? I dug through the sed documentation [3]. I can (barely)
accept that LC_COLLATE influence regular expressions this way.
[3] http://git.savannah.gnu.org/cgit/sed.git/tree/doc/sed.texi#n5669

The problem for me being a native Swedish speaker is that w and v are
considered separate characters [4] and that Swedish alphabetical order sorts w
after v [5].
[4] https://en.wikipedia.org/wiki/Swedish_alphabet#Uncommon_letters
[5]
https://en.wikipedia.org/wiki/Alphabetical_order#Language-specific_conventions

That suggested that my localedata must have problems, so I read the contents of
/usr/share/i18n/locales/sv_SE. I found one section discussing whether v and w
are distinct characters [6] and another mentioning alphabetical sorting order
[7].
[6]
https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/locales/sv_SE;h=aa28c23776408e593890883ebb4c8d70b971fe15;hb=HEAD#l106
[7]
https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/locales/sv_SE;h=aa28c23776408e593890883ebb4c8d70b971fe15;hb=HEAD#l63

The part mentioning v and w being the same character [6] was added to glibc's
localedata in a commit in 2000 [8], before the reform (according to [5]) in
2006. The part referring to the standard collation order [7] was added in a
commit in 2017 [9].
[8]
https://sourceware.org/git/?p=glibc.git;a=commit;h=cf365734e59f708b1610815196ee2303174308cf
[9]
https://sourceware.org/git/?p=glibc.git;a=commit;h=159738548130d5ac4fe6178977e940ed5f8cfdc4

It seems even the latter commit [9] failed to take notice of the Unicode
organization's CLDR-project [10] adding a draft of the collation order reform
back in 2005 [11][12], officially published in CLDR 1.4.0 [13]. Two years later
they also changed the default collation order [14][15], officially published in
CLDR 1.5.0 [16].

[10] https://github.com/unicode-org/cldr
[11]
https://github.com/unicode-org/cldr/commit/6daf16a4fb81ffbfc91de3dd12fbdd270ca89cfd
[12]
https://github.com/unicode-org/cldr/commit/d1354eaaa147aacf9fc2e7e30c563ed5be7d8227
[13] http://cldr.unicode.org/index/downloads/cldr-1-4
[14]
https://github.com/unicode-org/cldr/commit/aaa134f4f74db91b781be1c8b9d9e9f1fd123c0e
[15]
https://github.com/unicode-org/cldr/commit/ffe1610149231d8feb5e4d9ba4bb0857f224869d
[16] http://cldr.unicode.org/index/downloads/cldr-1-5

I'd like for glibc's localedata to refer to the reformed collation order.
Mainly because of the language reform, but of course also because this would
have prevented me encountering the original build issues in llpp. :) In the
attached patch I try to address this, if there are any issues with it or
something is unclear, please let me know.

-- 
You are receiving this mail because:
You are on the CC list for the bug.