The collation update to the new ISO file made all locales almost twice as big

[Carlos O'Donell]

On 03/05/2018 06:40 AM, Zack Weinberg wrote:
> On Mon, Mar 5, 2018 at 8:39 AM, Mike FABIAN <mfabian@redhat.com> wrote:
>> I added this to the wiki for hte 2.28 release notes:
>> https://sourceware.org/glibc/wiki/Release/2.28#The_locale-archive_file_is_much_bigger
> 
> Thanks for writing this up.
> 
>> As LC_COLLATE makes up the bulk of the locale data, the size of
>> the locales increased a lot. The locale-archive file which contains the
>> data for all localess grew from 126 MiB to 206 Mib.
> 
> I wonder if we should spend some time thinking about ways to compact
> this data or factor it out.  I realize it's not as simple as putting
> the compiled form of iso14651_t1_common in its own file that all the
> locales refer to, because of the "locale specific rules", but maybe it
> could be _almost_ that simple?  Alternatively, perhaps some simple
> compression could be applied?
 
Right, the problem is that the tables and weights for collation are built
as a singular set, and any additional rules would perturb the tables and
their weights. However, I haven't looked that closely at seeing how much
of the 3-level tables are the same across similar locales with variant
collation rules.

Note that the a user is free to delete certain locales from the 
locale-archive, so we must be able to revert such changes with minimal
metadata overhead.

I'm going to go back to C.UTF-8 in a couple of weeks to look over finalizing
those changes and the full code-point sorting fixes I have, and I'll see
if I can come up with any ideas.

-- 
Cheers,
Carlos.