[RFC] higher level tool for generating locale data needed

[Mike Frysinger]

On 14 Oct 2015 11:33, Carlos O'Donell wrote:
> On 10/14/2015 10:44 AM, Mike Frysinger wrote:
> > requirements:
> > - generate in parallel (generating 400+ locales in serial is sloooow)
> > - respect a config file (let's just use /etc/locale.gen as it too is standard)
> > - option to automatically generate all available locales (in case config is 
> >   empty)
> > - option to generate locales with the user-friendly name
> >   e.g. this is good: locale-gen en_US.UTF-8
> >   e.g. this is bad: localedef -i en_US -f UTF-8 en_US.UTF-8
> > 
> > nice-to-have:
> > - option to incrementally sync locale state to a config by either deleting or
> >   adding locales
> 
> This is a good start.
> 
> The only problem I see is that your mixing two things:
> 
> (a) Maintaining local state about which locales are selected by the user.
> 
>     (a.1) /etc/locale.gen is a local user choice saved to a data store (file)
> 
>     (a.2) stat synchronization.
> 
> with
> 
> (b) The process of generate locales.

the way locale-gen works in distros today is they simply call localedef to do
the work.  so locale-gen in the end merely figures out which locales to generate
before offloading everything to localedef.  i would expect the locale-gen code
in glibc to do the same thing either at the fork+exec level or simply by sharing
the object code.

> To give you some context around Fedora and Red Hat.
> 
> Fedora and Red Hat do not wish to rebuild the locale on the remote side.
> We want to send pre-built pre-tested locales to the installed system.
> This means that the above mechanism needs to express in some way that there
> are two parts of the equation: generation, and update.
> 
> We know generation and update are important because after every glibc update
> we need to generate (or skip this step in Fedora) and upgrade (all distros do
> this).
> 
> My only suggestion is that you split the discussion into two parts and try
> to address both aspects separately, this would allow Fedora to skip the generation
> discussions, but help out with the update/upgrade process.

my "incrementally sync" bullet point might have been more vague than i realized.
i was thinking of it in terms of complete locales rather than finer grained.  so
if you have set(A B) installed but the config wants set(A C), locale-gen would
take care of deleting set(B) and generating set(C).  it would not look to see if
set(A) needs to be refreshed due to internal changes.  so in that light, it is
at the same level as locale selection/generation.

i agree though that being able to detect differences between versions would
be useful.  either when the binary format changes, or if the data itself was
updated.  so it'd be more of a --refresh option where you'd be given a set of
inputs and the tool would determine which ones (if any) would need updating.

> Lastly, consider that a distribution may not ship locale source nor charmap
> source, therefore (b) should document explicitly what it needs to operate,
> and (a) should certainly not depend on any sources.

the only thing locale-gen needs today to work are:
- localedata in /usr/share/i18n/locales/
- charmap data in /usr/share/i18n/charmaps/
- a list of locales to generate such as /usr/share/i18n/SUPPORTED

this doesn't mean the archive can't be generated ahead of time and shipped,
just that when locale-gen runs, it needs those files.  i don't see any way
around those requirements purely by the nature of localedef/locale-gen.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20151014/3f359c4e/attachment.sig>