codeset problems in wprintf and wcsftime

[Corinna Vinschen]

On Feb 20 16:31, Andy Koppe wrote:
> Corinna Vinschen:
> > while working on finalizing locale support for Cygwin it suddenly
> > occured to me that we have a problem in wprintf and wcsftime.
> >
> > Let's assume a funny combination of localization variables in the user's
> > environment:
> >
> > Â LANG=de_DE.utf8
> > Â LC_TIME=ja_JP.eucjp
> > Â LC_NUMERIC=en_US.iso88591
> >
> > Yes, it's pretty unlikely, but nevertheless possible and valid.
> >
> > So, at setlocale time we read and store the localized strings in the
> > codeset specified by the localization variable:
> >
> > Â - __locale_charset() Â  Â  Â  Â  Â  Â  returns UTF-8
> > Â - __get_current_time_locale() Â  Â returns data stored in EUC-JP
> > Â - __get_current_numeric_locale() returns data stored in ISO-8859-1
> > Â - localeconv() Â  Â  Â  Â  Â  Â  Â  Â  Â  returns with decimal_point and
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  thousands_sep stored in ISO-8859-1,
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  and all other strings from the
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  LC_MONETARY category in UTF-8.
> > Â - nl_langinfo() Â  Â  Â  Â  Â  Â  Â  Â  Â CODESET is UTF-8,
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  strings from the LC_TIME category are
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  returned in EUC-JP,
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  strings from LC_MESSAGES are returned
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  in UTF-8
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  RADIXCHAR and THOUSEP are returned in
> > Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ISO-8859-1.
> >
> > This is no problem at all as long as you call the multibyte variations
> > printf and strftime, the user gets what she asked for, and who are we
> > to ask the user for the reason behind this choice.
> 
> Have you verified that the user does indeed get a mix of charsets when
> doing this on glibc?

Look at the output of the locale(1) tool:

  $ export LANG=de_DE.utf8
  $ export LC_TIME=ja_JP.eucjp
  $ export LC_NUMERIC=en_US.iso88591
  $ locale -k LC_CTYPE LC_TIME LC_NUMERIC | egrep 'codeset|charmap'
  charmap="UTF-8"
  time-codeset="EUC-JP"
  numeric-codeset="ISO-8859-1"

> I'm asking because another alternative to the solutions you outlined
> might be to store those strings as wchar versions only, to be used
> directly in wprintf and converted to the LC_CTYPE character set when
> needed in printf. That way, the user would always get readable output.

The multibyte variations are still much more often used than the
widechar functions.  I would prefer not to move the conversion burden
into these more often used functions.

> > - Store the charset not only for LC_CTYPE, but for each localization
> > Â category, and provide a function to request the charset.
> > Â This also requires to store the associated multibyte to widechar
> > Â conversion functions, obviously, and to call the correct functions
> > Â from wprintf and wcftime.
> >
> > - Redefine the locale data structs so that they contain multibyte and
> > Â widechar representations of all strings. Â Use the multibyte strings
> > Â in the multibyte functions, the widechar strings in the widechar
> > Â functions.
> >
> > Personally I'd prefer the second approach.
> 
> Agreed. Sounds like less overhead.


Corinna

-- 
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat