[1.7] Proposal: the filename encoding in C locale uses UTF-8 instead of SO/UTF-8

Corinna Vinschen corinna-cygwin@cygwin.com
Wed May 13 14:30:00 GMT 2009

On May 12 19:37, Corinna Vinschen wrote:
> On May 13 02:29, IWAMURO Motonori wrote:
> > I propose that the filename encoding in C locale uses UTF-8 instead of SO/UTF-8.
> > 
> > There are three reasons:
> That's an interesting thought.  Do you have a patch and, if so, did you
> try it?  Does it, for instance, help for the issue reported in the
> thread starting at http://cygwin.com/ml/cygwin/2009-05/msg00245.html?

After examining the issue Lenik reported in the above thread, I'm at 
a loss how to solve this problem in a generic way.

The problem is that the filename changes dependent on the character
set used in $LANG.  The reason is that every time a multibyte filename
has to be generated, it has to be converted from UTF-16 to multibyte.

For instance, taking one of the filename from Lenik's example.  It's
stored on the filesystem as the UTF-16 sequence \u684c \u9762.  If I set
LANG to en_US.UTF-8, a readdir(2) call returns the multibyte sequence

 0xe6 0xa1 0x8c 0xe9 0x9d 0xa2

If I set LANG to en_US.GBK, `ls' returns the filename

 0xd7 0xc0 0xc3 0xe6

And in case LANG=C, `ls' returns

 0x0e 0xe6 0xa1 0x8c 0x0e 0xe9 0x9d 0xa2

So, dependent on the character set setting in the application, the idea
of the filename differs.  That's not exactly helpful for interoperability
between different applications.

I can think of two potential solutions to fix this problem:

(1) Always return filenames in UTF-8 encoding and pretend that UTF-8
    is the way files are stored on disk.  That results in unchangable
    filenames which are always valid.
    But what if an application sets LANG="xxxx.SJIS" and tries to create
    a file using SJIS character encoding?  Should the file be created
    using the SJIS->UTF-16 conversion or should open fail with EILSEQ?
    That's not good.

(2) If none of $LC_ALL/$LC_CTYPE/$LANG is set in the environment, then
    Cygwin uses the LC_CTYPE setting which corresponds to the current
    codepage.  If one of $LC_ALL/$LC_CTYPE/$LANG is set in the environment,
    Cygwin uses that to convert pathnames.  If the application uses
    setlocale, Cygwin uses that setting to convert pathnames.

    One problem can't be solved this way:  If an application fetches
    and stores a filename, then switches the locale, and then tries
    to use the filename in another system call, the filename is
    potentially broken.

Any better ideas?


Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

More information about the Cygwin mailing list