Internationalization

Overview

Internationalization support is controlled by the LANG and LC_xxx environment variables. You can set all of them but Cygwin itself only honors the variables LC_ALL, LC_CTYPE, and LANG, in this order, according to the POSIX standard. The content of these variables should follow the POSIX standard for a locale specifier. The correct form of a locale specifier is

  language[[_TERRITORY][.charset][@modifier]]

"language" is a lowercase two character string per ISO 639-1, or, if there is no ISO 639-1 code for the language (for instance, "Lower Sorbian"), a three character string per ISO 639-3.

"TERRITORY" is an uppercase two character string per ISO 3166, charset is one of a list of supported character sets. The modifier doesn't matter here (though some are recognized, see below). If you're interested in the exact description, you can find it in the online publication of the POSIX manual pages on the homepage of the Open Group.

Typical locale specifiers are

  "de_CH"	   language = German, territory = Switzerland, default charset
  "fr_FR.UTF-8"    language = french, territory = France, charset = UTF-8
  "ko_KR.eucKR"    language = korean, territory = South Korea, charset = eucKR
  "syr_SY"         language = Syriac, territory = Syria, default charset

If the locale specifier does not follow the above form, Cygwin checks if the locale is one of the locale aliases defined in the file /usr/share/locale/locale.alias. If so, and if the replacement localename is supported by the underlying Windows, the locale is accepted, too. So, given the default content of the /usr/share/locale/locale.alias file, the below examples would be valid locale specifiers as well.

  "catalan"        defined as "ca_ES.ISO-8859-1" in locale.alias
  "japanese"       defined as "ja_JP.eucJP"      in locale.alias
  "turkish"        defined as "tr_TR.ISO-8859-9" in locale.alias

The file /usr/share/locale/locale.alias is provided by the gettext package under Cygwin.

At application startup, the application's locale is set to the default "C" or "POSIX" locale. Under Cygwin 1.7.2 and later, this locale defaults to the ASCII character set on the application level. If you want to stick to the "C" locale and only change to another charset, you can define this by setting one of the locale environment variables to "C.charset". For instance

  "C.ISO-8859-1"

Note

The default locale in the absence of the aforementioned locale environment variables is "C.UTF-8".

Windows uses the UTF-16 charset exclusively to store the names of any object used by the Operating System. This is especially important with filenames. Cygwin uses the setting of the locale environment variables LC_ALL, LC_CTYPE, and LANG, to determine how to convert Windows filenames from their UTF-16 representation to the singlebyte or multibyte character set used by Cygwin.

The setting of the locale environment variables at process startup is effective for Cygwin's internal conversions to and from the Windows UTF-16 object names for the entire lifetime of the current process. Changing the environment variables to another value changes the way filenames are converted in subsequently started child processes, but not within the same process.

However, even if one of the locale environment variables is set to some other value than "C", this does only affect how Cygwin itself converts filenames. As the POSIX standard requires, it's the application's responsibility to activate that locale for its own purposes, typically by using the call

  setlocale (LC_ALL, "");

early in the application code. Again, so that this doesn't get lost: If the application calls setlocale as above, and there is none of the important locale variables set in the environment, the locale is set to the default locale, which is "C.UTF-8".

But what about applications which are not locale-aware? Per POSIX, they are running in the "C" or "POSIX" locale, which implies the ASCII charset. The Cygwin DLL itself, however, will nevertheless use the locale set in the environment (or the "C.UTF-8" default locale) for converting filenames etc.

When the locale in the environment specifies an ASCII charset, for example "C" or "en_US.ASCII", Cygwin will still use UTF-8 under the hood to translate filenames. This allows for easier interoperability with applications running in the default "C.UTF-8" locale.

Starting with Cygwin 1.7.2, the language and territory are used to fetch locale-dependent information from Windows. If the language and territory are not known to Windows, the setlocale function fails.

The following modifiers are recognized. Any other modifier is simply ignored for now.

  • For locales which use the Euro (EUR) as currency, the modifier "@euro" can be added to enforce usage of the ISO-8859-15 character set, which includes a character for the "Euro" currency sign.

  • The default script used for all Serbian language locales (sr_BA, sr_ME, sr_RS, and the deprecated sr_CS and sr_SP) is cyrillic. With the "@latin" modifier it gets switched to the latin script with the respective collation behaviour.

  • The default charset of the "be_BY" locale (Belarusian/Belarus) is CP1251. With the "@latin" modifier it's UTF-8.

  • The default charset of the "tt_RU" locale (Tatar/Russia) is ISO-8859-5. With the "@iqtelif" modifier it's UTF-8.

  • The default charset of the "uz_UZ" locale (Uzbek/Uzbekistan) is ISO-8859-1. With the "@cyrillic" modifier it's UTF-8.

  • There's a class of characters in the Unicode character set, called the "CJK Ambiguous Width" characters. For these characters, the width returned by the wcwidth/wcswidth functions is usually 1. This can be a problem with East-Asian languages, which historically use character sets where these characters have a width of 2. Therefore, wcwidth/wcswidth return 2 as the width of these characters when an East-Asian charset such as GBK or SJIS is selected, or when UTF-8 is selected and the language is specified as "zh" (Chinese), "ja" (Japanese), or "ko" (Korean). This is not correct in all circumstances, hence the locale modifier "@cjknarrow" can be used to force wcwidth/wcswidth to return 1 for the ambiguous width characters.

How to set the locale

  • Assume that you've set one of the aforementioned environment variables to some valid POSIX locale value, other than "C" and "POSIX". Assume further that you're living in Japan. You might want to use the language code "ja" and the territory "JP", thus setting, say, LANG to "ja_JP". You didn't set a character set, so what will Cygwin use now? Starting with Cygwin 1.7.2, the default character set is determined by the default Windows ANSI codepage for this language and territory. Cygwin uses a character set which is the typical Unix-equivalent to the Windows ANSI codepage. For instance:

      "en_US"		ISO-8859-1
      "el_GR"		ISO-8859-7
      "pl_PL"		ISO-8859-2
      "pl_PL@euro"		ISO-8859-15
      "ja_JP"		EUCJP
      "ko_KR"		EUCKR
      "te_IN"		UTF-8
    
  • You don't want to use the default character set? In that case you have to specify the charset explicitly. For instance, assume you're from Japan and don't want to use the japanese default charset EUC-JP, but the Windows default charset SJIS. What you can do, for instance, is to set the LANG variable in the mintty Cygwin Terminal in the "Text" section of its "Options" dialog. If you're starting your Cygwin session via a batch file or a shortcut to a batch file, you can also just set LANG there:

      @echo off
    
      C:
      chdir C:\cygwin\bin
      set LANG=ja_JP.SJIS
      bash --login -i
    

    Note

    For a list of locales supported by your Windows machine, use the new locale -a command, which is part of the Cygwin package. For a description see the section called “locale”

    Note

    For a list of supported character sets, see the section called “List of supported character sets”

  • Last, but not least, most singlebyte or doublebyte charsets have a big disadvantage. Windows filesystems use the Unicode character set in the UTF-16 encoding to store filename information. Not all characters from the Unicode character set are available in a singlebyte or doublebyte charset. While Cygwin has a workaround to access files with unusual characters (see the section called “Filenames with unusual (foreign) characters”), a better workaround is to use always the UTF-8 character set.

    UTF-8 is the only multibyte character set which can represent every Unicode character.

      set LANG=es_MX.UTF-8
    

    For a description of the Unicode standard, see the homepage of the Unicode Consortium.

The Windows Console character set

Sometimes the Windows console is used to run Cygwin applications. While terminal emulations like the Cygwin Terminal mintty or xterm have a distinct way to set the character set used for in- and output, the Windows console hasn't such a way, since it's not an application in its own right.

This problem is solved in Cygwin as follows. When a Cygwin process is started in a Windows console (either explicitly from cmd.exe, or implicitly by, for instance, running the C:\cygwin\Cygwin.bat batch file), the Console character set is determined by the setting of the aforementioned internationalization environment variables, the same way as described in the section called “How to set the locale”.

What is that good for? Why not switch the console character set with the applications requirements? After all, the application knows if it uses localization or not. However, what if a non-localized application calls a remote application which itself is localized? This can happen with ssh or rlogin. Both commands don't have and don't need localization and they never call setlocale. Setting one of the internationalization environment variable to the same charset as the remote machine before starting ssh or rlogin fixes that problem.

Potential Problems when using Locales

You can set the above internationalization variables not only when starting the first Cygwin process, but also in your Cygwin shell on the fly, even switch to yet another character set, and yet another. In bash for instance:

  bash$ export LC_CTYPE="nl_BE.UTF-8"

However, here's a problem. At the start of the first Cygwin process in a session, the Windows environment is converted from UTF-16 to UTF-8. The environment is another of the system objects stored in UTF-16 in Windows.

As long as the environment only contains ASCII characters, this is no problem at all. But if it contains native characters, and you're planning to use, say, GBK, the environment will result in invalid characters in the GBK charset. This would be especially a problem in variables like PATH. To circumvent the worst problems, Cygwin converts the PATH environment variable to the charset set in the environment, if it's different from the UTF-8 charset.

Note

Per POSIX, the name of an environment variable should only consist of valid ASCII characters, and only of uppercase letters, digits, and the underscore for maximum portability.

Symbolic links, too, may pose a problem when switching charsets on the fly. A symbolic link contains the filename of the target file the symlink points to. When a symlink had been created with older versions of Cygwin, the current ANSI or OEM character set had been used to store the target filename, dependent on the old CYGWIN environment variable setting codepage (see the section called “Obsolete options”. If the target filename contains non-ASCII characters and you use another character set than your default ANSI/OEM charset, the target filename of the symlink is now potentially an invalid character sequence in the new character set. This behaviour is not different from the behaviour in other Operating Systems. So, if you suddenly can't access a symlink anymore which worked all these years before, maybe it's because you switched to another character set. This doesn't occur with symlinks created with Cygwin 1.7 or later.

Another problem you might encounter is that older versions of Windows did not install all charsets by default. If you are running Windows XP or older, you can open the "Regional and Language Options" portion of the Control Panel, select the "Advanced" tab, and select entries from the "Code page conversion tables" list. The following entries are useful to cygwin: 932/SJIS, 936/GBK, 949/EUC-KR, 950/Big5, 20932/EUC-JP.

List of supported character sets

Last but not least, here's the list of currently supported character sets. The left-hand expression is the name of the charset, as you would use it in the internationalization environment variables as outlined above. Note that charset specifiers are case-insensitive. EUCJP is equivalent to eucJP or eUcJp. Writing the charset in the exact case as given in the list below is a good convention, though.

The right-hand side is the number of the equivalent Windows codepage as well as the Windows name of the codepage. They are only noted here for reference. Don't try to use the bare codepage number or the Windows name of the codepage as charset in locale specifiers, unless they happen to be identical with the left-hand side. Especially in case of the "CPxxx" style charsets, always use them with the trailing "CP".

This works:

  set LC_ALL=en_US.CP437

This does not work:

  set LC_ALL=en_US.437

You can find a full list of Windows codepages on the Microsoft MSDN page Code Page Identifiers.

    Charset               Codepage
    -------------------   -------------------------------------------
    ASCII                 20127 (US_ASCII)

    CP437                   437 (OEM United States)
    CP720                   720 (DOS Arabic)
    CP737                   737 (OEM Greek)
    CP775                   775 (OEM Baltic)
    CP850                   850 (OEM Latin 1, Western European)
    CP852                   852 (OEM Latin 2, Central European)
    CP855                   855 (OEM Cyrillic)
    CP857                   857 (OEM Turkish)
    CP858                   858 (OEM Latin 1 + Euro Symbol)
    CP862                   862 (OEM Hebrew)
    CP866                   866 (OEM Russian)
    CP874                   874 (ANSI/OEM Thai)
    CP932		    932 (Shift_JIS, not exactly identical to SJIS)
    CP1125                 1125 (OEM Ukraine)
    CP1250                 1250 (ANSI Central European)
    CP1251                 1251 (ANSI Cyrillic)
    CP1252                 1252 (ANSI Latin 1, Western European)
    CP1253                 1253 (ANSI Greek)
    CP1254                 1254 (ANSI Turkish)
    CP1255                 1255 (ANSI Hebrew)
    CP1256                 1256 (ANSI Arabic)
    CP1257                 1257 (ANSI Baltic)
    CP1258                 1258 (ANSI/OEM Vietnamese)

    ISO-8859-1            28591 (ISO-8859-1)
    ISO-8859-2            28592 (ISO-8859-2)
    ISO-8859-3            28593 (ISO-8859-3)
    ISO-8859-4            28594 (ISO-8859-4)
    ISO-8859-5            28595 (ISO-8859-5)
    ISO-8859-6            28596 (ISO-8859-6)
    ISO-8859-7            28597 (ISO-8859-7)
    ISO-8859-8            28598 (ISO-8859-8)
    ISO-8859-9            28599 (ISO-8859-9)
    ISO-8859-10             -   (not available)
    ISO-8859-11             -   (not available)
    ISO-8859-13           28603 (ISO-8859-13)
    ISO-8859-14             -   (not available)
    ISO-8859-15           28605 (ISO-8859-15)
    ISO-8859-16             -   (not available)

    Big5                    950 (ANSI/OEM Traditional Chinese)
    EUCCN or euc-CN         936 (ANSI/OEM Simplified Chinese)
    EUCJP or euc-JP       20932 (EUC Japanese)
    EUCKR or euc-KR         949 (EUC Korean)
    GB2312                  936 (ANSI/OEM Simplified Chinese)
    GBK                     936 (ANSI/OEM Simplified Chinese)
    GEORGIAN-PS             -   (not available)
    KOI8-R                20866 (KOI8-R Russian Cyrillic)
    KOI8-U                21866 (KOI8-U Ukrainian Cyrillic)
    PT154                   -   (not available)
    SJIS                    -   (not available, almost, but not exactly CP932)
    TIS620 or TIS-620       874 (ANSI/OEM Thai)

    UTF-8 or utf8         65001 (UTF-8)