Locales in GLIBC

Contents

  1. Introduction
  2. Overview
  3. Using Locales
  4. Developing with Locales
  5. GLIBC Locale Internals
  6. Creating and Updating Locale Data
    1. Qualifications
    2. Locale File Format
      1. Comments
      2. LC_IDENTIFICATION
      3. LC_CTYPE
      4. LC_COLLATE
      5. LC_TIME
      6. LC_NUMERIC
      7. LC_MONETARY
      8. LC_MESSAGES
      9. LC_PAPER
      10. LC_MEASUREMENT
      11. LC_NAME
      12. LC_ADDRESS
      13. LC_TELEPHONE
    3. Testing Locales
    4. Contributing
    5. Miscellaneous Information
      1. Charsets
      2. Transliteration
  7. References

1. Introduction

This page provides an introduction to locales in general and a more detailed description on how to use locales (both as a user and a developer) and how to create and update actual locale definition files with the GNU C Library.

2. Overview

Locales in short are collections of language and country specific conventions allowing to adapt software to the user's preferences.

Each locale has following categories to specify related conventions which can then be selected for use by the user:

The locales of the GNU C library are named using the pattern ll_CC.SSS where ll refers to language, CC refers to country, and SSS refers to the character set to use. For example, the English language as used in Canada using the UTF-8 character set is to be referenced as en_CA.UTF-8. (For more information on characters sets, see charsets(7) and http://www.joelonsoftware.com/articles/Unicode.html.)

3. Using Locales

The user wishing to use one or more of the above categories can take them in use by defining the corresponding environment variables, for example in the ~/.i18n file which is used on many systems automatically.

There are two additional environment variables which are considered when selecting locales - LANG and LC_ALL. LANG sets the default locale for all categories which can be then overridden by defining additional selected categories. LC_ALL forces the locale for all categories and the selection cannot be overridden. Using LC_ALL is mostly recommended in scripts only (for example, to make sure characters and collation are as the developer was expecting regardless of the user's preferences), while LANG in general should be preferred for the flexibility it provides.

The following example sets the default locale as Mexican Spanish, then sets collation rules to be based on the standard C locale, and finally sets monetary formatting to follow US conventions. 

  LANG=es_MX.UTF-8
  LC_COLLATE=C
  LC_MONETARY=en_US.UTF-8

Since above LANG is defined but LC_MESSAGES is left undefined, message translations will use Mexican Spanish. The same logic goes for the other undefined categories, like LC_TIME, too.

The user can use custom locales by compiling them with localedef(1) and loading them from a directory pointed by LOCPATH; see the GNU C Library online manual at Locale Names, the locale(1) manual page, and the Testing Locales section below for details and examples.

4. Developing with Locales

The online GNU C Library manual provides a good starting point for developers creating applications supporting locales. At least the following pages are relevant:

Linux manual pages were greatly improved during 2014 and are now more or less complete, the following pages serve as a good starting point:

Application developers should note that while some of the resulting strings (like int_curr_symbol of LC_MONETARY) are required to be of certain length other resulting strings may vary, especially between locales (see this and this email for examples). Thus, testing applications with several locales is recommended to make sure different length strings do not cause inconsistencies in user experience.

5. GLIBC Locale Internals

The locale API and definitions of concrete locales are a rather individual part of the GNU C Library; related to this is the subsystem dealing with various charsets and converting between them.

6. Creating and Updating Locale Data

Please keep in mind the following before starting to work on glibc locales. If in doubt, send a question to libc-alpha/libc-locales mailing lists.

6.1. Qualifications

The glibc maintainers cannot easily judge on their own if your new version is correct.

The locale definitions are in a specific file format; it is described below and in the manual pages, please read them both for complete understanding.

6.2. Locale File Format

The locale definition are in a specific file format; most relevant notes can be found from the locale(5) manual page. POSIX also describes the format and some fields, but not all that are commonly used in glibc (e.g. week start definitions). Most numbers and strings in the glibc locale files use Unicode entity specifications instead of plain characters, for the exceptions refer to other locale files and locale(5). When working with a locale, to quickly inspect the file just do gcc -o show-ucs-data localedata/show-ucs-data.c (no need to do any build preparation for this, not even ./configure) and then just ./show-ucs-data localedata/locales/en_US to print out the data in more readable form.

One additional resource describing locale categories and category members is the ISO/IEC TR 14652:2002(E) Technical Report (PDF). However, some members are described below to address some glibc specific requirements and formatting issues. In case of doubt, please refer to manual pages, other glibc locales for examples, or send a question to libc-alpha/libc-locales mailing lists.

6.2.1. Comments

You should provide plenty of comments in the locale file, both about the individual members of each category and also any relevant references. If something is left undefined on purpose, the reason should be stated.

6.2.2. LC_IDENTIFICATION

This category is pretty much self-explaining. You should be able to fill this category by using other locales as examples.

6.2.3. LC_CTYPE

This category deals with character sets and transliteration rules both most often based on Unicode standards. It defines for example what characters are considered to be alphabetic or how to transliterate characters from one encoding to another. See locale(5) and other locales for inspiration how to implement these rules. See the Testing Locales section below for tips how to verify implementation.

6.2.4. LC_COLLATE

In many/most countries and languages there are official guidelines and standards on collation rules. Often these are based on the well-known ISO 14651 standard or on the Unicode collation algorithm. See other locales for inspiration how to implement the required collation rules. See the Testing Locales section below for tips how to verify that the implementation matches the relevant standards. Note also bug 14095.

6.2.5. LC_TIME

This is one of the most often used categories. The non-obvious members of this category are as follows:

Furthermore, there is the question of the abday and day keywords and which day of week should the lists start with. Specs say Sunday, but they do not mention any of the week start specifiers. Applications aware of these tend to interpret the abday and day lists in a more complicated way.

The tricky thing is how to reconcile information from WEEKSTARTDATE and first_weekday. PetrBaudis wrote some lenghty treatises about this on libc-locales; we present the outcome and thus our de facto current interpretation:

Thus, for example en_GB definition (English locale with week starting on Monday) is: 

  week          7;19971130;4
  first_weekday 2
  first_workday 2
  day           "Sunday;Monday;Tuesday;Wednesday;Thursday;Friday;Saturday"
  abday         "Sun;Mon;Tue;Wed;Thu;Fri;Sat"

When your locale is compiled, you can use a simple first_weekday test tool to check the day definitions are correct.

6.2.6. LC_NUMERIC

This category has only three members. See the manual page, POSIX, and ISO TR 14652 references and e.g. http://h71000.www7.hp.com/doc/73final/6494/6494pro_003.html for more information on grouping.

6.2.7. LC_MONETARY

This category is well described in the manual page, POSIX, and ISO TR 14652 references.

Note that if a locale uses a new, previously undefined currency, it should be added to locale/iso-4217.def.

6.2.8. LC_MESSAGES

This category defines regular expressions to be accepted as positive or negative response and equivalents of yes and no.

In yesstr and nostr the beginning should reflect whether the beginning of the word is capitalized or in lowercase. This is in line with other specifications in glibc locales, where e.g. day names and month names reflect the use of lower and upper case as prescribed in dictionaries, which records the canonical form of the word or phrase. This is also in accordance with use in POSIX and ISO TR 14652, and general guidelines for definitions in ISO/IEC.

6.2.9. LC_PAPER

Here A4 is 297x210 and US Letter is 279x216.

Rather than define height and width explicitly, locales should copy either the main language for that territory (e.g. most locales in India copy hi_IN), or they should copy i18n (for A4) or en_US (for US Letter).

6.2.10. LC_MEASUREMENT

Here 1 means metric, 2 means US.

Rather than define measurement explicitly, locales should copy either the main language for that territory (e.g. most locales in India copy hi_IN), or they should copy i18n (for metric) or en_US (for US).

Keep in mind that the measurement field reflects the "main" unit system. Some territories use metric for distances & weights, but Fahrenheit for temperature. In these cases, it should be configured for metric.

6.2.11. LC_NAME

This category is well explained in the manual page, POSIX, and ISO TR 14652.

name_fmt should be always defined, other members only if they are commonly used.

6.2.12. LC_ADDRESS

This category is explained in the manual page, POSIX, and ISO TR 14652. The following notes apply for glibc locales:

Note that if a locale uses a new, previously undefined country and/or language code, they should be added to locale/iso-3166.def and/or to locale/iso-639.def.

Applications should prefer lang_term over lang_lib. There are 20 specific ISO 639-2/B codes, both ISO 639-2/T and ISO 639-2/B are listed at http://www.loc.gov/standards/iso639-2/langhome.html.

6.2.13. LC_TELEPHONE

This category is well explained in the manual page, POSIX, and ISO TR 14652.

6.3. Testing Locales

After modifying a locale, make sure it compiles, and install it to a temporary directory for testing. The following example is ran on the glibc source tree root: 

  LOCALE=fi_FI
  export LOCPATH=$HOME/locale-test/
  mkdir -p $LOCPATH
  I18NPATH=./localedata/ localedef -f UTF-8 -i $LOCALE $LOCPATH/$LOCALE.UTF-8
  LC_ALL=$LOCALE.UTF-8 locale -ck LC_TIME
  LC_ALL=$LOCALE.UTF-8 locale -ck date_fmt
  LC_ALL=$LOCALE.UTF-8 date
  LC_ALL=$LOCALE.UTF-8 iconv -f UTF-8 -t ASCII//TRANSLIT < translit-test-input.txt
  LC_ALL=$LOCALE.UTF-8 sort < sorting-test-input.txt

These commands prepare a user directory for testing a locale, compile and install the locale, show contents of a category and a value of a certain keyword, and run commands which use different categories. Note that if the locale uses a new country, currency, or language code (see the LC_MONETARY and LC_ADDRESS sections above for details) then it needs to be compiled with localedef utility compiled in the glibc build directory.

If you have set up glibc compilation environment, you can mass-test compilation of all locales by installing to the locale archive and then installing the individual files into their respective directories (exercises both kinds of installs): 

  make localedata/install-locales DESTDIR=<PATH>
  make localedata/install-locale-files DESTDIR=<PATH>

The data in the locale-archive file will be used preferentially. If you want to test the installed files you need to remove the locale-archive or install the files in an alternate location and use LOCPATH to locate them. Lastly, note that invoking <prefix>/bin/localedef manually will of course install to the configured --prefix path (unless you use an absolute path as the output).

See localedata/README and localedef(1) for more information about localedef(1).

6.4. Contributing

See Contribution checklist for complete contributing instructions.

When contributing locale updates, always try to get in touch with the locale maintainer first; if this is unsuccessful, try to describe the changes you have made, and (this is important) provide some proofs that this reflects common usage - e.g. local government or big newspapers sites, references to language norms, etc.

Please test your changes before submitting them, see above for testing instructions.

6.5. Miscellaneous Information

6.5.1. Charsets

The iconv conversion internally always works by converting from source charset to UCS-4 and then from UCS-4 to the target charset. This implies that the charset modules need to implement only to/from Unicode mapping, and that characters not in Unicode are not convertable (luckily, this seems to be currently the case only for few obscure ancient kanji characters).

Most of the charsets are simple (single-byte with direct 1-1 Unicode mapping). .c files for these are trivial, depending on data provided by .h files, autogenerated by iconvdata/gen-8bit.sh from localedata/charmaps/ files at build time.

6.5.2. Transliteration

LC_CTYPE specifies the rules how to transliterate characters from one encoding to another, see the above LC_CTYPE section for more details.

7. References

None: Locales (last edited 2018-08-02 19:33:42 by CarlosODonell)