internationalized text processing
Mon May 23 13:44:00 GMT 2011
Never posted here, but somewhat off-topic for the general Cygwin list...
I've been frustrated at the lack of guidance I've found online regarding
how to build an internationalized text-based Unix application, in particular
with reference to handling different character sets. For example, "man"
recognizes the character set specified in locale settings and uses Unicode
punctuation appropriately. OTOH, the cygutils "ascii -e" utility does not
recognize that my LANG specifies UTF-8 and outputs garbage for the
extended half. Should this be considered a bug?
Closer to a real application, how should a program that reads, processes,
and outputs text detect the character set so that it can process the text
correctly? In the absence of a BOM in the input file, should the program
blindly presume the locale-specified character set or analyze the file to
guess it? And what advice is there for guessing? And, once detected, what
takes precedence for output: the input file's character set or the locale
I presume these issues are either settled or being debated somewhere.
More information about the Cygwin-talk