]>
Commit | Line | Data |
---|---|---|
390955cb UD |
1 | @node Character Set Handling, Locales, String and Array Utilities, Top |
2 | @c %MENU% Support for extended character sets | |
3 | @chapter Character Set Handling | |
4 | ||
5 | @ifnottex | |
6 | @macro cal{text} | |
7 | \text\ | |
8 | @end macro | |
9 | @end ifnottex | |
10 | ||
d731df03 UD |
11 | Character sets used in the early days of computing had only six, seven, |
12 | or eight bits for each character: there was never a case where more than | |
13 | eight bits (one byte) were used to represent a single character. The | |
14 | limitations of this approach became more apparent as more people | |
15 | grappled with non-Roman character sets, where not all the characters | |
16 | that make up a language's character set can be represented by @math{2^8} | |
17 | choices. This chapter shows the functionality which was added to the C | |
18 | library to correctly support multiple character sets. | |
390955cb UD |
19 | |
20 | @menu | |
21 | * Extended Char Intro:: Introduction to Extended Characters. | |
22 | * Charset Function Overview:: Overview about Character Handling | |
23 | Functions. | |
24 | * Restartable multibyte conversion:: Restartable multibyte conversion | |
25 | Functions. | |
26 | * Non-reentrant Conversion:: Non-reentrant Conversion Function. | |
27 | * Generic Charset Conversion:: Generic Charset Conversion. | |
28 | @end menu | |
29 | ||
30 | ||
31 | @node Extended Char Intro | |
32 | @section Introduction to Extended Characters | |
33 | ||
d731df03 UD |
34 | A variety of solutions to overcome the differences between |
35 | character sets with a 1:1 relation between bytes and characters and | |
36 | character sets with ratios of 2:1 or 4:1 exist. The remainder of this | |
37 | section gives a few examples to help understand the design decisions | |
38 | made while developing the functionality of the @w{C library}. | |
390955cb UD |
39 | |
40 | @cindex internal representation | |
41 | A distinction we have to make right away is between internal and | |
42 | external representation. @dfn{Internal representation} means the | |
43 | representation used by a program while keeping the text in memory. | |
44 | External representations are used when text is stored or transmitted | |
d731df03 UD |
45 | through whatever communication channel. Examples of external |
46 | representations include files lying in a directory that are going to be | |
47 | read and parsed. | |
390955cb UD |
48 | |
49 | Traditionally there was no difference between the two representations. | |
50 | It was equally comfortable and useful to use the same one-byte | |
51 | representation internally and externally. This changes with more and | |
52 | larger character sets. | |
53 | ||
54 | One of the problems to overcome with the internal representation is | |
d731df03 | 55 | handling text which is externally encoded using different character |
390955cb UD |
56 | sets. Assume a program which reads two texts and compares them using |
57 | some metric. The comparison can be usefully done only if the texts are | |
58 | internally kept in a common format. | |
59 | ||
60 | @cindex wide character | |
61 | For such a common format (@math{=} character set) eight bits are certainly | |
d731df03 UD |
62 | no longer enough. So the smallest entity will have to grow: @dfn{wide |
63 | characters} will now be used. Instead of one byte, two or four will | |
64 | be used instead. (Three are not good to address in memory and more | |
65 | than four bytes seem not to be necessary). | |
390955cb UD |
66 | |
67 | @cindex Unicode | |
68 | @cindex ISO 10646 | |
d731df03 | 69 | As shown in some other part of this manual, |
390955cb UD |
70 | @c !!! Ahem, wide char string functions are not yet covered -- drepper |
71 | there exists a completely new family of functions which can handle texts | |
d731df03 | 72 | of this kind in memory. The most commonly used character set for such |
390955cb | 73 | internal wide character representations are Unicode and @w{ISO 10646}. |
37a87f83 | 74 | The former is a subset of the latter and used when wide characters are |
390955cb UD |
75 | chosen to by 2 bytes (@math{= 16} bits) wide. The standard names of the |
76 | @cindex UCS2 | |
77 | @cindex UCS4 | |
78 | encodings used in these cases are UCS2 (@math{= 16} bits) and UCS4 | |
79 | (@math{= 32} bits). | |
80 | ||
d731df03 UD |
81 | To represent wide characters the @code{char} type is not suitable. For |
82 | this reason the @w{ISO C} standard introduces a new type which is | |
83 | designed to keep one character of a wide character string. To maintain | |
84 | the similarity there is also a type corresponding to @code{int} for | |
85 | those functions which take a single wide character. | |
390955cb UD |
86 | |
87 | @comment stddef.h | |
88 | @comment ISO | |
89 | @deftp {Data type} wchar_t | |
90 | This data type is used as the base type for wide character strings. | |
91 | I.e., arrays of objects of this type are the equivalent of @code{char[]} | |
92 | for multibyte character strings. The type is defined in @file{stddef.h}. | |
93 | ||
976780fd | 94 | The @w{ISO C90} standard, where this type was introduced, does not say |
390955cb UD |
95 | anything specific about the representation. It only requires that this |
96 | type is capable to store all elements of the basic character set. | |
97 | Therefore it would be legitimate to define @code{wchar_t} and | |
98 | @code{char}. This might make sense for embedded systems. | |
99 | ||
100 | But for GNU systems this type is always 32 bits wide. It is therefore | |
101 | capable to represent all UCS4 value therefore covering all of @w{ISO | |
102 | 10646}. Some Unix systems define @code{wchar_t} as a 16 bit type and | |
103 | thereby follow Unicode very strictly. This is perfectly fine with the | |
d731df03 | 104 | standard but it also means that to represent all characters from Unicode |
390955cb UD |
105 | and @w{ISO 10646} one has to use surrogate character which is in fact a |
106 | multi-wide-character encoding. But this contradicts the purpose of the | |
107 | @code{wchar_t} type. | |
108 | @end deftp | |
109 | ||
110 | @comment wchar.h | |
111 | @comment ISO | |
112 | @deftp {Data type} wint_t | |
113 | @code{wint_t} is a data type used for parameters and variables which | |
114 | contain a single wide character. As the name already suggests it is the | |
115 | equivalent to @code{int} when using the normal @code{char} strings. The | |
116 | types @code{wchar_t} and @code{wint_t} have often the same | |
117 | representation if their size if 32 bits wide but if @code{wchar_t} is | |
118 | defined as @code{char} the type @code{wint_t} must be defined as | |
119 | @code{int} due to the parameter promotion. | |
120 | ||
121 | @pindex wchar.h | |
122 | This type is defined in @file{wchar.h} and got introduced in the second | |
976780fd | 123 | amendment to @w{ISO C90}. |
390955cb UD |
124 | @end deftp |
125 | ||
126 | As there are for the @code{char} data type there also exist macros | |
127 | specifying the minimum and maximum value representable in an object of | |
128 | type @code{wchar_t}. | |
129 | ||
130 | @comment wchar.h | |
131 | @comment ISO | |
132 | @deftypevr Macro wint_t WCHAR_MIN | |
133 | The macro @code{WCHAR_MIN} evaluates to the minimum value representable | |
134 | by an object of type @code{wint_t}. | |
135 | ||
976780fd | 136 | This macro got introduced in the second amendment to @w{ISO C90}. |
390955cb UD |
137 | @end deftypevr |
138 | ||
139 | @comment wchar.h | |
140 | @comment ISO | |
141 | @deftypevr Macro wint_t WCHAR_MAX | |
142 | The macro @code{WCHAR_MIN} evaluates to the maximum value representable | |
143 | by an object of type @code{wint_t}. | |
144 | ||
976780fd | 145 | This macro got introduced in the second amendment to @w{ISO C90}. |
390955cb UD |
146 | @end deftypevr |
147 | ||
148 | Another special wide character value is the equivalent to @code{EOF}. | |
149 | ||
150 | @comment wchar.h | |
151 | @comment ISO | |
152 | @deftypevr Macro wint_t WEOF | |
153 | The macro @code{WEOF} evaluates to a constant expression of type | |
154 | @code{wint_t} whose value is different from any member of the extended | |
155 | character set. | |
156 | ||
157 | @code{WEOF} need not be the same value as @code{EOF} and unlike | |
158 | @code{EOF} it also need @emph{not} be negative. I.e., sloppy code like | |
159 | ||
160 | @smallexample | |
161 | @{ | |
162 | int c; | |
163 | ... | |
164 | while ((c = getc (fp)) < 0) | |
165 | ... | |
166 | @} | |
167 | @end smallexample | |
168 | ||
169 | @noindent | |
170 | has to be rewritten to explicitly use @code{WEOF} when wide characters | |
171 | are used. | |
172 | ||
173 | @smallexample | |
174 | @{ | |
175 | wint_t c; | |
176 | ... | |
177 | while ((c = wgetc (fp)) != WEOF) | |
178 | ... | |
179 | @} | |
180 | @end smallexample | |
181 | ||
182 | @pindex wchar.h | |
976780fd | 183 | This macro was introduced in the second amendment to @w{ISO C90} and is |
390955cb UD |
184 | defined in @file{wchar.h}. |
185 | @end deftypevr | |
186 | ||
187 | ||
188 | These internal representations present problems when it comes to storing | |
d731df03 | 189 | and transmittal, since a single wide character consists of more |
390955cb UD |
190 | than one byte they are effected by byte-ordering. I.e., machines with |
191 | different endianesses would see different value accessing the same data. | |
192 | This also applies for communication protocols which are all byte-based | |
193 | and therefore the sender has to decide about splitting the wide | |
d731df03 | 194 | character in bytes. A last (but not least important) point is that wide |
390955cb UD |
195 | characters often require more storage space than an customized byte |
196 | oriented character set. | |
197 | ||
198 | @cindex multibyte character | |
d731df03 UD |
199 | @cindex EBCDIC |
200 | For all the above reasons, an external encoding which is different | |
a58fe839 | 201 | from the internal encoding is often used if the latter is UCS2 or UCS4. |
d731df03 UD |
202 | The external encoding is byte-based and can be chosen appropriately for |
203 | the environment and for the texts to be handled. There exist a variety | |
204 | of different character sets which can be used for this external | |
205 | encoding. Information which will not be exhaustively presented | |
206 | here--instead, a description of the major groups will suffice. All of | |
207 | the ASCII-based character sets [_bkoz_: do you mean Roman character | |
208 | sets? If not, what do you mean here?] fulfill one requirement: they are | |
209 | "filesystem safe". This means that the character @code{'/'} is used in | |
210 | the encoding @emph{only} to represent itself. Things are a bit | |
211 | different for character sets like EBCDIC (Extended Binary Coded Decimal | |
212 | Interchange Code, a character set family used by IBM) but if the | |
390955cb UD |
213 | operation system does not understand EBCDIC directly the parameters to |
214 | system calls have to be converted first anyhow. | |
215 | ||
216 | @itemize @bullet | |
217 | @item | |
218 | The simplest character sets are one-byte character sets. There can be | |
219 | only up to 256 characters (for @w{8 bit} character sets) which is not | |
220 | sufficient to cover all languages but might be sufficient to handle a | |
221 | specific text. Another reason to choose this is because of constraints | |
d731df03 | 222 | from interaction with other programs (which might not be 8-bit clean). |
390955cb UD |
223 | |
224 | @cindex ISO 2022 | |
225 | @item | |
226 | The @w{ISO 2022} standard defines a mechanism for extended character | |
227 | sets where one character @emph{can} be represented by more than one | |
228 | byte. This is achieved by associating a state with the text. Embedded | |
229 | in the text can be characters which can be used to change the state. | |
230 | Each byte in the text might have a different interpretation in each | |
231 | state. The state might even influence whether a given byte stands for a | |
232 | character on its own or whether it has to be combined with some more | |
233 | bytes. | |
234 | ||
235 | @cindex EUC | |
236 | @cindex SJIS | |
237 | In most uses of @w{ISO 2022} the defined character sets do not allow | |
238 | state changes which cover more than the next character. This has the | |
239 | big advantage that whenever one can identify the beginning of the byte | |
240 | sequence of a character one can interpret a text correctly. Examples of | |
241 | character sets using this policy are the various EUC character sets | |
242 | (used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN) | |
243 | or SJIS (Shift JIS, a Japanese encoding). | |
244 | ||
245 | But there are also character sets using a state which is valid for more | |
246 | than one character and has to be changed by another byte sequence. | |
247 | Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN. | |
248 | ||
249 | @item | |
250 | @cindex ISO 6937 | |
251 | Early attempts to fix 8 bit character sets for other languages using the | |
252 | Roman alphabet lead to character sets like @w{ISO 6937}. Here bytes | |
d731df03 UD |
253 | representing characters like the acute accent do not produce output |
254 | themselves: one has to combine them with other characters to get the | |
255 | desired result. E.g., the byte sequence @code{0xc2 0x61} (non-spacing | |
256 | acute accent, following by lower-case `a') to get the ``small a with | |
257 | acute'' character. To get the acute accent character on its on one has | |
258 | to write @code{0xc2 0x20} (the non-spacing acute followed by a space). | |
390955cb UD |
259 | |
260 | This type of characters sets is quite frequently used in embedded | |
261 | systems such as video text. | |
262 | ||
263 | @item | |
264 | @cindex UTF-8 | |
265 | Instead of converting the Unicode or @w{ISO 10646} text used internally | |
a58fe839 | 266 | it is often also sufficient to simply use an encoding different than |
390955cb UD |
267 | UCS2/UCS4. The Unicode and @w{ISO 10646} standards even specify such an |
268 | encoding: UTF-8. This encoding is able to represent all of @w{ISO | |
269 | 10464} 31 bits in a byte string of length one to seven. | |
270 | ||
271 | @cindex UTF-7 | |
272 | There were a few other attempts to encode @w{ISO 10646} such as UTF-7 | |
273 | but UTF-8 is today the only encoding which should be used. In fact, | |
274 | UTF-8 will hopefully soon be the only external which has to be | |
d731df03 UD |
275 | supported. It proves to be universally usable and the only disadvantage |
276 | is that it favor Roman languages very much by making the byte string | |
390955cb | 277 | representation of other scripts (Cyrillic, Greek, Asian scripts) longer |
d731df03 UD |
278 | than necessary if using a specific character set for these scripts. |
279 | Methods like the Unicode compression scheme can alleviate these | |
280 | problems. | |
390955cb UD |
281 | @end itemize |
282 | ||
d731df03 UD |
283 | The question remaining is: how to select the character set or encoding |
284 | to use. The answer: you cannot decide about it yourself, it is decided | |
285 | by the developers of the system or the majority of the users. Since the | |
286 | goal is interoperability one has to use whatever the other people one | |
287 | works with use. If there are no constraints the selection is based on | |
288 | the requirements the expected circle of users will have. I.e., if a | |
289 | project is expected to only be used in, say, Russia it is fine to use | |
290 | KOI8-R or a similar character set. But if at the same time people from, | |
a58fe839 | 291 | say, Greece are participating one should use a character set which allows |
d731df03 UD |
292 | all people to collaborate. |
293 | ||
294 | The most widely useful solution seems to be: go with the most general | |
295 | character set, namely @w{ISO 10646}. Use UTF-8 as the external encoding | |
296 | and problems about users not being able to use their own language | |
297 | adequately are a thing of the past. | |
390955cb UD |
298 | |
299 | One final comment about the choice of the wide character representation | |
300 | is necessary at this point. We have said above that the natural choice | |
301 | is using Unicode or @w{ISO 10646}. This is not specified in any | |
302 | standard, though. The @w{ISO C} standard does not specify anything | |
303 | specific about the @code{wchar_t} type. There might be systems where | |
304 | the developers decided differently. Therefore one should as much as | |
305 | possible avoid making assumption about the wide character representation | |
306 | although GNU systems will always work as described above. If the | |
307 | programmer uses only the functions provided by the C library to handle | |
308 | wide character strings there should not be any compatibility problems | |
309 | with other systems. | |
310 | ||
311 | @node Charset Function Overview | |
312 | @section Overview about Character Handling Functions | |
313 | ||
314 | A Unix @w{C library} contains three different sets of functions in two | |
7be8096f | 315 | families to handle character set conversion. The one function family |
390955cb UD |
316 | is specified in the @w{ISO C} standard and therefore is portable even |
317 | beyond the Unix world. | |
318 | ||
976780fd | 319 | The most commonly known set of functions, coming from the @w{ISO C90} |
390955cb UD |
320 | standard, is unfortunately the least useful one. In fact, these |
321 | functions should be avoided whenever possible, especially when | |
322 | developing libraries (as opposed to applications). | |
323 | ||
d731df03 | 324 | The second family of functions got introduced in the early Unix standards |
390955cb UD |
325 | (XPG2) and is still part of the latest and greatest Unix standard: |
326 | @w{Unix 98}. It is also the most powerful and useful set of functions. | |
327 | But we will start with the functions defined in the second amendment to | |
976780fd | 328 | @w{ISO C90}. |
390955cb UD |
329 | |
330 | @node Restartable multibyte conversion | |
331 | @section Restartable Multibyte Conversion Functions | |
332 | ||
333 | The @w{ISO C} standard defines functions to convert strings from a | |
334 | multibyte representation to wide character strings. There are a number | |
335 | of peculiarities: | |
336 | ||
337 | @itemize @bullet | |
338 | @item | |
339 | The character set assumed for the multibyte encoding is not specified | |
340 | as an argument to the functions. Instead the character set specified by | |
341 | the @code{LC_CTYPE} category of the current locale is used; see | |
342 | @ref{Locale Categories}. | |
343 | ||
344 | @item | |
345 | The functions handling more than one character at a time require NUL | |
346 | terminated strings as the argument. I.e., converting blocks of text | |
347 | does not work unless one can add a NUL byte at an appropriate place. | |
348 | The GNU C library contains some extensions the standard which allow | |
349 | specifying a size but basically they also expect terminated strings. | |
350 | @end itemize | |
351 | ||
352 | Despite these limitations the @w{ISO C} functions can very well be used | |
353 | in many contexts. In graphical user interfaces, for instance, it is not | |
354 | uncommon to have functions which require text to be displayed in a wide | |
355 | character string if it is not simple ASCII. The text itself might come | |
7be8096f UD |
356 | from a file with translations and the user should decide about the |
357 | current locale which determines the translation and therefore also the | |
358 | external encoding used. In such a situation (and many others) the | |
390955cb UD |
359 | functions described here are perfect. If more freedom while performing |
360 | the conversion is necessary take a look at the @code{iconv} functions | |
464d646f | 361 | (@pxref{Generic Charset Conversion}). |
390955cb UD |
362 | |
363 | @menu | |
364 | * Selecting the Conversion:: Selecting the conversion and its properties. | |
365 | * Keeping the state:: Representing the state of the conversion. | |
366 | * Converting a Character:: Converting Single Characters. | |
367 | * Converting Strings:: Converting Multibyte and Wide Character | |
368 | Strings. | |
369 | * Multibyte Conversion Example:: A Complete Multibyte Conversion Example. | |
370 | @end menu | |
371 | ||
372 | @node Selecting the Conversion | |
373 | @subsection Selecting the conversion and its properties | |
374 | ||
375 | We already said above that the currently selected locale for the | |
376 | @code{LC_CTYPE} category decides about the conversion which is performed | |
377 | by the functions we are about to describe. Each locale uses its own | |
378 | character set (given as an argument to @code{localedef}) and this is the | |
379 | one assumed as the external multibyte encoding. The wide character | |
7be8096f | 380 | character set always is UCS4, at least on GNU systems. |
390955cb UD |
381 | |
382 | A characteristic of each multibyte character set is the maximum number | |
383 | of bytes which can be necessary to represent one character. This | |
384 | information is quite important when writing code which uses the | |
385 | conversion functions. In the examples below we will see some examples. | |
386 | The @w{ISO C} standard defines two macros which provide this information. | |
387 | ||
388 | ||
389 | @comment limits.h | |
390 | @comment ISO | |
391 | @deftypevr Macro int MB_LEN_MAX | |
392 | This macro specifies the maximum number of bytes in the multibyte | |
393 | sequence for a single character in any of the supported locales. It is | |
394 | a compile-time constant and it is defined in @file{limits.h}. | |
395 | @pindex limits.h | |
396 | @end deftypevr | |
397 | ||
398 | @comment stdlib.h | |
399 | @comment ISO | |
400 | @deftypevr Macro int MB_CUR_MAX | |
401 | @code{MB_CUR_MAX} expands into a positive integer expression that is the | |
402 | maximum number of bytes in a multibyte character in the current locale. | |
403 | The value is never greater than @code{MB_LEN_MAX}. Unlike | |
404 | @code{MB_LEN_MAX} this macro need not be a compile-time constant and in | |
405 | fact, in the GNU C library it is not. | |
406 | ||
407 | @pindex stdlib.h | |
408 | @code{MB_CUR_MAX} is defined in @file{stdlib.h}. | |
409 | @end deftypevr | |
410 | ||
976780fd | 411 | Two different macros are necessary since strictly @w{ISO C90} compilers |
390955cb UD |
412 | do not allow variable length array definitions but still it is desirable |
413 | to avoid dynamic allocation. This incomplete piece of code shows the | |
414 | problem: | |
415 | ||
416 | @smallexample | |
417 | @{ | |
418 | char buf[MB_LEN_MAX]; | |
419 | ssize_t len = 0; | |
420 | ||
421 | while (! feof (fp)) | |
422 | @{ | |
423 | fread (&buf[len], 1, MB_CUR_MAX - len, fp); | |
424 | /* @r{... process} buf */ | |
425 | len -= used; | |
426 | @} | |
427 | @} | |
428 | @end smallexample | |
429 | ||
430 | The code in the inner loop is expected to have always enough bytes in | |
431 | the array @var{buf} to convert one multibyte character. The array | |
432 | @var{buf} has to be sized statically since many compilers do not allow a | |
433 | variable size. The @code{fread} call makes sure that always | |
d731df03 UD |
434 | @code{MB_CUR_MAX} bytes are available in @var{buf}. Note that it isn't |
435 | a problem if @code{MB_CUR_MAX} is not a compile-time constant. | |
390955cb UD |
436 | |
437 | ||
438 | @node Keeping the state | |
439 | @subsection Representing the state of the conversion | |
440 | ||
441 | @cindex stateful | |
442 | In the introduction of this chapter it was said that certain character | |
443 | sets use a @dfn{stateful} encoding. I.e., the encoded values depend in | |
7be8096f | 444 | some way on the previous bytes in the text. |
390955cb UD |
445 | |
446 | Since the conversion functions allow converting a text in more than one | |
447 | step we must have a way to pass this information from one call of the | |
448 | functions to another. | |
449 | ||
450 | @comment wchar.h | |
451 | @comment ISO | |
452 | @deftp {Data type} mbstate_t | |
453 | @cindex shift state | |
454 | A variable of type @code{mbstate_t} can contain all the information | |
455 | about the @dfn{shift state} needed from one call to a conversion | |
456 | function to another. | |
457 | ||
458 | @pindex wchar.h | |
459 | This type is defined in @file{wchar.h}. It got introduced in the second | |
976780fd | 460 | amendment to @w{ISO C90}. |
390955cb UD |
461 | @end deftp |
462 | ||
463 | To use objects of this type the programmer has to define such objects | |
464 | (normally as local variables on the stack) and pass a pointer to the | |
465 | object to the conversion functions. This way the conversion function | |
466 | can update the object if the current multibyte character set is | |
467 | stateful. | |
468 | ||
469 | There is no specific function or initializer to put the state object in | |
470 | any specific state. The rules are that the object should always | |
471 | represent the initial state before the first use and this is achieved by | |
472 | clearing the whole variable with code such as follows: | |
473 | ||
474 | @smallexample | |
475 | @{ | |
476 | mbstate_t state; | |
477 | memset (&state, '\0', sizeof (state)); | |
478 | /* @r{from now on @var{state} can be used.} */ | |
479 | ... | |
480 | @} | |
481 | @end smallexample | |
482 | ||
483 | When using the conversion functions to generate output it is often | |
7be8096f | 484 | necessary to test whether the current state corresponds to the initial |
390955cb UD |
485 | state. This is necessary, for example, to decide whether or not to emit |
486 | escape sequences to set the state to the initial state at certain | |
487 | sequence points. Communication protocols often require this. | |
488 | ||
489 | @comment wchar.h | |
490 | @comment ISO | |
491 | @deftypefun int mbsinit (const mbstate_t *@var{ps}) | |
492 | This function determines whether the state object pointed to by @var{ps} | |
7be8096f | 493 | is in the initial state or not. If @var{ps} is a null pointer or the |
390955cb UD |
494 | object is in the initial state the return value is nonzero. Otherwise |
495 | it is zero. | |
496 | ||
497 | @pindex wchar.h | |
976780fd | 498 | This function was introduced in the second amendment to @w{ISO C90} and |
390955cb UD |
499 | is declared in @file{wchar.h}. |
500 | @end deftypefun | |
501 | ||
502 | Code using this function often looks similar to this: | |
503 | ||
37a87f83 UD |
504 | @c Fix the example to explicitly say how to generate the escape sequence |
505 | @c to restore the initial state. | |
390955cb UD |
506 | @smallexample |
507 | @{ | |
508 | mbstate_t state; | |
509 | memset (&state, '\0', sizeof (state)); | |
510 | /* @r{Use @var{state}.} */ | |
511 | ... | |
512 | if (! mbsinit (&state)) | |
513 | @{ | |
514 | /* @r{Emit code to return to initial state.} */ | |
976780fd UD |
515 | const wchar_t empty[] = L""; |
516 | const wchar_t *srcp = empty; | |
37a87f83 | 517 | wcsrtombs (outbuf, &srcp, outbuflen, &state); |
390955cb UD |
518 | @} |
519 | ... | |
520 | @} | |
521 | @end smallexample | |
522 | ||
37a87f83 UD |
523 | The code to emit the escape sequence to get back to the initial state is |
524 | interesting. The @code{wcsrtombs} function can be used to determine the | |
525 | necessary output code (@pxref{Converting Strings}). Please note that on | |
526 | GNU systems it is not necessary to perform this extra action for the | |
527 | conversion from multibyte text ot wide character text since the wide | |
528 | character encoding is not stateful. But there is nothing mentioned in | |
529 | any standard which prohibits making @code{wchar_t} using a stateful | |
530 | encoding. | |
531 | ||
390955cb UD |
532 | @node Converting a Character |
533 | @subsection Converting Single Characters | |
534 | ||
535 | The most fundamental of the conversion functions are those dealing with | |
536 | single characters. Please note that this does not always mean single | |
537 | bytes. But since there is very often a subset of the multibyte | |
538 | character set which consists of single byte sequences there are | |
539 | functions to help with converting bytes. One very important and often | |
540 | applicable scenario is where ASCII is a subpart of the multibyte | |
541 | character set. I.e., all ASCII characters stand for itself and all | |
542 | other characters have at least a first byte which is beyond the range | |
543 | @math{0} to @math{127}. | |
544 | ||
545 | @comment wchar.h | |
546 | @comment ISO | |
547 | @deftypefun wint_t btowc (int @var{c}) | |
548 | The @code{btowc} function (``byte to wide character'') converts a valid | |
7be8096f UD |
549 | single byte character @var{c} in the initial shift state into the wide |
550 | character equivalent using the conversion rules from the currently | |
551 | selected locale of the @code{LC_CTYPE} category. | |
390955cb UD |
552 | |
553 | If @code{(unsigned char) @var{c}} is no valid single byte multibyte | |
554 | character or if @var{c} is @code{EOF} the function returns @code{WEOF}. | |
555 | ||
556 | Please note the restriction of @var{c} being tested for validity only in | |
557 | the initial shift state. There is no @code{mbstate_t} object used from | |
558 | which the state information is taken and the function also does not use | |
559 | any static state. | |
560 | ||
561 | @pindex wchar.h | |
976780fd | 562 | This function was introduced in the second amendment of @w{ISO C90} and |
390955cb UD |
563 | is declared in @file{wchar.h}. |
564 | @end deftypefun | |
565 | ||
566 | Despite the limitation that the single byte value always is interpreted | |
567 | in the initial state this function is actually useful most of the time. | |
d731df03 | 568 | Most characters are either entirely single-byte character sets or they |
390955cb | 569 | are extension to ASCII. But then it is possible to write code like this |
7be8096f | 570 | (not that this specific example is very useful): |
390955cb UD |
571 | |
572 | @smallexample | |
573 | wchar_t * | |
574 | itow (unsigned long int val) | |
575 | @{ | |
576 | static wchar_t buf[30]; | |
577 | wchar_t *wcp = &buf[29]; | |
578 | *wcp = L'\0'; | |
579 | while (val != 0) | |
580 | @{ | |
581 | *--wcp = btowc ('0' + val % 10); | |
582 | val /= 10; | |
583 | @} | |
584 | if (wcp == &buf[29]) | |
d731df03 | 585 | *--wcp = L'0'; |
390955cb UD |
586 | return wcp; |
587 | @} | |
588 | @end smallexample | |
589 | ||
d731df03 | 590 | Why is it necessary to use such a complicated implementation and not |
7be8096f | 591 | simply cast @code{'0' + val % 10} to a wide character? The answer is |
d731df03 UD |
592 | that there is no guarantee that one can perform this kind of arithmetic |
593 | on the character of the character set used for @code{wchar_t} | |
7be8096f UD |
594 | representation. In other situations the bytes are not constant at |
595 | compile time and so the compiler cannot do the work. In situations like | |
596 | this it is necessary @code{btowc}. | |
390955cb | 597 | |
d731df03 | 598 | @noindent |
390955cb UD |
599 | There also is a function for the conversion in the other direction. |
600 | ||
601 | @comment wchar.h | |
602 | @comment ISO | |
603 | @deftypefun int wctob (wint_t @var{c}) | |
604 | The @code{wctob} function (``wide character to byte'') takes as the | |
c1b2d472 | 605 | parameter a valid wide character. If the multibyte representation for |
390955cb UD |
606 | this character in the initial state is exactly one byte long the return |
607 | value of this function is this character. Otherwise the return value is | |
608 | @code{EOF}. | |
609 | ||
610 | @pindex wchar.h | |
976780fd | 611 | This function was introduced in the second amendment of @w{ISO C90} and |
390955cb UD |
612 | is declared in @file{wchar.h}. |
613 | @end deftypefun | |
614 | ||
615 | There are more general functions to convert single character from | |
616 | multibyte representation to wide characters and vice versa. These | |
617 | functions pose no limit on the length of the multibyte representation | |
618 | and they also do not require it to be in the initial state. | |
619 | ||
620 | @comment wchar.h | |
621 | @comment ISO | |
622 | @deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps}) | |
623 | @cindex stateful | |
624 | The @code{mbrtowc} function (``multibyte restartable to wide | |
625 | character'') converts the next multibyte character in the string pointed | |
626 | to by @var{s} into a wide character and stores it in the wide character | |
627 | string pointed to by @var{pwc}. The conversion is performed according | |
628 | to the locale currently selected for the @code{LC_CTYPE} category. If | |
7be8096f UD |
629 | the conversion for the character set used in the locale requires a state |
630 | the multibyte string is interpreted in the state represented by the | |
631 | object pointed to by @var{ps}. If @var{ps} is a null pointer an static, | |
632 | internal state variable used only by the @code{mbrtowc} variable is | |
633 | used. | |
390955cb UD |
634 | |
635 | If the next multibyte character corresponds to the NUL wide character | |
636 | the return value of the function is @math{0} and the state object is | |
637 | afterwards in the initial state. If the next @var{n} or fewer bytes | |
638 | form a correct multibyte character the return value is the number of | |
639 | bytes starting from @var{s} which form the multibyte character. The | |
640 | conversion state is updated according to the bytes consumed in the | |
641 | conversion. In both cases the wide character (either the @code{L'\0'} | |
642 | or the one found in the conversion) is stored in the string pointer to | |
643 | by @var{pwc} iff @var{pwc} is not null. | |
644 | ||
645 | If the first @var{n} bytes of the multibyte string possibly form a valid | |
646 | multibyte character but there are more than @var{n} bytes needed to | |
647 | complete it the return value of the function is @code{(size_t) -2} and | |
648 | no value is stored. Please note that this can happen even if @var{n} | |
649 | has a value greater or equal to @code{MB_CUR_MAX} since the input might | |
650 | contain redundant shift sequences. | |
651 | ||
7be8096f UD |
652 | If the first @code{n} bytes of the multibyte string cannot possibly form |
653 | a valid multibyte character also no value is stored, the global variable | |
654 | @code{errno} is set to the value @code{EILSEQ} and the function returns | |
390955cb UD |
655 | @code{(size_t) -1}. The conversion state is afterwards undefined. |
656 | ||
657 | @pindex wchar.h | |
976780fd | 658 | This function was introduced in the second amendment to @w{ISO C90} and |
390955cb UD |
659 | is declared in @file{wchar.h}. |
660 | @end deftypefun | |
661 | ||
662 | Using this function is straight forward. A function which copies a | |
663 | multibyte string into a wide character string while at the same time | |
664 | converting all lowercase character into uppercase could look like this | |
665 | (this is not the final version, just an example; it has no error | |
7be8096f | 666 | checking, and leaks sometimes memory): |
390955cb UD |
667 | |
668 | @smallexample | |
669 | wchar_t * | |
670 | mbstouwcs (const char *s) | |
671 | @{ | |
672 | size_t len = strlen (s); | |
673 | wchar_t *result = malloc ((len + 1) * sizeof (wchar_t)); | |
674 | wchar_t *wcp = result; | |
675 | wchar_t tmp[1]; | |
676 | mbstate_t state; | |
677 | memset (&state, '\0', sizeof (state)); | |
678 | size_t nbytes; | |
679 | while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0) | |
680 | @{ | |
681 | if (nbytes >= (size_t) -2) | |
682 | /* Invalid input string. */ | |
683 | return NULL; | |
684 | *result++ = towupper (tmp[0]); | |
685 | len -= nbytes; | |
686 | s += nbytes; | |
687 | @} | |
688 | return result; | |
689 | @} | |
690 | @end smallexample | |
691 | ||
692 | The use of @code{mbrtowc} should be clear. A single wide character is | |
693 | stored in @code{@var{tmp}[0]} and the number of consumed bytes is stored | |
694 | in the variable @var{nbytes}. In case the the conversion was successful | |
695 | the uppercase variant of the wide character is stored in the | |
696 | @var{result} array and the pointer to the input string and the number of | |
697 | available bytes is adjusted. | |
698 | ||
699 | The only non-obvious thing about the function might be the way memory is | |
700 | allocated for the result. The above code uses the fact that there can | |
701 | never be more wide characters in the converted results than there are | |
702 | bytes in the multibyte input string. This method yields to a | |
703 | pessimistic guess about the size of the result and if many wide | |
704 | character strings have to be constructed this way or the strings are | |
7be8096f UD |
705 | long, the extra memory required allocated because the input string |
706 | contains multibzte characters might be significant. It would be | |
707 | possible to resize the allocated memory block to the correct size before | |
708 | returning it. A better solution might be to allocate just the right | |
709 | amount of space for the result right away. Unfortunately there is no | |
710 | function to compute the length of the wide character string directly | |
711 | from the multibyte string. But there is a function which does part of | |
712 | the work. | |
390955cb UD |
713 | |
714 | @comment wchar.h | |
715 | @comment ISO | |
716 | @deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps}) | |
717 | The @code{mbrlen} function (``multibyte restartable length'') computes | |
718 | the number of at most @var{n} bytes starting at @var{s} which form the | |
719 | next valid and complete multibyte character. | |
720 | ||
721 | If the next multibyte character corresponds to the NUL wide character | |
722 | the return value is @math{0}. If the next @var{n} bytes form a valid | |
723 | multibyte character the number of bytes belonging to this multibyte | |
724 | character byte sequence is returned. | |
725 | ||
726 | If the the first @var{n} bytes possibly form a valid multibyte | |
727 | character but it is incomplete the return value is @code{(size_t) -2}. | |
728 | Otherwise the multibyte character sequence is invalid and the return | |
729 | value is @code{(size_t) -1}. | |
730 | ||
731 | The multibyte sequence is interpreted in the state represented by the | |
732 | object pointer to by @var{ps}. If @var{ps} is a null pointer an state | |
733 | object local to @code{mbrlen} is used. | |
734 | ||
735 | @pindex wchar.h | |
976780fd | 736 | This function was introduced in the second amendment to @w{ISO C90} and |
390955cb UD |
737 | is declared in @file{wchar.h}. |
738 | @end deftypefun | |
739 | ||
740 | The tentative reader now will of course note that @code{mbrlen} can be | |
741 | implemented as | |
742 | ||
743 | @smallexample | |
744 | mbrtowc (NULL, s, n, ps != NULL ? ps : &internal) | |
745 | @end smallexample | |
746 | ||
747 | This is true and in fact is mentioned in the official specification. | |
748 | Now, how can this function be used to determine the length of the wide | |
749 | character string created from a multibyte character string? It is not | |
750 | directly usable but we can define a function @code{mbslen} using it: | |
751 | ||
752 | @smallexample | |
753 | size_t | |
754 | mbslen (const char *s) | |
755 | @{ | |
756 | mbstate_t state; | |
757 | size_t result = 0; | |
758 | size_t nbytes; | |
759 | memset (&state, '\0', sizeof (state)); | |
760 | while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0) | |
761 | @{ | |
762 | if (nbytes >= (size_t) -2) | |
763 | /* @r{Something is wrong.} */ | |
764 | return (size_t) -1; | |
765 | s += nbytes; | |
766 | ++result; | |
767 | @} | |
768 | return result; | |
769 | @} | |
770 | @end smallexample | |
771 | ||
772 | This function simply calls @code{mbrlen} for each multibyte character | |
773 | in the string and counts the number of function calls. Please note that | |
774 | we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen} | |
775 | call. This is OK since a) this value is larger then the length of the | |
776 | longest multibyte character sequence and b) because we know that the | |
7be8096f UD |
777 | string @var{s} ends with a NUL byte which cannot be part of any other |
778 | multibyte character sequence but the one representing the NUL wide | |
390955cb UD |
779 | character. Therefore the @code{mbrlen} function will never read invalid |
780 | memory. | |
781 | ||
782 | Now that this function is available (just to make this clear, this | |
783 | function is @emph{not} part of the GNU C library) we can compute the | |
784 | number of wide character required to store the converted multibyte | |
785 | character string @var{s} using | |
786 | ||
787 | @smallexample | |
788 | wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t); | |
789 | @end smallexample | |
790 | ||
791 | Please note that the @code{mbslen} function is quite inefficient. The | |
792 | implementation of @code{mbstouwcs} implemented using @code{mbslen} would | |
793 | have to perform the conversion of the multibyte character input string | |
794 | twice and this conversion might be quite expensive. So it is necessary | |
c1b2d472 | 795 | to think about the consequences of using the easier but imprecise method |
390955cb UD |
796 | before doing the work twice. |
797 | ||
798 | @comment wchar.h | |
799 | @comment ISO | |
800 | @deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps}) | |
801 | The @code{wcrtomb} function (``wide character restartable to | |
802 | multibyte'') converts a single wide character into a multibyte string | |
803 | corresponding to that wide character. | |
804 | ||
7be8096f UD |
805 | If @var{s} is a null pointer the function resets the the state stored in |
806 | the objects pointer to by @var{ps} (or the internal @code{mbstate_t} | |
807 | object) to the initial state. This can also be achieved by a call like | |
808 | this: | |
390955cb UD |
809 | |
810 | @smallexample | |
811 | wcrtombs (temp_buf, L'\0', ps) | |
812 | @end smallexample | |
813 | ||
814 | @noindent | |
7be8096f | 815 | since if @var{s} is a null pointer @code{wcrtomb} performs as if it |
390955cb UD |
816 | writes into an internal buffer which is guaranteed to be large enough. |
817 | ||
818 | If @var{wc} is the NUL wide character @code{wcrtomb} emits, if | |
819 | necessary, a shift sequence to get the state @var{ps} into the initial | |
820 | state followed by a single NUL byte is stored in the string @var{s}. | |
821 | ||
822 | Otherwise a byte sequence (possibly including shift sequences) is | |
7be8096f UD |
823 | written into the string @var{s}. This of only happens if @var{wc} is a |
824 | valid wide character, i.e., it has a multibyte representation in the | |
825 | character set selected by locale of the @code{LC_CTYPE} category. If | |
826 | @var{wc} is no valid wide character nothing is stored in the strings | |
827 | @var{s}, @code{errno} is set to @code{EILSEQ}, the conversion state in | |
828 | @var{ps} is undefined and the return value is @code{(size_t) -1}. | |
390955cb UD |
829 | |
830 | If no error occurred the function returns the number of bytes stored in | |
831 | the string @var{s}. This includes all byte representing shift | |
832 | sequences. | |
833 | ||
834 | One word about the interface of the function: there is no parameter | |
835 | specifying the length of the array @var{s}. Instead the function | |
836 | assumes that there are at least @code{MB_CUR_MAX} bytes available since | |
837 | this is the maximum length of any byte sequence representing a single | |
838 | character. So the caller has to make sure that there is enough space | |
839 | available, otherwise buffer overruns can occur. | |
840 | ||
841 | @pindex wchar.h | |
842 | This function was introduced in the second amendment to @w{ISO C} and is | |
843 | declared in @file{wchar.h}. | |
844 | @end deftypefun | |
845 | ||
846 | Using this function is as easy as using @code{mbrtowc}. The following | |
847 | example appends a wide character string to a multibyte character string. | |
7be8096f UD |
848 | Again, the code is not really useful (and correct), it is simply here to |
849 | demonstrate the use and some problems. | |
390955cb UD |
850 | |
851 | @smallexample | |
852 | char * | |
976780fd | 853 | mbscatwcs (char *s, size_t len, const wchar_t *ws) |
390955cb UD |
854 | @{ |
855 | mbstate_t state; | |
7be8096f | 856 | /* @r{Find the end of the existing string.} */ |
390955cb UD |
857 | char *wp = strchr (s, '\0'); |
858 | len -= wp - s; | |
859 | memset (&state, '\0', sizeof (state)); | |
860 | do | |
861 | @{ | |
862 | size_t nbytes; | |
863 | if (len < MB_CUR_LEN) | |
864 | @{ | |
865 | /* @r{We cannot guarantee that the next} | |
866 | @r{character fits into the buffer, so} | |
867 | @r{return an error.} */ | |
868 | errno = E2BIG; | |
869 | return NULL; | |
870 | @} | |
871 | nbytes = wcrtomb (wp, *ws, &state); | |
872 | if (nbytes == (size_t) -1) | |
873 | /* @r{Error in the conversion.} */ | |
874 | return NULL; | |
875 | len -= nbytes; | |
876 | wp += nbytes; | |
877 | @} | |
878 | while (*ws++ != L'\0'); | |
879 | return s; | |
880 | @} | |
881 | @end smallexample | |
882 | ||
883 | First the function has to find the end of the string currently in the | |
884 | array @var{s}. The @code{strchr} call does this very efficiently since a | |
885 | requirement for multibyte character representations is that the NUL byte | |
886 | never is used except to represent itself (and in this context, the end | |
887 | of the string). | |
888 | ||
889 | After initializing the state object the loop is entered where the first | |
890 | task is to make sure there is enough room in the array @var{s}. We | |
891 | abort if there are not at least @code{MB_CUR_LEN} bytes available. This | |
892 | is not always optimal but we have no other choice. We might have less | |
893 | than @code{MB_CUR_LEN} bytes available but the next multibyte character | |
894 | might also be only one byte long. At the time the @code{wcrtomb} call | |
895 | returns it is too late to decide whether the buffer was large enough or | |
896 | not. If this solution is really unsuitable there is a very slow but | |
897 | more accurate solution. | |
898 | ||
899 | @smallexample | |
900 | ... | |
901 | if (len < MB_CUR_LEN) | |
902 | @{ | |
903 | mbstate_t temp_state; | |
904 | memcpy (&temp_state, &state, sizeof (state)); | |
905 | if (wcrtomb (NULL, *ws, &temp_state) > len) | |
906 | @{ | |
907 | /* @r{We cannot guarantee that the next} | |
908 | @r{character fits into the buffer, so} | |
909 | @r{return an error.} */ | |
910 | errno = E2BIG; | |
911 | return NULL; | |
912 | @} | |
913 | @} | |
914 | ... | |
915 | @end smallexample | |
916 | ||
917 | Here we do perform the conversion which might overflow the buffer so | |
918 | that we are afterwards in the position to make an exact decision about | |
919 | the buffer size. Please note the @code{NULL} argument for the | |
920 | destination buffer in the new @code{wcrtomb} call; since we are not | |
7be8096f UD |
921 | interested in the converted text at this point this is a nice way to |
922 | express this. The most unusual thing about this piece of code certainly | |
923 | is the duplication of the conversion state object. But think about | |
924 | this: if a change of the state is necessary to emit the next multibyte | |
925 | character we want to have the same shift state change performed in the | |
926 | real conversion. Therefore we have to preserve the initial shift state | |
390955cb UD |
927 | information. |
928 | ||
929 | There are certainly many more and even better solutions to this problem. | |
930 | This example is only meant for educational purposes. | |
931 | ||
932 | @node Converting Strings | |
933 | @subsection Converting Multibyte and Wide Character Strings | |
934 | ||
935 | The functions described in the previous section only convert a single | |
936 | character at a time. Most operations to be performed in real-world | |
937 | programs include strings and therefore the @w{ISO C} standard also | |
d731df03 UD |
938 | defines conversions on entire strings. However, the defined set of |
939 | functions is quite limited, thus the GNU C library contains a few | |
7be8096f | 940 | extensions which can help in some important situations. |
390955cb UD |
941 | |
942 | @comment wchar.h | |
943 | @comment ISO | |
944 | @deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
945 | The @code{mbsrtowcs} function (``multibyte string restartable to wide | |
946 | character string'') converts an NUL terminated multibyte character | |
947 | string at @code{*@var{src}} into an equivalent wide character string, | |
948 | including the NUL wide character at the end. The conversion is started | |
949 | using the state information from the object pointed to by @var{ps} or | |
950 | from an internal object of @code{mbsrtowcs} if @var{ps} is a null | |
951 | pointer. Before returning the state object to match the state after the | |
952 | last converted character. The state is the initial state if the | |
953 | terminating NUL byte is reached and converted. | |
954 | ||
955 | If @var{dst} is not a null pointer the result is stored in the array | |
956 | pointed to by @var{dst}, otherwise the conversion result is not | |
957 | available since it is stored in an internal buffer. | |
958 | ||
959 | If @var{len} wide characters are stored in the array @var{dst} before | |
960 | reaching the end of the input string the conversion stops and @var{len} | |
961 | is returned. If @var{dst} is a null pointer @var{len} is never checked. | |
962 | ||
963 | Another reason for a premature return from the function call is if the | |
964 | input string contains an invalid multibyte sequence. In this case the | |
965 | global variable @code{errno} is set to @code{EILSEQ} and the function | |
966 | returns @code{(size_t) -1}. | |
967 | ||
968 | @c XXX The ISO C9x draft seems to have a problem here. It says that PS | |
969 | @c is not updated if DST is NULL. This is not said straight forward and | |
970 | @c none of the other functions is described like this. It would make sense | |
971 | @c to define the function this way but I don't think it is meant like this. | |
972 | ||
973 | In all other cases the function returns the number of wide characters | |
974 | converted during this call. If @var{dst} is not null @code{mbsrtowcs} | |
975 | stores in the pointer pointed to by @var{src} a null pointer (if the NUL | |
976 | byte in the input string was reached) or the address of the byte | |
977 | following the last converted multibyte character. | |
978 | ||
979 | @pindex wchar.h | |
980 | This function was introduced in the second amendment to @w{ISO C} and is | |
981 | declared in @file{wchar.h}. | |
982 | @end deftypefun | |
983 | ||
984 | The definition of this function has one limitation which has to be | |
985 | understood. The requirement that @var{dst} has to be a NUL terminated | |
986 | string provides problems if one wants to convert buffers with text. A | |
987 | buffer is normally no collection of NUL terminated strings but instead a | |
988 | continuous collection of lines, separated by newline characters. Now | |
989 | assume a function to convert one line from a buffer is needed. Since | |
990 | the line is not NUL terminated the source pointer cannot directly point | |
991 | into the unmodified text buffer. This means, either one inserts the NUL | |
992 | byte at the appropriate place for the time of the @code{mbsrtowcs} | |
993 | function call (which is not doable for a read-only buffer or in a | |
994 | multi-threaded application) or one copies the line in an extra buffer | |
995 | where it can be terminated by a NUL byte. Note that it is not in | |
996 | general possible to limit the number of characters to convert by setting | |
997 | the parameter @var{len} to any specific value. Since it is not known | |
998 | how many bytes each multibyte character sequence is in length one always | |
999 | could do only a guess. | |
1000 | ||
1001 | @cindex stateful | |
1002 | There is still a problem with the method of NUL-terminating a line right | |
1003 | after the newline character which could lead to very strange results. | |
1004 | As said in the description of the @var{mbsrtowcs} function above the | |
1005 | conversion state is guaranteed to be in the initial shift state after | |
1006 | processing the NUL byte at the end of the input string. But this NUL | |
1007 | byte is not really part of the text. I.e., the conversion state after | |
1008 | the newline in the original text could be something different than the | |
1009 | initial shift state and therefore the first character of the next line | |
1010 | is encoded using this state. But the state in question is never | |
7be8096f UD |
1011 | accessible to the user since the conversion stops after the NUL byte |
1012 | (which resets the state). Most stateful character sets in use today | |
1013 | require that the shift state after a newline is the initial state--but | |
1014 | this is not a strict guarantee. Therefore simply NUL terminating a | |
1015 | piece of a running text is not always an adequate solution and therefore | |
1016 | never should be used in generally used code. | |
390955cb | 1017 | |
61d97e92 | 1018 | The generic conversion interface (@pxref{Generic Charset Conversion}) |
d731df03 | 1019 | does not have this limitation (it simply works on buffers, not |
7be8096f | 1020 | strings), and the GNU C library contains a set of functions which take |
d731df03 UD |
1021 | additional parameters specifying the maximal number of bytes which are |
1022 | consumed from the input string. This way the problem of | |
1023 | @code{mbsrtowcs}'s example above could be solved by determining the line | |
1024 | length and passing this length to the function. | |
390955cb UD |
1025 | |
1026 | @comment wchar.h | |
1027 | @comment ISO | |
1028 | @deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
1029 | The @code{wcsrtombs} function (``wide character string restartable to | |
1030 | multibyte string'') converts the NUL terminated wide character string at | |
1031 | @code{*@var{src}} into an equivalent multibyte character string and | |
1032 | stores the result in the array pointed to by @var{dst}. The NUL wide | |
1033 | character is also converted. The conversion starts in the state | |
1034 | described in the object pointed to by @var{ps} or by a state object | |
1035 | locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If | |
1036 | @var{dst} is a null pointer the conversion is performed as usual but the | |
1037 | result is not available. If all characters of the input string were | |
1038 | successfully converted and if @var{dst} is not a null pointer the | |
1039 | pointer pointed to by @var{src} gets assigned a null pointer. | |
1040 | ||
1041 | If one of the wide characters in the input string has no valid multibyte | |
1042 | character equivalent the conversion stops early, sets the global | |
1043 | variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}. | |
1044 | ||
1045 | Another reason for a premature stop is if @var{dst} is not a null | |
1046 | pointer and the next converted character would require more than | |
1047 | @var{len} bytes in total to the array @var{dst}. In this case (and if | |
1048 | @var{dest} is not a null pointer) the pointer pointed to by @var{src} is | |
1049 | assigned a value pointing to the wide character right after the last one | |
1050 | successfully converted. | |
1051 | ||
1052 | Except in the case of an encoding error the return value of the function | |
1053 | is the number of bytes in all the multibyte character sequences stored | |
1054 | in @var{dst}. Before returning the state in the object pointed to by | |
1055 | @var{ps} (or the internal object in case @var{ps} is a null pointer) is | |
1056 | updated to reflect the state after the last conversion. The state is | |
1057 | the initial shift state in case the terminating NUL wide character was | |
1058 | converted. | |
1059 | ||
1060 | @pindex wchar.h | |
1061 | This function was introduced in the second amendment to @w{ISO C} and is | |
1062 | declared in @file{wchar.h}. | |
1063 | @end deftypefun | |
1064 | ||
1065 | The restriction mentions above for the @code{mbsrtowcs} function applies | |
1066 | also here. There is no possibility to directly control the number of | |
1067 | input characters. One has to place the NUL wide character at the | |
1068 | correct place or control the consumed input indirectly via the available | |
1069 | output array size (the @var{len} parameter). | |
1070 | ||
1071 | @comment wchar.h | |
1072 | @comment GNU | |
1073 | @deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
1074 | The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs} | |
1075 | function. All the parameters are the same except for @var{nmc} which is | |
1076 | new. The return value is the same as for @code{mbsrtowcs}. | |
1077 | ||
1078 | This new parameter specifies how many bytes at most can be used from the | |
1079 | multibyte character string. I.e., the multibyte character string | |
1080 | @code{*@var{src}} need not be NUL terminated. But if a NUL byte is | |
1081 | found within the @var{nmc} first bytes of the string the conversion | |
1082 | stops here. | |
1083 | ||
1084 | This function is a GNU extensions. It is meant to work around the | |
1085 | problems mentioned above. Now it is possible to convert buffer with | |
1086 | multibyte character text piece for piece without having to care about | |
1087 | inserting NUL bytes and the effect of NUL bytes on the conversion state. | |
1088 | @end deftypefun | |
1089 | ||
1090 | A function to convert a multibyte string into a wide character string | |
d731df03 | 1091 | and display it could be written like this (this is not a really useful |
390955cb UD |
1092 | example): |
1093 | ||
1094 | @smallexample | |
1095 | void | |
1096 | showmbs (const char *src, FILE *fp) | |
1097 | @{ | |
1098 | mbstate_t state; | |
1099 | int cnt = 0; | |
1100 | memset (&state, '\0', sizeof (state)); | |
1101 | while (1) | |
1102 | @{ | |
1103 | wchar_t linebuf[100]; | |
1104 | const char *endp = strchr (src, '\n'); | |
1105 | size_t n; | |
1106 | ||
1107 | /* @r{Exit if there is no more line.} */ | |
1108 | if (endp == NULL) | |
1109 | break; | |
1110 | ||
1111 | n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state); | |
1112 | linebuf[n] = L'\0'; | |
1113 | fprintf (fp, "line %d: \"%S\"\n", linebuf); | |
1114 | @} | |
1115 | @} | |
1116 | @end smallexample | |
1117 | ||
d731df03 UD |
1118 | There is no problem with the state after a call to @code{mbsnrtowcs}. |
1119 | Since we don't insert characters in the strings which were not in there | |
1120 | right from the beginning and we use @var{state} only for the conversion | |
1121 | of the given buffer there is no problem with altering the state. | |
390955cb UD |
1122 | |
1123 | @comment wchar.h | |
1124 | @comment GNU | |
1125 | @deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps}) | |
1126 | The @code{wcsnrtombs} function implements the conversion from wide | |
1127 | character strings to multibyte character strings. It is similar to | |
1128 | @code{wcsrtombs} but it takes, just like @code{mbsnrtowcs}, an extra | |
1129 | parameter which specifies the length of the input string. | |
1130 | ||
1131 | No more than @var{nwc} wide characters from the input string | |
1132 | @code{*@var{src}} are converted. If the input string contains a NUL | |
1133 | wide character in the first @var{nwc} character to conversion stops at | |
1134 | this place. | |
1135 | ||
1136 | This function is a GNU extension and just like @code{mbsnrtowcs} is | |
1137 | helps in situations where no NUL terminated input strings are available. | |
1138 | @end deftypefun | |
1139 | ||
1140 | ||
1141 | @node Multibyte Conversion Example | |
1142 | @subsection A Complete Multibyte Conversion Example | |
1143 | ||
1144 | The example programs given in the last sections are only brief and do | |
d731df03 | 1145 | not contain all the error checking etc. Presented here is a complete |
390955cb UD |
1146 | and documented example. It features the @code{mbrtowc} function but it |
1147 | should be easy to derive versions using the other functions. | |
1148 | ||
1149 | @smallexample | |
1150 | int | |
1151 | file_mbsrtowcs (int input, int output) | |
1152 | @{ | |
1153 | /* @r{Note the use of @code{MB_LEN_MAX}.} | |
1154 | @r{@code{MB_CUR_MAX} cannot portably be used here.} */ | |
1155 | char buffer[BUFSIZ + MB_LEN_MAX]; | |
1156 | mbstate_t state; | |
1157 | int filled = 0; | |
1158 | int eof = 0; | |
1159 | ||
1160 | /* @r{Initialize the state.} */ | |
1161 | memset (&state, '\0', sizeof (state)); | |
1162 | ||
1163 | while (!eof) | |
1164 | @{ | |
1165 | ssize_t nread; | |
1166 | ssize_t nwrite; | |
1167 | char *inp = buffer; | |
1168 | wchar_t outbuf[BUFSIZ]; | |
1169 | wchar_t *outp = outbuf; | |
1170 | ||
1171 | /* @r{Fill up the buffer from the input file.} */ | |
1172 | nread = read (input, buffer + filled, BUFSIZ); | |
1173 | if (nread < 0) | |
1174 | @{ | |
1175 | perror ("read"); | |
1176 | return 0; | |
1177 | @} | |
1178 | /* @r{If we reach end of file, make a note to read no more.} */ | |
1179 | if (nread == 0) | |
1180 | eof = 1; | |
1181 | ||
1182 | /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */ | |
1183 | filled += nread; | |
1184 | ||
1185 | /* @r{Convert those bytes to wide characters--as many as we can.} */ | |
1186 | while (1) | |
1187 | @{ | |
1188 | size_t thislen = mbrtowc (outp, inp, filled, &state); | |
1189 | /* @r{Stop converting at invalid character;} | |
1190 | @r{this can mean we have read just the first part} | |
1191 | @r{of a valid character.} */ | |
1192 | if (thislen == (size_t) -1) | |
1193 | break; | |
1194 | /* @r{We want to handle embedded NUL bytes} | |
1195 | @r{but the return value is 0. Correct this.} */ | |
1196 | if (thislen == 0) | |
1197 | thislen = 1; | |
1198 | /* @r{Advance past this character.} */ | |
1199 | inp += thislen; | |
1200 | filled -= thislen; | |
1201 | ++outp; | |
1202 | @} | |
1203 | ||
1204 | /* @r{Write the wide characters we just made.} */ | |
1205 | nwrite = write (output, outbuf, | |
1206 | (outp - outbuf) * sizeof (wchar_t)); | |
1207 | if (nwrite < 0) | |
1208 | @{ | |
1209 | perror ("write"); | |
1210 | return 0; | |
1211 | @} | |
1212 | ||
1213 | /* @r{See if we have a @emph{real} invalid character.} */ | |
1214 | if ((eof && filled > 0) || filled >= MB_CUR_MAX) | |
1215 | @{ | |
1216 | error (0, 0, "invalid multibyte character"); | |
1217 | return 0; | |
1218 | @} | |
1219 | ||
1220 | /* @r{If any characters must be carried forward,} | |
1221 | @r{put them at the beginning of @code{buffer}.} */ | |
1222 | if (filled > 0) | |
1223 | memmove (inp, buffer, filled); | |
1224 | @} | |
1225 | ||
1226 | return 1; | |
1227 | @} | |
1228 | @end smallexample | |
1229 | ||
1230 | ||
1231 | @node Non-reentrant Conversion | |
1232 | @section Non-reentrant Conversion Function | |
1233 | ||
1234 | The functions described in the last chapter are defined in the second | |
976780fd | 1235 | amendment to @w{ISO C90}. But the original @w{ISO C90} standard also |
390955cb UD |
1236 | contained functions for character set conversion. The reason that they |
1237 | are not described in the first place is that they are almost entirely | |
1238 | useless. | |
1239 | ||
1240 | The problem is that all the functions for conversion defined in @w{ISO | |
976780fd | 1241 | C90} use a local state. This implies that multiple conversions at the |
d731df03 UD |
1242 | same time (not only when using threads) cannot be done, and that you |
1243 | cannot first convert single characters and then strings since you cannot | |
1244 | tell the conversion functions which state to use. | |
390955cb UD |
1245 | |
1246 | These functions are therefore usable only in a very limited set of | |
7be8096f | 1247 | situations. One must complete converting the entire string before |
390955cb UD |
1248 | starting a new one and each string/text must be converted with the same |
1249 | function (there is no problem with the library itself; it is guaranteed | |
1250 | that no library function changes the state of any of these functions). | |
d731df03 UD |
1251 | @strong{For the above reasons it is highly requested that the functions |
1252 | from the last section are used in place of non-reentrant conversion | |
1253 | functions.} | |
390955cb UD |
1254 | |
1255 | @menu | |
1256 | * Non-reentrant Character Conversion:: Non-reentrant Conversion of Single | |
1257 | Characters. | |
1258 | * Non-reentrant String Conversion:: Non-reentrant Conversion of Strings. | |
1259 | * Shift State:: States in Non-reentrant Functions. | |
1260 | @end menu | |
1261 | ||
1262 | @node Non-reentrant Character Conversion | |
1263 | @subsection Non-reentrant Conversion of Single Characters | |
1264 | ||
1265 | @comment stdlib.h | |
1266 | @comment ISO | |
7be8096f | 1267 | @deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size}) |
390955cb UD |
1268 | The @code{mbtowc} (``multibyte to wide character'') function when called |
1269 | with non-null @var{string} converts the first multibyte character | |
1270 | beginning at @var{string} to its corresponding wide character code. It | |
1271 | stores the result in @code{*@var{result}}. | |
1272 | ||
1273 | @code{mbtowc} never examines more than @var{size} bytes. (The idea is | |
1274 | to supply for @var{size} the number of bytes of data you have in hand.) | |
1275 | ||
1276 | @code{mbtowc} with non-null @var{string} distinguishes three | |
1277 | possibilities: the first @var{size} bytes at @var{string} start with | |
1278 | valid multibyte character, they start with an invalid byte sequence or | |
1279 | just part of a character, or @var{string} points to an empty string (a | |
1280 | null character). | |
1281 | ||
1282 | For a valid multibyte character, @code{mbtowc} converts it to a wide | |
1283 | character and stores that in @code{*@var{result}}, and returns the | |
7be8096f | 1284 | number of bytes in that character (always at least @math{1}, and never |
390955cb UD |
1285 | more than @var{size}). |
1286 | ||
7be8096f UD |
1287 | For an invalid byte sequence, @code{mbtowc} returns @math{-1}. For an |
1288 | empty string, it returns @math{0}, also storing @code{'\0'} in | |
390955cb UD |
1289 | @code{*@var{result}}. |
1290 | ||
1291 | If the multibyte character code uses shift characters, then | |
1292 | @code{mbtowc} maintains and updates a shift state as it scans. If you | |
1293 | call @code{mbtowc} with a null pointer for @var{string}, that | |
1294 | initializes the shift state to its standard initial value. It also | |
1295 | returns nonzero if the multibyte character code in use actually has a | |
1296 | shift state. @xref{Shift State}. | |
1297 | @end deftypefun | |
1298 | ||
1299 | @comment stdlib.h | |
1300 | @comment ISO | |
1301 | @deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar}) | |
1302 | The @code{wctomb} (``wide character to multibyte'') function converts | |
1303 | the wide character code @var{wchar} to its corresponding multibyte | |
1304 | character sequence, and stores the result in bytes starting at | |
1305 | @var{string}. At most @code{MB_CUR_MAX} characters are stored. | |
1306 | ||
1307 | @code{wctomb} with non-null @var{string} distinguishes three | |
1308 | possibilities for @var{wchar}: a valid wide character code (one that can | |
7be8096f | 1309 | be translated to a multibyte character), an invalid code, and @code{L'\0'}. |
390955cb UD |
1310 | |
1311 | Given a valid code, @code{wctomb} converts it to a multibyte character, | |
1312 | storing the bytes starting at @var{string}. Then it returns the number | |
7be8096f | 1313 | of bytes in that character (always at least @math{1}, and never more |
390955cb UD |
1314 | than @code{MB_CUR_MAX}). |
1315 | ||
1316 | If @var{wchar} is an invalid wide character code, @code{wctomb} returns | |
7be8096f UD |
1317 | @math{-1}. If @var{wchar} is @code{L'\0'}, it returns @code{0}, also |
1318 | storing @code{'\0'} in @code{*@var{string}}. | |
390955cb UD |
1319 | |
1320 | If the multibyte character code uses shift characters, then | |
1321 | @code{wctomb} maintains and updates a shift state as it scans. If you | |
1322 | call @code{wctomb} with a null pointer for @var{string}, that | |
1323 | initializes the shift state to its standard initial value. It also | |
1324 | returns nonzero if the multibyte character code in use actually has a | |
1325 | shift state. @xref{Shift State}. | |
1326 | ||
1327 | Calling this function with a @var{wchar} argument of zero when | |
1328 | @var{string} is not null has the side-effect of reinitializing the | |
1329 | stored shift state @emph{as well as} storing the multibyte character | |
7be8096f | 1330 | @code{'\0'} and returning @math{0}. |
390955cb UD |
1331 | @end deftypefun |
1332 | ||
1333 | Similar to @code{mbrlen} there is also a non-reentrant function which | |
1334 | computes the length of a multibyte character. It can be defined in | |
1335 | terms of @code{mbtowc}. | |
1336 | ||
1337 | @comment stdlib.h | |
1338 | @comment ISO | |
1339 | @deftypefun int mblen (const char *@var{string}, size_t @var{size}) | |
1340 | The @code{mblen} function with a non-null @var{string} argument returns | |
1341 | the number of bytes that make up the multibyte character beginning at | |
1342 | @var{string}, never examining more than @var{size} bytes. (The idea is | |
1343 | to supply for @var{size} the number of bytes of data you have in hand.) | |
1344 | ||
1345 | The return value of @code{mblen} distinguishes three possibilities: the | |
1346 | first @var{size} bytes at @var{string} start with valid multibyte | |
1347 | character, they start with an invalid byte sequence or just part of a | |
1348 | character, or @var{string} points to an empty string (a null character). | |
1349 | ||
1350 | For a valid multibyte character, @code{mblen} returns the number of | |
1351 | bytes in that character (always at least @code{1}, and never more than | |
1352 | @var{size}). For an invalid byte sequence, @code{mblen} returns | |
7be8096f | 1353 | @math{-1}. For an empty string, it returns @math{0}. |
390955cb UD |
1354 | |
1355 | If the multibyte character code uses shift characters, then @code{mblen} | |
1356 | maintains and updates a shift state as it scans. If you call | |
1357 | @code{mblen} with a null pointer for @var{string}, that initializes the | |
7be8096f UD |
1358 | shift state to its standard initial value. It also returns a nonzero |
1359 | value if the multibyte character code in use actually has a shift state. | |
390955cb UD |
1360 | @xref{Shift State}. |
1361 | ||
1362 | @pindex stdlib.h | |
1363 | The function @code{mblen} is declared in @file{stdlib.h}. | |
1364 | @end deftypefun | |
1365 | ||
1366 | ||
1367 | @node Non-reentrant String Conversion | |
1368 | @subsection Non-reentrant Conversion of Strings | |
1369 | ||
976780fd | 1370 | For convenience reasons the @w{ISO C90} standard defines also functions |
390955cb UD |
1371 | to convert entire strings instead of single characters. These functions |
1372 | suffer from the same problems as their reentrant counterparts from the | |
976780fd | 1373 | second amendment to @w{ISO C90}; see @ref{Converting Strings}. |
390955cb UD |
1374 | |
1375 | @comment stdlib.h | |
1376 | @comment ISO | |
1377 | @deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size}) | |
1378 | The @code{mbstowcs} (``multibyte string to wide character string'') | |
1379 | function converts the null-terminated string of multibyte characters | |
1380 | @var{string} to an array of wide character codes, storing not more than | |
1381 | @var{size} wide characters into the array beginning at @var{wstring}. | |
1382 | The terminating null character counts towards the size, so if @var{size} | |
1383 | is less than the actual number of wide characters resulting from | |
1384 | @var{string}, no terminating null character is stored. | |
1385 | ||
1386 | The conversion of characters from @var{string} begins in the initial | |
1387 | shift state. | |
1388 | ||
1389 | If an invalid multibyte character sequence is found, this function | |
7be8096f | 1390 | returns a value of @math{-1}. Otherwise, it returns the number of wide |
390955cb UD |
1391 | characters stored in the array @var{wstring}. This number does not |
1392 | include the terminating null character, which is present if the number | |
1393 | is less than @var{size}. | |
1394 | ||
1395 | Here is an example showing how to convert a string of multibyte | |
1396 | characters, allocating enough space for the result. | |
1397 | ||
1398 | @smallexample | |
1399 | wchar_t * | |
1400 | mbstowcs_alloc (const char *string) | |
1401 | @{ | |
1402 | size_t size = strlen (string) + 1; | |
1403 | wchar_t *buf = xmalloc (size * sizeof (wchar_t)); | |
1404 | ||
1405 | size = mbstowcs (buf, string, size); | |
1406 | if (size == (size_t) -1) | |
1407 | return NULL; | |
1408 | buf = xrealloc (buf, (size + 1) * sizeof (wchar_t)); | |
1409 | return buf; | |
1410 | @} | |
1411 | @end smallexample | |
1412 | ||
1413 | @end deftypefun | |
1414 | ||
1415 | @comment stdlib.h | |
1416 | @comment ISO | |
1417 | @deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size}) | |
1418 | The @code{wcstombs} (``wide character string to multibyte string'') | |
1419 | function converts the null-terminated wide character array @var{wstring} | |
1420 | into a string containing multibyte characters, storing not more than | |
1421 | @var{size} bytes starting at @var{string}, followed by a terminating | |
1422 | null character if there is room. The conversion of characters begins in | |
1423 | the initial shift state. | |
1424 | ||
1425 | The terminating null character counts towards the size, so if @var{size} | |
1426 | is less than or equal to the number of bytes needed in @var{wstring}, no | |
1427 | terminating null character is stored. | |
1428 | ||
1429 | If a code that does not correspond to a valid multibyte character is | |
7be8096f | 1430 | found, this function returns a value of @math{-1}. Otherwise, the |
390955cb UD |
1431 | return value is the number of bytes stored in the array @var{string}. |
1432 | This number does not include the terminating null character, which is | |
1433 | present if the number is less than @var{size}. | |
1434 | @end deftypefun | |
1435 | ||
1436 | @node Shift State | |
1437 | @subsection States in Non-reentrant Functions | |
1438 | ||
1439 | In some multibyte character codes, the @emph{meaning} of any particular | |
1440 | byte sequence is not fixed; it depends on what other sequences have come | |
1441 | earlier in the same string. Typically there are just a few sequences | |
1442 | that can change the meaning of other sequences; these few are called | |
1443 | @dfn{shift sequences} and we say that they set the @dfn{shift state} for | |
1444 | other sequences that follow. | |
1445 | ||
1446 | To illustrate shift state and shift sequences, suppose we decide that | |
1447 | the sequence @code{0200} (just one byte) enters Japanese mode, in which | |
1448 | pairs of bytes in the range from @code{0240} to @code{0377} are single | |
1449 | characters, while @code{0201} enters Latin-1 mode, in which single bytes | |
1450 | in the range from @code{0240} to @code{0377} are characters, and | |
1451 | interpreted according to the ISO Latin-1 character set. This is a | |
1452 | multibyte code which has two alternative shift states (``Japanese mode'' | |
1453 | and ``Latin-1 mode''), and two shift sequences that specify particular | |
1454 | shift states. | |
1455 | ||
1456 | When the multibyte character code in use has shift states, then | |
1457 | @code{mblen}, @code{mbtowc} and @code{wctomb} must maintain and update | |
1458 | the current shift state as they scan the string. To make this work | |
1459 | properly, you must follow these rules: | |
1460 | ||
1461 | @itemize @bullet | |
1462 | @item | |
1463 | Before starting to scan a string, call the function with a null pointer | |
1464 | for the multibyte character address---for example, @code{mblen (NULL, | |
1465 | 0)}. This initializes the shift state to its standard initial value. | |
1466 | ||
1467 | @item | |
1468 | Scan the string one character at a time, in order. Do not ``back up'' | |
1469 | and rescan characters already scanned, and do not intersperse the | |
1470 | processing of different strings. | |
1471 | @end itemize | |
1472 | ||
1473 | Here is an example of using @code{mblen} following these rules: | |
1474 | ||
1475 | @smallexample | |
1476 | void | |
1477 | scan_string (char *s) | |
1478 | @{ | |
1479 | int length = strlen (s); | |
1480 | ||
d731df03 | 1481 | /* @r{Initialize shift state.} */ |
390955cb UD |
1482 | mblen (NULL, 0); |
1483 | ||
1484 | while (1) | |
1485 | @{ | |
1486 | int thischar = mblen (s, length); | |
d731df03 | 1487 | /* @r{Deal with end of string and invalid characters.} */ |
390955cb UD |
1488 | if (thischar == 0) |
1489 | break; | |
1490 | if (thischar == -1) | |
1491 | @{ | |
1492 | error ("invalid multibyte character"); | |
1493 | break; | |
1494 | @} | |
d731df03 | 1495 | /* @r{Advance past this character.} */ |
390955cb UD |
1496 | s += thischar; |
1497 | length -= thischar; | |
1498 | @} | |
1499 | @} | |
1500 | @end smallexample | |
1501 | ||
1502 | The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not | |
1503 | reentrant when using a multibyte code that uses a shift state. However, | |
1504 | no other library functions call these functions, so you don't have to | |
1505 | worry that the shift state will be changed mysteriously. | |
1506 | ||
1507 | ||
1508 | @node Generic Charset Conversion | |
1509 | @section Generic Charset Conversion | |
1510 | ||
1511 | The conversion functions mentioned so far in this chapter all had in | |
1512 | common that they operate on character sets which are not directly | |
1513 | specified by the functions. The multibyte encoding used is specified by | |
1514 | the currently selected locale for the @code{LC_CTYPE} category. The | |
1515 | wide character set is fixed by the implementation (in the case of GNU C | |
d731df03 | 1516 | library it always is UCS4 encoded @w{ISO 10646}. |
390955cb UD |
1517 | |
1518 | This has of course several problems when it comes to general character | |
1519 | conversion: | |
1520 | ||
1521 | @itemize @bullet | |
1522 | @item | |
1523 | For every conversion where neither the source or destination character | |
1524 | set is the character set of the locale for the @code{LC_CTYPE} category, | |
1525 | one has to change the @code{LC_CTYPE} locale using @code{setlocale}. | |
1526 | ||
1527 | This introduces major problems for the rest of the programs since | |
1528 | several more functions (e.g., the character classification functions, | |
464d646f | 1529 | @pxref{Classification of Characters}) use the @code{LC_CTYPE} category. |
390955cb UD |
1530 | |
1531 | @item | |
1532 | Parallel conversions to and from different character sets are not | |
1533 | possible since the @code{LC_CTYPE} selection is global and shared by all | |
1534 | threads. | |
1535 | ||
1536 | @item | |
1537 | If neither the source nor the destination character set is the character | |
1538 | set used for @code{wchar_t} representation there is at least a two-step | |
1539 | process necessary to convert a text using the functions above. One | |
1540 | would have to select the source character set as the multibyte encoding, | |
1541 | convert the text into a @code{wchar_t} text, select the destination | |
1542 | character set as the multibyte encoding and convert the wide character | |
7be8096f | 1543 | text to the multibyte (@math{=} destination) character set. |
390955cb UD |
1544 | |
1545 | Even if this is possible (which is not guaranteed) it is a very tiring | |
1546 | work. Plus it suffers from the other two raised points even more due to | |
1547 | the steady changing of the locale. | |
1548 | @end itemize | |
1549 | ||
1550 | ||
1551 | The XPG2 standard defines a completely new set of functions which has | |
1552 | none of these limitations. They are not at all coupled to the selected | |
1553 | locales and they but no constraints on the character sets selected for | |
1554 | source and destination. Only the set of available conversions is | |
1555 | limiting them. The standard does not specify that any conversion at all | |
1556 | must be available. It is a measure of the quality of the implementation. | |
1557 | ||
d731df03 UD |
1558 | In the following text first the interface to @code{iconv}, the |
1559 | conversion function, will be described. Comparisons with other | |
1560 | implementations will show what pitfalls lie on the way of portable | |
1561 | applications. At last, the implementation is described as far as | |
1562 | interesting to the advanced user who wants to extend the conversion | |
1563 | capabilities. | |
390955cb UD |
1564 | |
1565 | @menu | |
1566 | * Generic Conversion Interface:: Generic Character Set Conversion Interface. | |
1567 | * iconv Examples:: A complete @code{iconv} example. | |
1568 | * Other iconv Implementations:: Some Details about other @code{iconv} | |
1569 | Implementations. | |
1570 | * glibc iconv Implementation:: The @code{iconv} Implementation in the GNU C | |
1571 | library. | |
1572 | @end menu | |
1573 | ||
1574 | @node Generic Conversion Interface | |
1575 | @subsection Generic Character Set Conversion Interface | |
1576 | ||
1577 | This set of functions follows the traditional cycle of using a resource: | |
1578 | open--use--close. The interface consists of three functions, each of | |
1579 | which implement one step. | |
1580 | ||
1581 | Before the interfaces are described it is necessary to introduce a | |
1582 | datatype. Just like other open--use--close interface the functions | |
1583 | introduced here work using a handles and the @file{iconv.h} header | |
1584 | defines a special type for the handles used. | |
1585 | ||
1586 | @comment iconv.h | |
1587 | @comment XPG2 | |
1588 | @deftp {Data Type} iconv_t | |
1589 | This data type is an abstract type defined in @file{iconv.h}. The user | |
1590 | must not assume anything about the definition of this type, it must be | |
1591 | completely opaque. | |
1592 | ||
1593 | Objects of this type can get assigned handles for the conversions using | |
1594 | the @code{iconv} functions. The objects themselves need not be freed but | |
1595 | the conversions for which the handles stand for have to. | |
1596 | @end deftp | |
1597 | ||
1598 | @noindent | |
1599 | The first step is the function to create a handle. | |
1600 | ||
1601 | @comment iconv.h | |
1602 | @comment XPG2 | |
1603 | @deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode}) | |
1604 | The @code{iconv_open} function has to be used before starting a | |
1605 | conversion. The two parameters this function takes determine the | |
c1b2d472 | 1606 | source and destination character set for the conversion and if the |
390955cb UD |
1607 | implementation has the possibility to perform such a conversion the |
1608 | function returns a handle. | |
1609 | ||
1610 | If the wanted conversion is not available the function returns | |
1611 | @code{(iconv_t) -1}. In this case the global variable @code{errno} can | |
1612 | have the following values: | |
1613 | ||
1614 | @table @code | |
1615 | @item EMFILE | |
1616 | The process already has @code{OPEN_MAX} file descriptors open. | |
1617 | @item ENFILE | |
1618 | The system limit of open file is reached. | |
1619 | @item ENOMEM | |
1620 | Not enough memory to carry out the operation. | |
1621 | @item EINVAL | |
1622 | The conversion from @var{fromcode} to @var{tocode} is not supported. | |
1623 | @end table | |
1624 | ||
1625 | It is not possible to use the same descriptor in different threads to | |
1626 | perform independent conversions. Within the data structures associated | |
1627 | with the descriptor there is information about the conversion state. | |
d731df03 | 1628 | This must not be messed up by using it in different conversions. |
390955cb | 1629 | |
c1b2d472 | 1630 | An @code{iconv} descriptor is like a file descriptor as for every use a |
390955cb UD |
1631 | new descriptor must be created. The descriptor does not stand for all |
1632 | of the conversions from @var{fromset} to @var{toset}. | |
1633 | ||
1634 | The GNU C library implementation of @code{iconv_open} has one | |
1635 | significant extension to other implementations. To ease the extension | |
1636 | of the set of available conversions the implementation allows to store | |
1637 | the necessary files with data and code in arbitrary many directories. | |
1638 | How this extensions have to be written will be explained below | |
1639 | (@pxref{glibc iconv Implementation}). Here it is only important to say | |
1640 | that all directories mentioned in the @code{GCONV_PATH} environment | |
1641 | variable are considered if they contain a file @file{gconv-modules}. | |
1642 | These directories need not necessarily be created by the system | |
1643 | administrator. In fact, this extension is introduced to help users | |
1644 | writing and using own, new conversions. Of course this does not work | |
1645 | for security reasons in SUID binaries; in this case only the system | |
1646 | directory is considered and this normally is | |
1647 | @file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment | |
1648 | variable is examined exactly once at the first call of the | |
1649 | @code{iconv_open} function. Later modifications of the variable have no | |
1650 | effect. | |
1651 | ||
1652 | @pindex iconv.h | |
1653 | This function got introduced early in the X/Open Portability Guide, | |
1654 | @w{version 2}. It is supported by all commercial Unices as it is | |
d731df03 UD |
1655 | required for the Unix branding. However, the quality and completeness |
1656 | of the implementation varies widely. The function is declared in | |
390955cb UD |
1657 | @file{iconv.h}. |
1658 | @end deftypefun | |
1659 | ||
1660 | The @code{iconv} implementation can associate large data structure with | |
1661 | the handle returned by @code{iconv_open}. Therefore it is crucial to | |
1662 | free all the resources once all conversions are carried out and the | |
1663 | conversion is not needed anymore. | |
1664 | ||
1665 | @comment iconv.h | |
1666 | @comment XPG2 | |
1667 | @deftypefun int iconv_close (iconv_t @var{cd}) | |
1668 | The @code{iconv_close} function frees all resources associated with the | |
1669 | handle @var{cd} which must have been returned by a successful call to | |
1670 | the @code{iconv_open} function. | |
1671 | ||
1672 | If the function call was successful the return value is @math{0}. | |
1673 | Otherwise it is @math{-1} and @code{errno} is set appropriately. | |
1674 | Defined error are: | |
1675 | ||
1676 | @table @code | |
1677 | @item EBADF | |
1678 | The conversion descriptor is invalid. | |
1679 | @end table | |
1680 | ||
1681 | @pindex iconv.h | |
1682 | This function was introduced together with the rest of the @code{iconv} | |
1683 | functions in XPG2 and it is declared in @file{iconv.h}. | |
1684 | @end deftypefun | |
1685 | ||
1686 | The standard defines only one actual conversion function. This has | |
1687 | therefore the most general interface: it allows conversion from one | |
1688 | buffer to another. Conversion from a file to a buffer, vice versa, or | |
1689 | even file to file can be implemented on top of it. | |
1690 | ||
1691 | @comment iconv.h | |
1692 | @comment XPG2 | |
1693 | @deftypefun size_t iconv (iconv_t @var{cd}, const char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft}) | |
1694 | @cindex stateful | |
1695 | The @code{iconv} function converts the text in the input buffer | |
1696 | according to the rules associated with the descriptor @var{cd} and | |
1697 | stores the result in the output buffer. It is possible to call the | |
1698 | function for the same text several times in a row since for stateful | |
1699 | character sets the necessary state information is kept in the data | |
1700 | structures associated with the descriptor. | |
1701 | ||
1702 | The input buffer is specified by @code{*@var{inbuf}} and it contains | |
1703 | @code{*@var{inbytesleft}} bytes. The extra indirection is necessary for | |
1704 | communicating the used input back to the caller (see below). It is | |
1705 | important to note that the buffer pointer is of type @code{char} and the | |
1706 | length is measured in bytes even if the input text is encoded in wide | |
1707 | characters. | |
1708 | ||
1709 | The output buffer is specified in a similar way. @code{*@var{outbuf}} | |
1710 | points to the beginning of the buffer with at least | |
1711 | @code{*@var{outbytesleft}} bytes room for the result. The buffer | |
1712 | pointer again is of type @code{char} and the length is measured in | |
1713 | bytes. If @var{outbuf} or @code{*@var{outbuf}} is a null pointer the | |
1714 | conversion is performed but no output is available. | |
1715 | ||
1716 | If @var{inbuf} is a null pointer the @code{iconv} function performs the | |
1717 | necessary action to put the state of the conversion into the initial | |
1718 | state. This is obviously a no-op for non-stateful encodings, but if the | |
1719 | encoding has a state such a function call might put some byte sequences | |
1720 | in the output buffer which perform the necessary state changes. The | |
1721 | next call with @var{inbuf} not being a null pointer then simply goes on | |
1722 | from the initial state. It is important that the programmer never makes | |
1723 | any assumption on whether the conversion has to deal with states or not. | |
1724 | Even if the input and output character sets are not stateful the | |
1725 | implementation might still have to keep states. This is due to the | |
1726 | implementation chosen for the GNU C library as it is described below. | |
1727 | Therefore an @code{iconv} call to reset the state should always be | |
1728 | performed if some protocol requires this for the output text. | |
1729 | ||
1730 | The conversion stops for three reasons. The first is that all | |
1731 | characters from the input buffer are converted. This actually can mean | |
c1b2d472 | 1732 | two things: really all bytes from the input buffer are consumed or |
390955cb UD |
1733 | there are some bytes at the end of the buffer which possibly can form a |
1734 | complete character but the input is incomplete. The second reason for a | |
1735 | stop is when the output buffer is full. And the third reason is that | |
1736 | the input contains invalid characters. | |
1737 | ||
1738 | In all these cases the buffer pointers after the last successful | |
1739 | conversion, for input and output buffer, are stored in @var{inbuf} and | |
1740 | @var{outbuf} and the available room in each buffer is stored in | |
1741 | @var{inbytesleft} and @var{outbytesleft}. | |
1742 | ||
1743 | Since the character sets selected in the @code{iconv_open} call can be | |
1744 | almost arbitrary there can be situations where the input buffer contains | |
1745 | valid characters which have no identical representation in the output | |
1746 | character set. The behavior in this situation is undefined. The | |
1747 | @emph{current} behavior of the GNU C library in this situation is to | |
1748 | return with an error immediately. This certainly is not the most | |
1749 | desirable solution. Therefore future versions will provide better ones | |
1750 | but they are not yet finished. | |
1751 | ||
1752 | If all input from the input buffer is successfully converted and stored | |
c1b2d472 | 1753 | in the output buffer the function returns the number of conversions |
390955cb UD |
1754 | performed. In all other cases the return value is @code{(size_t) -1} |
1755 | and @code{errno} is set appropriately. In this case the value pointed | |
1756 | to by @var{inbytesleft} is nonzero. | |
1757 | ||
1758 | @table @code | |
1759 | @item EILSEQ | |
1760 | The conversion stopped because of an invalid byte sequence in the input. | |
1761 | After the call @code{*@var{inbuf}} points at the first byte of the | |
1762 | invalid byte sequence. | |
1763 | ||
1764 | @item E2BIG | |
1765 | The conversion stopped because it ran out of space in the output buffer. | |
1766 | ||
1767 | @item EINVAL | |
1768 | The conversion stopped because of an incomplete byte sequence at the end | |
1769 | of the input buffer. | |
1770 | ||
1771 | @item EBADF | |
1772 | The @var{cd} argument is invalid. | |
1773 | @end table | |
1774 | ||
1775 | @pindex iconv.h | |
1776 | This function was introduced in the XPG2 standard and is declared in the | |
1777 | @file{iconv.h} header. | |
1778 | @end deftypefun | |
1779 | ||
1780 | The definition of the @code{iconv} function is quite good overall. It | |
1781 | provides quite flexible functionality. The only problems lie in the | |
1782 | boundary cases which are incomplete byte sequences at the end of the | |
d731df03 UD |
1783 | input buffer and invalid input. A third problem, which is not really |
1784 | a design problem, is the way conversions are selected. The standard | |
1785 | does not say anything about the legitimate names, a minimal set of | |
1786 | available conversions. We will see how this negatively impacts other | |
1787 | implementations, as is demonstrated below. | |
390955cb UD |
1788 | |
1789 | ||
1790 | @node iconv Examples | |
1791 | @subsection A complete @code{iconv} example | |
1792 | ||
1793 | The example below features a solution for a common problem. Given that | |
1794 | one knows the internal encoding used by the system for @code{wchar_t} | |
1795 | strings one often is in the position to read text from a file and store | |
1796 | it in wide character buffers. One can do this using @code{mbsrtowcs} | |
1797 | but then we run into the problems discussed above. | |
1798 | ||
1799 | @smallexample | |
1800 | int | |
1801 | file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail) | |
1802 | @{ | |
1803 | char inbuf[BUFSIZ]; | |
1804 | size_t insize = 0; | |
1805 | char *wrptr = (char *) outbuf; | |
1806 | int result = 0; | |
1807 | iconv_t cd; | |
1808 | ||
1809 | cd = iconv_open ("UCS4", charset); | |
1810 | if (cd == (iconv_t) -1) | |
1811 | @{ | |
1812 | /* @r{Something went wrong.} */ | |
1813 | if (errno == EINVAL) | |
1814 | error (0, 0, "conversion from `%s' to `UCS4' no available", | |
1815 | charset); | |
1816 | else | |
1817 | perror ("iconv_open"); | |
1818 | ||
1819 | /* @r{Terminate the output string.} */ | |
1820 | *outbuf = L'\0'; | |
1821 | ||
1822 | return -1; | |
1823 | @} | |
1824 | ||
1825 | while (avail > 0) | |
1826 | @{ | |
1827 | size_t nread; | |
1828 | size_t nconv; | |
1829 | char *inptr = inbuf; | |
1830 | ||
1831 | /* @r{Read more input.} */ | |
1832 | nread = read (fd, inbuf + insize, sizeof (inbuf) - insize); | |
1833 | if (nread == 0) | |
1834 | @{ | |
1835 | /* @r{When we come here the file is completely read.} | |
1836 | @r{This still could mean there are some unused} | |
1837 | @r{characters in the @code{inbuf}. Put them back.} */ | |
1838 | if (lseek (fd, -insize, SEEK_CUR) == -1) | |
1839 | result = -1; | |
c559a3ca UD |
1840 | |
1841 | /* @r{Now write out the byte sequence to get into the} | |
1842 | @r{initial state if this is necessary.} */ | |
1843 | iconv (cd, NULL, NULL, &wrptr, &avail); | |
1844 | ||
390955cb UD |
1845 | break; |
1846 | @} | |
1847 | insize += nread; | |
1848 | ||
1849 | /* @r{Do the conversion.} */ | |
1850 | nconv = iconv (cd, &inptr, &insize, &wrptr, &avail); | |
1851 | if (nconv == (size_t) -1) | |
1852 | @{ | |
1853 | /* @r{Not everything went right. It might only be} | |
1854 | @r{an unfinished byte sequence at the end of the} | |
1855 | @r{buffer. Or it is a real problem.} */ | |
1856 | if (errno == EINVAL) | |
1857 | /* @r{This is harmless. Simply move the unused} | |
1858 | @r{bytes to the beginning of the buffer so that} | |
1859 | @r{they can be used in the next round.} */ | |
1860 | memmove (inbuf, inptr, insize); | |
1861 | else | |
1862 | @{ | |
1863 | /* @r{It is a real problem. Maybe we ran out of} | |
1864 | @r{space in the output buffer or we have invalid} | |
1865 | @r{input. In any case back the file pointer to} | |
1866 | @r{the position of the last processed byte.} */ | |
1867 | lseek (fd, -insize, SEEK_CUR); | |
1868 | result = -1; | |
1869 | break; | |
1870 | @} | |
1871 | @} | |
1872 | @} | |
1873 | ||
1874 | /* @r{Terminate the output string.} */ | |
c559a3ca UD |
1875 | if (avail >= sizeof (wchar_t)) |
1876 | *((wchar_t *) wrptr) = L'\0'; | |
390955cb UD |
1877 | |
1878 | if (iconv_close (cd) != 0) | |
1879 | perror ("iconv_close"); | |
1880 | ||
1881 | return (wchar_t *) wrptr - outbuf; | |
1882 | @} | |
1883 | @end smallexample | |
1884 | ||
1885 | @cindex stateful | |
1886 | This example shows the most important aspects of using the @code{iconv} | |
1887 | functions. It shows how successive calls to @code{iconv} can be used to | |
1888 | convert large amounts of text. The user does not have to care about | |
1889 | stateful encodings as the functions take care of everything. | |
1890 | ||
1891 | An interesting point is the case where @code{iconv} return an error and | |
1892 | @code{errno} is set to @code{EINVAL}. This is not really an error in | |
1893 | the transformation. It can happen whenever the input character set | |
1894 | contains byte sequences of more than one byte for some character and | |
1895 | texts are not processed in one piece. In this case there is a chance | |
1896 | that a multibyte sequence is cut. The caller than can simply read the | |
1897 | remainder of the takes and feed the offending bytes together with new | |
1898 | character from the input to @code{iconv} and continue the work. The | |
1899 | internal state kept in the descriptor is @emph{not} unspecified after | |
1900 | such an event as it is the case with the conversion functions from the | |
1901 | @w{ISO C} standard. | |
1902 | ||
1903 | The example also shows the problem of using wide character strings with | |
1904 | @code{iconv}. As explained in the description of the @code{iconv} | |
1905 | function above the function always takes a pointer to a @code{char} | |
1906 | array and the available space is measured in bytes. In the example the | |
1907 | output buffer is a wide character buffer. Therefore we use a local | |
1908 | variable @var{wrptr} of type @code{char *} which is used in the | |
1909 | @code{iconv} calls. | |
1910 | ||
1911 | This looks rather innocent but can lead to problems on platforms which | |
1912 | have tight restriction on alignment. Therefore the caller of | |
1913 | @code{iconv} has to make sure that the pointers passed are suitable for | |
1914 | access of characters from the appropriate character set. Since in the | |
1915 | above case the input parameter to the function is a @code{wchar_t} | |
1916 | pointer this is the case (unless the user violates alignment when | |
1917 | computing the parameter). But in other situations, especially when | |
1918 | writing generic functions where one does not know what type of character | |
c1b2d472 | 1919 | set one uses and therefore treats text as a sequence of bytes, it might |
390955cb UD |
1920 | become tricky. |
1921 | ||
1922 | ||
1923 | @node Other iconv Implementations | |
1924 | @subsection Some Details about other @code{iconv} Implementations | |
1925 | ||
1926 | This is not really the place to discuss the @code{iconv} implementation | |
1927 | of other systems but it is necessary to know a bit about them to write | |
1928 | portable programs. The above mentioned problems with the specification | |
1929 | of the @code{iconv} functions can lead to portability issues. | |
1930 | ||
1931 | The first thing to notice is that due to the large number of character | |
1932 | sets in use it is certainly not practical to encode the conversions | |
1933 | directly in the C library. Therefore the conversion information must | |
d731df03 UD |
1934 | come from files outside the C library. This is usually done in one or |
1935 | both of the following ways: | |
390955cb UD |
1936 | |
1937 | @itemize @bullet | |
1938 | @item | |
1939 | The C library contains a set of generic conversion functions which can | |
1940 | read the needed conversion tables and other information from data files. | |
1941 | These files get loaded when necessary. | |
1942 | ||
d731df03 UD |
1943 | This solution is problematic as it requires a great deal of effort to |
1944 | apply to all character sets (potentially an infinite set). The | |
1945 | differences in the structure of the different character sets is so large | |
390955cb UD |
1946 | that many different variants of the table processing functions must be |
1947 | developed. On top of this the generic nature of these functions make | |
1948 | them slower than specifically implemented functions. | |
1949 | ||
1950 | @item | |
1951 | The C library only contains a framework which can dynamically load | |
1952 | object files and execute the therein contained conversion functions. | |
1953 | ||
1954 | This solution provides much more flexibility. The C library itself | |
1955 | contains only very little code and therefore reduces the general memory | |
1956 | footprint. Also, with a documented interface between the C library and | |
1957 | the loadable modules it is possible for third parties to extend the set | |
1958 | of available conversion modules. A drawback of this solution is that | |
1959 | dynamic loading must be available. | |
1960 | @end itemize | |
1961 | ||
1962 | Some implementations in commercial Unices implement a mixture of these | |
d731df03 UD |
1963 | these possibilities, the majority only the second solution. Using |
1964 | loadable modules moves the code out of the library itself and keeps the | |
1965 | door open for extensions and improvements. But this design is also | |
1966 | limiting on some platforms since not many platforms support dynamic | |
1967 | loading in statically linked programs. On platforms without his | |
1968 | capability it is therefore not possible to use this interface in | |
1969 | statically linked programs. The GNU C library has on ELF platforms no | |
1970 | problems with dynamic loading in in these situations and therefore this | |
1971 | point is mood. The danger is that one gets acquainted with this and | |
1972 | forgets about the restrictions on other systems. | |
390955cb UD |
1973 | |
1974 | A second thing to know about other @code{iconv} implementations is that | |
c1b2d472 | 1975 | the number of available conversions is often very limited. Some |
d731df03 UD |
1976 | implementations provide in the standard release (not special |
1977 | international or developer releases) at most 100 to 200 conversion | |
1978 | possibilities. This does not mean 200 different character sets are | |
1979 | supported. E.g., conversions from one character set to a set of, say, | |
1980 | 10 others counts as 10 conversion. Together with the other direction | |
1981 | this makes already 20. One can imagine the thin coverage these platform | |
1982 | provide. Some Unix vendors even provide only a handful of conversions | |
1983 | which renders them useless for almost all uses. | |
390955cb UD |
1984 | |
1985 | This directly leads to a third and probably the most problematic point. | |
1986 | The way the @code{iconv} conversion functions are implemented on all | |
c1b2d472 | 1987 | known Unix system and the availability of the conversion functions from |
390955cb UD |
1988 | character set @math{@cal{A}} to @math{@cal{B}} and the conversion from |
1989 | @math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the | |
1990 | conversion from @math{@cal{A}} to @math{@cal{C}} is available. | |
1991 | ||
1992 | This might not seem unreasonable and problematic at first but it is a | |
1993 | quite big problem as one will notice shortly after hitting it. To show | |
1994 | the problem we assume to write a program which has to convert from | |
1995 | @math{@cal{A}} to @math{@cal{C}}. A call like | |
1996 | ||
1997 | @smallexample | |
1998 | cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}"); | |
1999 | @end smallexample | |
2000 | ||
2001 | @noindent | |
2002 | does fail according to the assumption above. But what does the program | |
2003 | do now? The conversion is really necessary and therefore simply giving | |
2004 | up is no possibility. | |
2005 | ||
d731df03 UD |
2006 | This is a nuisance. The @code{iconv} function should take care of this. |
2007 | But how should the program proceed from here on? If it would try to | |
2008 | convert to character set @math{@cal{B}} first the two @code{iconv_open} | |
2009 | calls | |
390955cb UD |
2010 | |
2011 | @smallexample | |
2012 | cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}"); | |
2013 | @end smallexample | |
2014 | ||
2015 | @noindent | |
2016 | and | |
2017 | ||
2018 | @smallexample | |
2019 | cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}"); | |
2020 | @end smallexample | |
2021 | ||
2022 | @noindent | |
2023 | will succeed but how to find @math{@cal{B}}? | |
2024 | ||
d731df03 | 2025 | Unfortunately, the answer is: there is no general solution. On some |
390955cb | 2026 | systems guessing might help. On those systems most character sets can |
d731df03 UD |
2027 | convert to and from UTF8 encoded @w{ISO 10646} or Unicode text. |
2028 | Beside this only some very system-specific methods can help. Since the | |
390955cb UD |
2029 | conversion functions come from loadable modules and these modules must |
2030 | be stored somewhere in the filesystem, one @emph{could} try to find them | |
2031 | and determine from the available file which conversions are available | |
2032 | and whether there is an indirect route from @math{@cal{A}} to | |
2033 | @math{@cal{C}}. | |
2034 | ||
2035 | This shows one of the design errors of @code{iconv} mentioned above. It | |
2036 | should at least be possible to determine the list of available | |
2037 | conversion programmatically so that if @code{iconv_open} says there is | |
2038 | no such conversion, one could make sure this also is true for indirect | |
2039 | routes. | |
2040 | ||
2041 | ||
2042 | @node glibc iconv Implementation | |
2043 | @subsection The @code{iconv} Implementation in the GNU C library | |
2044 | ||
2045 | After reading about the problems of @code{iconv} implementations in the | |
d731df03 UD |
2046 | last section it is certainly good to note that the implementation in |
2047 | the GNU C library has none of the problems mentioned above. What | |
2048 | follows is a step-by-step analysis of the points raised above. The | |
390955cb UD |
2049 | evaluation is based on the current state of the development (as of |
2050 | January 1999). The development of the @code{iconv} functions is not | |
d731df03 | 2051 | complete, but basic funtionality has solidified. |
390955cb UD |
2052 | |
2053 | The GNU C library's @code{iconv} implementation uses shared loadable | |
2054 | modules to implement the conversions. A very small number of | |
2055 | conversions are built into the library itself but these are only rather | |
2056 | trivial conversions. | |
2057 | ||
2058 | All the benefits of loadable modules are available in the GNU C library | |
d731df03 | 2059 | implementation. This is especially appealing since the interface is |
390955cb | 2060 | well documented (see below) and it therefore is easy to write new |
d731df03 | 2061 | conversion modules. The drawback of using loadable objects is not a |
390955cb UD |
2062 | problem in the GNU C library, at least on ELF systems. Since the |
2063 | library is able to load shared objects even in statically linked | |
d731df03 UD |
2064 | binaries this means that static linking needs not to be forbidden in |
2065 | case one wants to use @code{iconv}. | |
390955cb | 2066 | |
d731df03 UD |
2067 | The second mentioned problem is the number of supported conversions. |
2068 | Currently, the GNU C library supports more than 150 character sets. The | |
c1b2d472 UD |
2069 | way the implementation is designed the number of supported conversions |
2070 | is greater than 22350 (@math{150} times @math{149}). If any conversion | |
390955cb UD |
2071 | from or to a character set is missing it can easily be added. |
2072 | ||
d731df03 UD |
2073 | Particularly impressive as it may be, this high number is due to the |
2074 | fact that the GNU C library implementation of @code{iconv} does not have | |
2075 | the third problem mentioned above. I.e., whenever there is a conversion | |
2076 | from a character set @math{@cal{A}} to @math{@cal{B}} and from | |
2077 | @math{@cal{B}} to @math{@cal{C}} it is always possible to convert from | |
2078 | @math{@cal{A}} to @math{@cal{C}} directly. If the @code{iconv_open} | |
2079 | returns an error and sets @code{errno} to @code{EINVAL} this really | |
2080 | means there is no known way, directly or indirectly, to perform the | |
2081 | wanted conversion. | |
390955cb UD |
2082 | |
2083 | @cindex triangulation | |
2084 | This is achieved by providing for each character set a conversion from | |
2085 | and to UCS4 encoded @w{ISO 10646}. Using @w{ISO 10646} as an | |
d731df03 UD |
2086 | intermediate representation it is possible to @dfn{triangulate}, i.e., |
2087 | converting with an intermediate representation. | |
390955cb UD |
2088 | |
2089 | There is no inherent requirement to provide a conversion to @w{ISO | |
2090 | 10646} for a new character set and it is also possible to provide other | |
d731df03 | 2091 | conversions where neither source nor destination character set is @w{ISO |
c1b2d472 | 2092 | 10646}. The currently existing set of conversions is simply meant to |
d731df03 | 2093 | cover all conversions which might be of interest. |
390955cb UD |
2094 | |
2095 | @cindex ISO-2022-JP | |
2096 | @cindex EUC-JP | |
d731df03 UD |
2097 | All currently available conversions use the triangulation method above, |
2098 | making conversion run unnecessarily slow. If, e.g., somebody often | |
2099 | needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution | |
2100 | would involve direct conversion between the two character sets, skipping | |
2101 | the input to @w{ISO 10646} first. The two character sets of interest | |
2102 | are much more similar to each other than to @w{ISO 10646}. | |
390955cb UD |
2103 | |
2104 | In such a situation one can easy write a new conversion and provide it | |
2105 | as a better alternative. The GNU C library @code{iconv} implementation | |
2106 | would automatically use the module implementing the conversion if it is | |
2107 | specified to be more efficient. | |
2108 | ||
2109 | @subsubsection Format of @file{gconv-modules} files | |
2110 | ||
2111 | All information about the available conversions comes from a file named | |
2112 | @file{gconv-modules} which can be found in any of the directories along | |
2113 | the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented | |
2114 | text files, where each of the lines has one of the following formats: | |
2115 | ||
2116 | @itemize @bullet | |
2117 | @item | |
2118 | If the first non-whitespace character is a @kbd{#} the line contains | |
c1b2d472 | 2119 | only comments and is ignored. |
390955cb UD |
2120 | |
2121 | @item | |
2122 | Lines starting with @code{alias} define an alias name for a character | |
2123 | set. There are two more words expected on the line. The first one | |
2124 | defines the alias name and the second defines the original name of the | |
2125 | character set. The effect is that it is possible to use the alias name | |
2126 | in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and | |
2127 | achieve the same result as when using the real character set name. | |
2128 | ||
2129 | This is quite important as a character set has often many different | |
2130 | names. There is normally always an official name but this need not | |
2131 | correspond to the most popular name. Beside this many character sets | |
2132 | have special names which are somehow constructed. E.g., all character | |
2133 | sets specified by the ISO have an alias of the form | |
2134 | @code{ISO-IR-@var{nnn}} where @var{nnn} is the registration number. | |
2135 | This allows programs which know about the registration number to | |
2136 | construct character set names and use them in @code{iconv_open} calls. | |
c1b2d472 | 2137 | More on the available names and aliases follows below. |
390955cb UD |
2138 | |
2139 | @item | |
2140 | Lines starting with @code{module} introduce an available conversion | |
2141 | module. These lines must contain three or four more words. | |
2142 | ||
2143 | The first word specifies the source character set, the second word the | |
2144 | destination character set of conversion implemented in this module. The | |
2145 | third word is the name of the loadable module. The filename is | |
110215a9 | 2146 | constructed by appending the usual shared object suffix (normally |
390955cb UD |
2147 | @file{.so}) and this file is then supposed to be found in the same |
2148 | directory the @file{gconv-modules} file is in. The last word on the | |
2149 | line, which is optional, is a numeric value representing the cost of the | |
2150 | conversion. If this word is missing a cost of @math{1} is assumed. The | |
2151 | numeric value itself does not matter that much; what counts are the | |
2152 | relative values of the sums of costs for all possible conversion paths. | |
2153 | Below is a more precise description of the use of the cost value. | |
2154 | @end itemize | |
2155 | ||
d731df03 | 2156 | Returning to the example above where one has written a module to directly |
390955cb UD |
2157 | convert from ISO-2022-JP to EUC-JP and back. All what has to be done is |
2158 | to put the new module, be its name ISO2022JP-EUCJP.so, in a directory | |
2159 | and add a file @file{gconv-modules} with the following content in the | |
2160 | same directory: | |
2161 | ||
2162 | @smallexample | |
2163 | module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1 | |
2164 | module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1 | |
2165 | @end smallexample | |
2166 | ||
d731df03 UD |
2167 | To see why this is sufficient, it is necessary to understand how the |
2168 | conversion used by @code{iconv} (and described in the descriptor) is | |
390955cb UD |
2169 | selected. The approach to this problem is quite simple. |
2170 | ||
2171 | At the first call of the @code{iconv_open} function the program reads | |
2172 | all available @file{gconv-modules} files and builds up two tables: one | |
2173 | containing all the known aliases and another which contains the | |
2174 | information about the conversions and which shared object implements | |
2175 | them. | |
2176 | ||
2177 | @subsubsection Finding the conversion path in @code{iconv} | |
2178 | ||
2179 | The set of available conversions form a directed graph with weighted | |
d731df03 UD |
2180 | edges. The weights on the edges are the costs specified in the |
2181 | @file{gconv-modules} files. The @code{iconv_open} function uses an | |
2182 | algorithm suitable for search for the best path in such a graph and so | |
2183 | constructs a list of conversions which must be performed in succession | |
2184 | to get the transformation from the source to the destination character | |
2185 | set. | |
2186 | ||
2187 | Explaining why the above @file{gconv-modules} files allows the | |
2188 | @code{iconv} implementation to resolve the specific ISO-2022-JP to | |
2189 | EUC-JP conversion module instead of the conversion coming with the | |
2190 | library itself is straighforward. Since the later conversion takes two | |
2191 | steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to | |
390955cb UD |
2192 | EUC-JP) the cost is @math{1+1 = 2}. But the above @file{gconv-modules} |
2193 | file specifies that the new conversion modules can perform this | |
2194 | conversion with only the cost of @math{1}. | |
2195 | ||
d731df03 UD |
2196 | A mysterious piece about the @file{gconv-modules} file above (and also |
2197 | the file coming with the GNU C library) are the names of the character | |
2198 | sets specified in the @code{module} lines. Why do almost all the names | |
2199 | end in @code{//}? And this is not all: the names can actually be | |
2200 | regular expressions. At this point of time this mystery should not be | |
2201 | revealed, unless you have the relevant spell-casting materials: ashes | |
2202 | from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix | |
2203 | blessed by St.@: Emacs, assorted herbal roots from Central America, sand | |
2204 | from Cebu, etc. Sorry! @strong{The part of the implementation where | |
2205 | this is used is not yet finished. For now please simply follow the | |
2206 | existing examples. It'll become clearer once it is. --drepper} | |
390955cb UD |
2207 | |
2208 | A last remark about the @file{gconv-modules} is about the names not | |
2209 | ending with @code{//}. There often is a character set named | |
2210 | @code{INTERNAL} mentioned. From the discussion above and the chosen | |
110215a9 | 2211 | name it should have become clear that this is the name for the |
390955cb UD |
2212 | representation used in the intermediate step of the triangulation. We |
2213 | have said that this is UCS4 but actually it is not quite right. The | |
2214 | UCS4 specification also includes the specification of the byte ordering | |
110215a9 | 2215 | used. Since a UCS4 value consists of four bytes a stored value is |
390955cb UD |
2216 | effected by byte ordering. The internal representation is @emph{not} |
2217 | the same as UCS4 in case the byte ordering of the processor (or at least | |
2218 | the running process) is not the same as the one required for UCS4. This | |
2219 | is done for performance reasons as one does not want to perform | |
2220 | unnecessary byte-swapping operations if one is not interested in actually | |
2221 | seeing the result in UCS4. To avoid trouble with endianess the internal | |
2222 | representation consistently is named @code{INTERNAL} even on big-endian | |
2223 | systems where the representations are identical. | |
2224 | ||
2225 | @subsubsection @code{iconv} module data structures | |
2226 | ||
2227 | So far this section described how modules are located and considered to | |
2228 | be used. What remains to be described is the interface of the modules | |
2229 | so that one can write new ones. This section describes the interface as | |
2230 | it is in use in January 1999. The interface will change in future a bit | |
2231 | but hopefully only in an upward compatible way. | |
2232 | ||
2233 | The definitions necessary to write new modules are publically available | |
2234 | in the non-standard header @file{gconv.h}. The following text will | |
2235 | therefore describe the definitions from this header file. But first it | |
2236 | is necessary to get an overview. | |
2237 | ||
2238 | From the perspective of the user of @code{iconv} the interface is quite | |
2239 | simple: the @code{iconv_open} function returns a handle which can be | |
110215a9 | 2240 | used in calls to @code{iconv} and finally the handle is freed with a call |
390955cb UD |
2241 | to @code{iconv_close}. The problem is: the handle has to be able to |
2242 | represent the possibly long sequences of conversion steps and also the | |
2243 | state of each conversion since the handle is all which is passed to the | |
2244 | @code{iconv} function. Therefore the data structures are really the | |
2245 | elements to understanding the implementation. | |
2246 | ||
2247 | We need two different kinds of data structures. The first describes the | |
2248 | conversion and the second describes the state etc. There are really two | |
2249 | type definitions like this in @file{gconv.h}. | |
2250 | @pindex gconv.h | |
2251 | ||
2252 | @comment gconv.h | |
2253 | @comment GNU | |
976780fd | 2254 | @deftp {Data type} {struct __gconv_step} |
390955cb UD |
2255 | This data structure describes one conversion a module can perform. For |
2256 | each function in a loaded module with conversion functions there is | |
2257 | exactly one object of this type. This object is shared by all users of | |
2258 | the conversion. I.e., this object does not contain any information | |
2259 | corresponding to an actual conversion. It only describes the conversion | |
2260 | itself. | |
2261 | ||
2262 | @table @code | |
976780fd UD |
2263 | @item struct __gconv_loaded_object *__shlib_handle |
2264 | @itemx const char *__modname | |
2265 | @itemx int __counter | |
390955cb UD |
2266 | All these elements of the structure are used internally in the C library |
2267 | to coordinate loading and unloading the shared. One must not expect any | |
2268 | of the other elements be available or initialized. | |
2269 | ||
976780fd UD |
2270 | @item const char *__from_name |
2271 | @itemx const char *__to_name | |
2272 | @code{__from_name} and @code{__to_name} contain the names of the source and | |
390955cb UD |
2273 | destination character sets. They can be used to identify the actual |
2274 | conversion to be carried out since one module might implement | |
2275 | conversions for more than one character set and/or direction. | |
2276 | ||
976780fd UD |
2277 | @item gconv_fct __fct |
2278 | @itemx gconv_init_fct __init_fct | |
2279 | @itemx gconv_end_fct __end_fct | |
390955cb UD |
2280 | These elements contain pointers to the functions in the loadable module. |
2281 | The interface will be explained below. | |
2282 | ||
976780fd UD |
2283 | @item int __min_needed_from |
2284 | @itemx int __max_needed_from | |
2285 | @itemx int __min_needed_to | |
2286 | @itemx int __max_needed_to; | |
2287 | These values have to be filled in the init function of the module. The | |
2288 | @code{__min_needed_from} value specifies how many bytes a character of | |
2289 | the source character set at least needs. The @code{__max_needed_from} | |
390955cb UD |
2290 | specifies the maximum value which also includes possible shift |
2291 | sequences. | |
2292 | ||
976780fd UD |
2293 | The @code{__min_needed_to} and @code{__max_needed_to} values serve the |
2294 | same purpose but this time for the destination character set. | |
390955cb UD |
2295 | |
2296 | It is crucial that these values are accurate since otherwise the | |
2297 | conversion functions will have problems or not work at all. | |
2298 | ||
976780fd | 2299 | @item int __stateful |
390955cb UD |
2300 | This element must also be initialized by the init function. It is |
2301 | nonzero if the source character set is stateful. Otherwise it is zero. | |
2302 | ||
976780fd | 2303 | @item void *__data |
390955cb UD |
2304 | This element can be used freely by the conversion functions in the |
2305 | module. It can be used to communicate extra information from one call | |
2306 | to another. It need not be initialized if not needed at all. If this | |
2307 | element gets assigned a pointer to dynamically allocated memory | |
2308 | (presumably in the init function) it has to be made sure that the end | |
2309 | function deallocates the memory. Otherwise the application will leak | |
2310 | memory. | |
2311 | ||
2312 | It is important to be aware that this data structure is shared by all | |
976780fd | 2313 | users of this specification conversion and therefore the @code{__data} |
390955cb UD |
2314 | element must not contain data specific to one specific use of the |
2315 | conversion function. | |
2316 | @end table | |
2317 | @end deftp | |
2318 | ||
2319 | @comment gconv.h | |
2320 | @comment GNU | |
976780fd | 2321 | @deftp {Data type} {struct __gconv_step_data} |
390955cb UD |
2322 | This is the data structure which contains the information specific to |
2323 | each use of the conversion functions. | |
2324 | ||
2325 | @table @code | |
976780fd UD |
2326 | @item char *__outbuf |
2327 | @itemx char *__outbufend | |
390955cb | 2328 | These elements specify the output buffer for the conversion step. The |
976780fd UD |
2329 | @code{__outbuf} element points to the beginning of the buffer and |
2330 | @code{__outbufend} points to the byte following the last byte in the | |
390955cb UD |
2331 | buffer. The conversion function must not assume anything about the size |
2332 | of the buffer but it can be safely assumed the there is room for at | |
2333 | least one complete character in the output buffer. | |
2334 | ||
2335 | Once the conversion is finished and the conversion is the last step the | |
976780fd | 2336 | @code{__outbuf} element must be modified to point after last last byte |
390955cb UD |
2337 | written into the buffer to signal how much output is available. If this |
2338 | conversion step is not the last one the element must not be modified. | |
976780fd | 2339 | The @code{__outbufend} element must not be modified. |
390955cb | 2340 | |
976780fd | 2341 | @item int __is_last |
390955cb UD |
2342 | This element is nonzero if this conversion step is the last one. This |
2343 | information is necessary for the recursion. See the description of the | |
2344 | conversion function internals below. This element must never be | |
2345 | modified. | |
2346 | ||
976780fd | 2347 | @item int __invocation_counter |
390955cb UD |
2348 | The conversion function can use this element to see how many calls of |
2349 | the conversion function already happened. Some character sets require | |
2350 | when generating output a certain prolog and by comparing this value with | |
2351 | zero one can find out whether it is the first call and therefore the | |
2352 | prolog should be emitted or not. This element must never be modified. | |
2353 | ||
976780fd | 2354 | @item int __internal_use |
390955cb UD |
2355 | This element is another one rarely used but needed in certain |
2356 | situations. It got assigned a nonzero value in case the conversion | |
2357 | functions are used to implement @code{mbsrtowcs} et.al. I.e., the | |
2358 | function is not used directly through the @code{iconv} interface. | |
2359 | ||
2360 | This sometimes makes a difference as it is expected that the | |
2361 | @code{iconv} functions are used to translate entire texts while the | |
2362 | @code{mbsrtowcs} functions are normally only used to convert single | |
2363 | strings and might be used multiple times to convert entire texts. | |
2364 | ||
2365 | But in this situation we would have problem complying with some rules of | |
2366 | the character set specification. Some character sets require a prolog | |
2367 | which must appear exactly once for an entire text. If a number of | |
2368 | @code{mbsrtowcs} calls are used to convert the text only the first call | |
2369 | must add the prolog. But since there is no communication between the | |
2370 | different calls of @code{mbsrtowcs} the conversion functions have no | |
2371 | possibility to find this out. The situation is different for sequences | |
2372 | of @code{iconv} calls since the handle allows to access the needed | |
2373 | information. | |
2374 | ||
976780fd UD |
2375 | This element is mostly used together with @code{__invocation_counter} in |
2376 | a way like this: | |
390955cb UD |
2377 | |
2378 | @smallexample | |
976780fd UD |
2379 | if (!data->__internal_use |
2380 | && data->__invocation_counter == 0) | |
390955cb UD |
2381 | /* @r{Emit prolog.} */ |
2382 | ... | |
2383 | @end smallexample | |
2384 | ||
2385 | This element must never be modified. | |
2386 | ||
976780fd UD |
2387 | @item mbstate_t *__statep |
2388 | The @code{__statep} element points to an object of type @code{mbstate_t} | |
c1b2d472 | 2389 | (@pxref{Keeping the state}). The conversion of an stateful character |
390955cb | 2390 | set must use the object pointed to by this element to store information |
976780fd UD |
2391 | about the conversion state. The @code{__statep} element itself must |
2392 | never be modified. | |
390955cb UD |
2393 | |
2394 | @item mbstate_t __state | |
2395 | This element @emph{never} must be used directly. It is only part of | |
2396 | this structure to have the needed space allocated. | |
2397 | @end table | |
2398 | @end deftp | |
2399 | ||
2400 | @subsubsection @code{iconv} module interfaces | |
2401 | ||
2402 | With the knowledge about the data structures we now can describe the | |
2403 | conversion functions itself. To understand the interface a bit of | |
2404 | knowledge about the functionality in the C library which loads the | |
2405 | objects with the conversions is necessary. | |
2406 | ||
2407 | It is often the case that one conversion is used more than once. I.e., | |
2408 | there are several @code{iconv_open} calls for the same set of character | |
2409 | sets during one program run. The @code{mbsrtowcs} et.al.@: functions in | |
2410 | the GNU C library also use the @code{iconv} functionality which | |
2411 | increases the number of uses of the same functions even more. | |
2412 | ||
2413 | For this reason the modules do not get loaded exclusively for one | |
2414 | conversion. Instead a module once loaded can be used by arbitrary many | |
2415 | @code{iconv} or @code{mbsrtowcs} calls at the same time. The splitting | |
2416 | of the information between conversion function specific information and | |
2417 | conversion data makes this possible. The last section showed the two | |
2418 | data structure used to do this. | |
2419 | ||
2420 | This is of course also reflected in the interface and semantic of the | |
2421 | functions the modules must provide. There are three functions which | |
2422 | must have the following names: | |
2423 | ||
2424 | @table @code | |
2425 | @item gconv_init | |
2426 | The @code{gconv_init} function initializes the conversion function | |
2427 | specific data structure. This very same object is shared by all | |
2428 | conversion which use this conversion and therefore no state information | |
2429 | about the conversion itself must be stored in here. If a module | |
2430 | implements more than one conversion the @code{gconv_init} function will be | |
2431 | called multiple times. | |
2432 | ||
2433 | @item gconv_end | |
2434 | The @code{gconv_end} function is responsible to free all resources | |
2435 | allocated by the @code{gconv_init} function. If there is nothing to do | |
2436 | this function can be missing. Special care must be taken if the module | |
2437 | implements more than one conversion and the @code{gconv_init} function | |
2438 | does not allocate the same resources for all conversions. | |
2439 | ||
2440 | @item gconv | |
2441 | This is the actual conversion function. It is called to convert one | |
2442 | block of text. It gets passed the conversion step information | |
2443 | initialized by @code{gconv_init} and the conversion data, specific to | |
2444 | this use of the conversion functions. | |
2445 | @end table | |
2446 | ||
2447 | There are three data types defined for the three module interface | |
2448 | function and these define the interface. | |
2449 | ||
2450 | @comment gconv.h | |
2451 | @comment GNU | |
976780fd | 2452 | @deftypevr {Data type} int (*__gconv_init_fct) (struct __gconv_step *) |
390955cb UD |
2453 | This specifies the interface of the initialization function of the |
2454 | module. It is called exactly once for each conversion the module | |
2455 | implements. | |
2456 | ||
976780fd | 2457 | As explained int the description of the @code{struct __gconv_step} data |
390955cb UD |
2458 | structure above the initialization function has to initialize parts of |
2459 | it. | |
2460 | ||
2461 | @table @code | |
976780fd UD |
2462 | @item __min_needed_from |
2463 | @itemx __max_needed_from | |
2464 | @itemx __min_needed_to | |
2465 | @itemx __max_needed_to | |
390955cb UD |
2466 | These elements must be initialized to the exact numbers of the minimum |
2467 | and maximum number of bytes used by one character in the source and | |
2468 | destination character set respectively. If the characters all have the | |
2469 | same size the minimum and maximum values are the same. | |
2470 | ||
976780fd | 2471 | @item __stateful |
390955cb UD |
2472 | This element must be initialized to an nonzero value if the source |
2473 | character set is stateful. Otherwise it must be zero. | |
2474 | @end table | |
2475 | ||
2476 | If the initialization function needs to communication some information | |
976780fd UD |
2477 | to the conversion function this can happen using the @code{__data} |
2478 | element of the @code{__gconv_step} structure. But since this data is | |
2479 | shared by all the conversion is must not be modified by the conversion | |
2480 | function. How this can be used is shown in the example below. | |
390955cb UD |
2481 | |
2482 | @smallexample | |
2483 | #define MIN_NEEDED_FROM 1 | |
2484 | #define MAX_NEEDED_FROM 4 | |
2485 | #define MIN_NEEDED_TO 4 | |
2486 | #define MAX_NEEDED_TO 4 | |
2487 | ||
2488 | int | |
976780fd | 2489 | gconv_init (struct __gconv_step *step) |
390955cb UD |
2490 | @{ |
2491 | /* @r{Determine which direction.} */ | |
2492 | struct iso2022jp_data *new_data; | |
2493 | enum direction dir = illegal_dir; | |
2494 | enum variant var = illegal_var; | |
2495 | int result; | |
2496 | ||
976780fd | 2497 | if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0) |
390955cb UD |
2498 | @{ |
2499 | dir = from_iso2022jp; | |
2500 | var = iso2022jp; | |
2501 | @} | |
976780fd | 2502 | else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0) |
390955cb UD |
2503 | @{ |
2504 | dir = to_iso2022jp; | |
2505 | var = iso2022jp; | |
2506 | @} | |
976780fd | 2507 | else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0) |
390955cb UD |
2508 | @{ |
2509 | dir = from_iso2022jp; | |
2510 | var = iso2022jp2; | |
2511 | @} | |
976780fd | 2512 | else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0) |
390955cb UD |
2513 | @{ |
2514 | dir = to_iso2022jp; | |
2515 | var = iso2022jp2; | |
2516 | @} | |
2517 | ||
976780fd | 2518 | result = __GCONV_NOCONV; |
390955cb UD |
2519 | if (dir != illegal_dir) |
2520 | @{ | |
2521 | new_data = (struct iso2022jp_data *) | |
2522 | malloc (sizeof (struct iso2022jp_data)); | |
2523 | ||
976780fd | 2524 | result = __GCONV_NOMEM; |
390955cb UD |
2525 | if (new_data != NULL) |
2526 | @{ | |
2527 | new_data->dir = dir; | |
2528 | new_data->var = var; | |
976780fd | 2529 | step->__data = new_data; |
390955cb UD |
2530 | |
2531 | if (dir == from_iso2022jp) | |
2532 | @{ | |
976780fd UD |
2533 | step->__min_needed_from = MIN_NEEDED_FROM; |
2534 | step->__max_needed_from = MAX_NEEDED_FROM; | |
2535 | step->__min_needed_to = MIN_NEEDED_TO; | |
2536 | step->__max_needed_to = MAX_NEEDED_TO; | |
390955cb UD |
2537 | @} |
2538 | else | |
2539 | @{ | |
976780fd UD |
2540 | step->__min_needed_from = MIN_NEEDED_TO; |
2541 | step->__max_needed_from = MAX_NEEDED_TO; | |
2542 | step->__min_needed_to = MIN_NEEDED_FROM; | |
2543 | step->__max_needed_to = MAX_NEEDED_FROM + 2; | |
390955cb UD |
2544 | @} |
2545 | ||
2546 | /* @r{Yes, this is a stateful encoding.} */ | |
976780fd | 2547 | step->__stateful = 1; |
390955cb | 2548 | |
976780fd | 2549 | result = __GCONV_OK; |
390955cb UD |
2550 | @} |
2551 | @} | |
2552 | ||
2553 | return result; | |
2554 | @} | |
2555 | @end smallexample | |
2556 | ||
2557 | The function first checks which conversion is wanted. The module from | |
2558 | which this function is taken implements four different conversion and | |
2559 | which one is selected can be determined by comparing the names. The | |
2560 | comparison should always be done without paying attention to the case. | |
2561 | ||
2562 | Then a data structure is allocated which contains the necessary | |
2563 | information about which conversion is selected. The data structure | |
2564 | @code{struct iso2022jp_data} is locally defined since outside the module | |
2565 | this data is not used at all. Please note that if all four conversions | |
2566 | this modules supports are requested there are four data blocks. | |
2567 | ||
976780fd UD |
2568 | One interesting thing is the initialization of the @code{__min_} and |
2569 | @code{__max_} elements of the step data object. A single ISO-2022-JP | |
390955cb UD |
2570 | character can consist of one to four bytes. Therefore the |
2571 | @code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined | |
2572 | this way. The output is always the @code{INTERNAL} character set (aka | |
2573 | UCS4) and therefore each character consists of exactly four bytes. For | |
2574 | the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into | |
2575 | account that escape sequences might be necessary to switch the character | |
976780fd | 2576 | sets. Therefore the @code{__max_needed_to} element for this direction |
390955cb UD |
2577 | gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the |
2578 | two bytes needed for the escape sequences to single the switching. The | |
2579 | asymmetry in the maximum values for the two directions can be explained | |
2580 | easily: when reading ISO-2022-JP text escape sequences can be handled | |
2581 | alone. I.e., it is not necessary to process a real character since the | |
2582 | effect of the escape sequence can be recorded in the state information. | |
2583 | The situation is different for the other direction. Since it is in | |
2584 | general not known which character comes next one cannot emit escape | |
2585 | sequences to change the state in advance. This means the escape | |
2586 | sequences which have to be emitted together with the next character. | |
2587 | Therefore one needs more room then only for the character itself. | |
2588 | ||
2589 | The possible return values of the initialization function are: | |
2590 | ||
2591 | @table @code | |
976780fd | 2592 | @item __GCONV_OK |
390955cb | 2593 | The initialization succeeded |
976780fd | 2594 | @item __GCONV_NOCONV |
390955cb UD |
2595 | The requested conversion is not supported in the module. This can |
2596 | happen if the @file{gconv-modules} file has errors. | |
976780fd | 2597 | @item __GCONV_NOMEM |
390955cb UD |
2598 | Memory required to store additional information could not be allocated. |
2599 | @end table | |
2600 | @end deftypevr | |
2601 | ||
2602 | The functions called before the module is unloaded is significantly | |
2603 | easier. It often has nothing at all to do in which case it can be left | |
2604 | out completely. | |
2605 | ||
2606 | @comment gconv.h | |
2607 | @comment GNU | |
976780fd | 2608 | @deftypevr {Data type} void (*__gconv_end_fct) (struct gconv_step *) |
390955cb | 2609 | The task of this function is it to free all resources allocated in the |
976780fd UD |
2610 | initialization function. Therefore only the @code{__data} element of |
2611 | the object pointed to by the argument is of interest. Continuing the | |
390955cb UD |
2612 | example from the initialization function, the finalization function |
2613 | looks like this: | |
2614 | ||
2615 | @smallexample | |
2616 | void | |
976780fd | 2617 | gconv_end (struct __gconv_step *data) |
390955cb | 2618 | @{ |
976780fd | 2619 | free (data->__data); |
390955cb UD |
2620 | @} |
2621 | @end smallexample | |
2622 | @end deftypevr | |
2623 | ||
d731df03 UD |
2624 | The most important function is the conversion function itself. It can |
2625 | get quite complicated for complex character sets. But since this is not | |
2626 | of interest here we will only describe a possible skeleton for the | |
2627 | conversion function. | |
390955cb UD |
2628 | |
2629 | @comment gconv.h | |
2630 | @comment GNU | |
976780fd | 2631 | @deftypevr {Data type} int (*__gconv_fct) (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int) |
390955cb UD |
2632 | The conversion function can be called for two basic reason: to convert |
2633 | text or to reset the state. From the description of the @code{iconv} | |
2634 | function it can be seen why the flushing mode is necessary. What mode | |
2635 | is selected is determined by the sixth argument, an integer. If it is | |
2636 | nonzero it means that flushing is selected. | |
2637 | ||
2638 | Common to both mode is where the output buffer can be found. The | |
2639 | information about this buffer is stored in the conversion step data. A | |
2640 | pointer to this is passed as the second argument to this function. The | |
976780fd | 2641 | description of the @code{struct __gconv_step_data} structure has more |
390955cb UD |
2642 | information on this. |
2643 | ||
2644 | @cindex stateful | |
2645 | What has to be done for flushing depends on the source character set. | |
2646 | If it is not stateful nothing has to be done. Otherwise the function | |
2647 | has to emit a byte sequence to bring the state object in the initial | |
2648 | state. Once this all happened the other conversion modules in the chain | |
2649 | of conversions have to get the same chance. Whether another step | |
976780fd | 2650 | follows can be determined from the @code{__is_last} element of the step |
390955cb UD |
2651 | data structure to which the first parameter points. |
2652 | ||
2653 | The more interesting mode is when actually text has to be converted. | |
2654 | The first step in this case is to convert as much text as possible from | |
2655 | the input buffer and store the result in the output buffer. The start | |
2656 | of the input buffer is determined by the third argument which is a | |
2657 | pointer to a pointer variable referencing the beginning of the buffer. | |
2658 | The fourth argument is a pointer to the byte right after the last byte | |
2659 | in the buffer. | |
2660 | ||
2661 | The conversion has to be performed according to the current state if the | |
2662 | character set is stateful. The state is stored in an object pointed to | |
976780fd | 2663 | by the @code{__statep} element of the step data (second argument). Once |
390955cb UD |
2664 | either the input buffer is empty or the output buffer is full the |
2665 | conversion stops. At this point the pointer variable referenced by the | |
2666 | third parameter must point to the byte following the last processed | |
2667 | byte. I.e., if all of the input is consumed this pointer and the fourth | |
2668 | parameter have the same value. | |
2669 | ||
2670 | What now happens depends on whether this step is the last one or not. | |
2671 | If it is the last step the only thing which has to be done is to update | |
976780fd | 2672 | the @code{__outbuf} element of the step data structure to point after the |
390955cb UD |
2673 | last written byte. This gives the caller the information on how much |
2674 | text is available in the output buffer. Beside this the variable | |
2675 | pointed to by the fifth parameter, which is of type @code{size_t}, must | |
2676 | be incremented by the number of characters (@emph{not bytes}) which were | |
2677 | written in the output buffer. Then the function can return. | |
2678 | ||
2679 | In case the step is not the last one the later conversion functions have | |
2680 | to get a chance to do their work. Therefore the appropriate conversion | |
2681 | function has to be called. The information about the functions is | |
2682 | stored in the conversion data structures, passed as the first parameter. | |
2683 | This information and the step data are stored in arrays so the next | |
2684 | element in both cases can be found by simple pointer arithmetic: | |
2685 | ||
2686 | @smallexample | |
2687 | int | |
976780fd | 2688 | gconv (struct __gconv_step *step, struct __gconv_step_data *data, |
390955cb UD |
2689 | const char **inbuf, const char *inbufend, size_t *written, |
2690 | int do_flush) | |
2691 | @{ | |
976780fd UD |
2692 | struct __gconv_step *next_step = step + 1; |
2693 | struct __gconv_step_data *next_data = data + 1; | |
390955cb UD |
2694 | ... |
2695 | @end smallexample | |
2696 | ||
2697 | The @code{next_step} pointer references the next step information and | |
2698 | @code{next_data} the next data record. The call of the next function | |
2699 | therefore will look similar to this: | |
2700 | ||
2701 | @smallexample | |
976780fd UD |
2702 | next_step->__fct (next_step, next_data, &outerr, outbuf, |
2703 | written, 0) | |
390955cb UD |
2704 | @end smallexample |
2705 | ||
2706 | But this is not yet all. Once the function call returns the conversion | |
2707 | function might have some more to do. If the return value of the | |
976780fd | 2708 | function is @code{__GCONV_EMPTY_INPUT} this means there is more room in |
390955cb UD |
2709 | the output buffer. Unless the input buffer is empty the conversion |
2710 | functions start all over again and processes the rest of the input | |
976780fd | 2711 | buffer. If the return value is not @code{__GCONV_EMPTY_INPUT} something |
390955cb UD |
2712 | went wrong and we have to recover from this. |
2713 | ||
2714 | A requirement for the conversion function is that the input buffer | |
2715 | pointer (the third argument) always points to the last character which | |
2716 | was put in the converted form in the output buffer. This is trivial | |
2717 | true after the conversion performed in the current step. But if the | |
2718 | conversion functions deeper down the stream stop prematurely not all | |
2719 | characters from the output buffer are consumed and therefore the input | |
2720 | buffer pointers must be backed of to the right position. | |
2721 | ||
2722 | This is easy to do if the input and output character sets have a fixed | |
2723 | width for all characters. In this situation we can compute how many | |
2724 | characters are left in the output buffer and therefore can correct the | |
2725 | input buffer pointer appropriate with a similar computation. Things are | |
2726 | getting tricky if either character set has character represented with | |
2727 | variable length byte sequences and it gets even more complicated if the | |
2728 | conversion has to take care of the state. In these cases the conversion | |
2729 | has to be performed once again, from the known state before the initial | |
2730 | conversion. I.e., if necessary the state of the conversion has to be | |
2731 | reset and the conversion loop has to be executed again. The difference | |
2732 | now is that it is known how much input must be created and the | |
2733 | conversion can stop before converting the first unused character. Once | |
2734 | this is done the input buffer pointers must be updated again and the | |
2735 | function can return. | |
2736 | ||
2737 | One final thing should be mentioned. If it is necessary for the | |
2738 | conversion to know whether it is the first invocation (in case a prolog | |
2739 | has to be emitted) the conversion function should just before returning | |
976780fd | 2740 | to the caller increment the @code{__invocation_counter} element of the |
390955cb | 2741 | step data structure. See the description of the @code{struct |
976780fd UD |
2742 | __gconv_step_data} structure above for more information on how this can |
2743 | be used. | |
390955cb UD |
2744 | |
2745 | The return value must be one of the following values: | |
2746 | ||
2747 | @table @code | |
976780fd | 2748 | @item __GCONV_EMPTY_INPUT |
390955cb | 2749 | All input was consumed and there is room left in the output buffer. |
976780fd | 2750 | @item __GCONV_OUTPUT_FULL |
390955cb UD |
2751 | No more room in the output buffer. In case this is not the last step |
2752 | this value is propagated down from the call of the next conversion | |
2753 | function in the chain. | |
976780fd | 2754 | @item __GCONV_INCOMPLETE_INPUT |
390955cb UD |
2755 | The input buffer is not entirely empty since it contains an incomplete |
2756 | character sequence. | |
2757 | @end table | |
2758 | ||
2759 | The following example provides a framework for a conversion function. | |
2760 | In case a new conversion has to be written the holes in this | |
2761 | implementation have to be filled and that is it. | |
2762 | ||
2763 | @smallexample | |
2764 | int | |
976780fd | 2765 | gconv (struct __gconv_step *step, struct __gconv_step_data *data, |
390955cb UD |
2766 | const char **inbuf, const char *inbufend, size_t *written, |
2767 | int do_flush) | |
2768 | @{ | |
976780fd UD |
2769 | struct __gconv_step *next_step = step + 1; |
2770 | struct __gconv_step_data *next_data = data + 1; | |
2771 | gconv_fct fct = next_step->__fct; | |
390955cb UD |
2772 | int status; |
2773 | ||
2774 | /* @r{If the function is called with no input this means we have} | |
2775 | @r{to reset to the initial state. The possibly partly} | |
2776 | @r{converted input is dropped.} */ | |
2777 | if (do_flush) | |
2778 | @{ | |
976780fd | 2779 | status = __GCONV_OK; |
390955cb UD |
2780 | |
2781 | /* @r{Possible emit a byte sequence which put the state object} | |
2782 | @r{into the initial state.} */ | |
2783 | ||
2784 | /* @r{Call the steps down the chain if there are any but only} | |
2785 | @r{if we successfully emitted the escape sequence.} */ | |
976780fd | 2786 | if (status == __GCONV_OK && ! data->__is_last) |
390955cb UD |
2787 | status = fct (next_step, next_data, NULL, NULL, |
2788 | written, 1); | |
2789 | @} | |
2790 | else | |
2791 | @{ | |
2792 | /* @r{We preserve the initial values of the pointer variables.} */ | |
2793 | const char *inptr = *inbuf; | |
976780fd UD |
2794 | char *outbuf = data->__outbuf; |
2795 | char *outend = data->__outbufend; | |
390955cb UD |
2796 | char *outptr; |
2797 | ||
2798 | /* @r{This variable is used to count the number of characters} | |
2799 | @r{we actually converted.} */ | |
2800 | size_t converted = 0; | |
2801 | ||
2802 | do | |
2803 | @{ | |
2804 | /* @r{Remember the start value for this round.} */ | |
2805 | inptr = *inbuf; | |
2806 | /* @r{The outbuf buffer is empty.} */ | |
2807 | outptr = outbuf; | |
2808 | ||
2809 | /* @r{For stateful encodings the state must be safe here.} */ | |
2810 | ||
2811 | /* @r{Run the conversion loop. @code{status} is set} | |
2812 | @r{appropriately afterwards.} */ | |
2813 | ||
2814 | /* @r{If this is the last step leave the loop, there is} | |
2815 | @r{nothing we can do.} */ | |
976780fd | 2816 | if (data->__is_last) |
390955cb UD |
2817 | @{ |
2818 | /* @r{Store information about how many bytes are} | |
2819 | @r{available.} */ | |
976780fd | 2820 | data->__outbuf = outbuf; |
390955cb UD |
2821 | |
2822 | /* @r{Remember how many characters we converted.} */ | |
2823 | *written += converted; | |
2824 | ||
2825 | break; | |
2826 | @} | |
2827 | ||
2828 | /* @r{Write out all output which was produced.} */ | |
2829 | if (outbuf > outptr) | |
2830 | @{ | |
976780fd | 2831 | const char *outerr = data->__outbuf; |
390955cb UD |
2832 | int result; |
2833 | ||
2834 | result = fct (next_step, next_data, &outerr, | |
2835 | outbuf, written, 0); | |
2836 | ||
976780fd | 2837 | if (result != __GCONV_EMPTY_INPUT) |
390955cb UD |
2838 | @{ |
2839 | if (outerr != outbuf) | |
2840 | @{ | |
2841 | /* @r{Reset the input buffer pointer. We} | |
2842 | @r{document here the complex case.} */ | |
2843 | size_t nstatus; | |
2844 | ||
2845 | /* @r{Reload the pointers.} */ | |
2846 | *inbuf = inptr; | |
2847 | outbuf = outptr; | |
2848 | ||
2849 | /* @r{Possibly reset the state.} */ | |
2850 | ||
2851 | /* @r{Redo the conversion, but this time} | |
2852 | @r{the end of the output buffer is at} | |
2853 | @r{@code{outerr}.} */ | |
2854 | @} | |
2855 | ||
2856 | /* @r{Change the status.} */ | |
2857 | status = result; | |
2858 | @} | |
2859 | else | |
2860 | /* @r{All the output is consumed, we can make} | |
2861 | @r{ another run if everything was ok.} */ | |
976780fd UD |
2862 | if (status == __GCONV_FULL_OUTPUT) |
2863 | status = __GCONV_OK; | |
390955cb UD |
2864 | @} |
2865 | @} | |
976780fd | 2866 | while (status == __GCONV_OK); |
390955cb UD |
2867 | |
2868 | /* @r{We finished one use of this step.} */ | |
976780fd | 2869 | ++data->__invocation_counter; |
390955cb UD |
2870 | @} |
2871 | ||
2872 | return status; | |
2873 | @} | |
2874 | @end smallexample | |
2875 | @end deftypevr | |
2876 | ||
2877 | This information should be sufficient to write new modules. Anybody | |
2878 | doing so should also take a look at the available source code in the GNU | |
2879 | C library sources. It contains many examples of working and optimized | |
2880 | modules. |