Re: Unicode update of width and other character properties

[Thomas Wolff <towo at towo dot net>]

Am 08.08.2017 um 10:30 schrieb Corinna Vinschen:
> On Aug  7 21:18, Thomas Wolff wrote:
> > Am 07.08.2017 um 12:30 schrieb Corinna Vinschen:
> > > On Aug  6 07:36, Thomas Wolff wrote:
> > > > Hi,
> > > > this is a proposal to update wcwidth and the character properties functions
> > > > isw*/towupper/towlower to Unicode 10.0, as discussed in the mail thread
> > > > https://cygwin.com/ml/cygwin/2017-07/msg00366.html,
> > > > as well as to simplify automatic generation of respective tables for an
> > > > easier update step.
> > > > Table size is moderate (using ranges for character properties) but there is
> > > > still an option to reduce the two big tables in size.
> > > As per the aforementioned discussion the table sizes are at least
> > > twice as big, so this should be done with all due caution towards
> > > the goals of smaller targets.
> > If I'm going to implement the packed versions, they will be even smaller
> > than the current tables.
> >
> > ...
> > how to produce the desired patch format/series.
> Just as with any other git-based project:
>
>    $ git co -b my-stuff
>    [hack, hack, hack]
>    $ git commit [in useful chunks]
>    $ git format-patch -X (X == number of commits)
>
> > And then the patch would be included here by email?
> Yes:
>
> $ git send-email --to="newlib@sourceware.org"
I'm attaching my patches here for assessment.
I have revised table handling further, using gcc bit struct packing. The 
two big tables have a total size of 14340 bytes now, for Unicode 10.0.
I have fixed locale handling in the isw* and tow* functions, but I've 
not yet changed JP conversion. Unfortunately, the routines from 
newlib/iconvdata are not as straight-forward to be employed as I 
thought, because the work on multi-byte representations.
Also the mapping of ctype charsets (JIS, SJIS, EUC-JP) to the subsets 
handled in iconvdata (JIS-201/208/212) is a little bit obscure.
Likewise obscure is the relation between newlib/iconvdata and 
newlib/libc/iconv.
To be on the safe side, I’m leaving the actual jp2uc conversion 
untouched for now, and I’ve just added a dummy back-conversion uc2jp 
with a #warning. If the #warning is ignored or removed, the non-Cygwin 
build should work as before, fixing just locale handling.

I'm attaching the wcwidth part here, all patches are available at 
http://towo.net/cygwin/Unicode_and_locale_tweaks.zip (don't fit in the 
mailbox size limit).
Thomas

Attachment: 0001-creation-of-width-data-supporting-Unicode-updates.patch
Description: Text document

Attachment: 0002-generated-width-data-included-in-repository-because-.patch
Description: Text document

Attachment: 0003-use-generated-width-data.patch
Description: Text document