Re: glibc git commit hooks update.

[Carlos O'Donell <codonell at redhat dot com>]

On 5/16/19 3:35 PM, Joseph Myers wrote:
> On Wed, 17 Apr 2019, Carlos O'Donell wrote:
> 
>>> One problem with the old hooks was that they did not generate a
>>> Content-Type header, which was unhelpful when the mails also weren't pure
>>> ASCII.  I see the new ones are generating 'Content-Type: text/plain;
>>> charset="us-ascii"'.  Will they also be smart about specifying an
>>> appropriate character set (so UTF-8 if the commit message / author name /
>>> diff contents are valid UTF-8 but not ASCII, for example)?
>>
>> All of this is handled by python's email package and some handling
>> on the hooks part. If everything is ASCII then we don't do any special
>> encoding and just send ASCII. Otherwise we choose UTF-8 first and if
>> that fails a decode test then we fallback to ISO-8859-1.
> 
> This does not seem to be working as intended.  See e.g. 
> https://sourceware.org/ml/glibc-cvs/2019-q2/msg00147.html (UTF-8 bytes 
> marked as us-ascii, so ’ appears as â??, for example).

The outbound message is marked:
Content-Type: text/plain; charset="us-ascii"

So it makes sense that the UTF-8 characters are not correctly
parsed. The API used here is the MIMEText one and it defaults
always to "us-ascii", so this needs changing.

All files in glibc sources should be UTF-8 unless they are testing
input files that have specific encodings, and even then we cannot
fix all cases e.g. tst-langinfo.sh and tst-fnmatch.input have
multiple mixed encodings (and are almost impossible to edit with
an editor).

Therefore I think all projects should just default to UTF-8
encoded output for emails going to the list.

Thoughts on defaulting to UTF-8?

-- 
Cheers,
Carlos.