Bug 10165

Summary: windres crashes when processing resource files with UTF-8 (cp 65001) encoding.
Product: binutils Reporter: Tom Bramer <tjb>
Component: binutilsAssignee: unassigned
Severity: normal CC: bug-binutils
Priority: P2    
Version: 2.19   
Target Milestone: ---   
Host: i686-pc-mingw32 Target: i686-pc-mingw32
Build: i686-pc-mingw32 Last reconfirmed:
Attachments: Proposed patch to windres
Test case input.

Description Tom Bramer 2009-05-17 22:48:42 UTC
OS: Windows XP SP2

windres, version 2.18 and up, when processing an rc file that contains at least
one quoted string, will crash with an access violation (on dereference of NULL
pointer) if the code page is explicitly set to 65001 (UTF-8).  

Here is an example input that causes the issue:

======================== BEGIN =========================
#define IDS_STRING1 1
#pragma code_page(65001)

        IDS_STRING1 "Any string will do..."
========================= END ==========================

The cause:

When calling the MultiByteToWideChar function with a code page of 65000, 65001,
or others (see http://msdn.microsoft.com/en-us/library/dd319072(VS.85).aspx),
the dwFlags parameter must be 0, unless when using Windows XP and later, in
which it may be 0 or MB_ERR_INVALID_CHARS.  If the value given for dwFlags is in
fact set to a value other than the aforementioned valid cases, the function will
set the last error to ERROR_INVALID_FLAGS.  The wind_MultiByteToWideChar
function always passes MB_PRECOMPOSED as the dwFlags parameter to
MultiByteToWideChar, causing the function to fail and not convert the UTF-7/8
string to UTF-16.  The wind_MultiByteToWideChar function assumes that the
MultiByteToWideChar function is successful, when in reality, the wide character
buffer still contains uninitialized data.

What worked for me:

======================== BEGIN =========================
--- binutils-2.19.1/binutils/winduni.c	2007-07-05 12:54:44 -0400
+++ binutils-2.19.1-new/binutils/winduni.c	2009-05-17 17:52:33 -0400
@@ -661,7 +661,15 @@ wind_MultiByteToWideChar (rc_uint_type c
   rc_uint_type ret = 0;
 #if defined (_WIN32) || defined (__CYGWIN__)
-  ret = (rc_uint_type) MultiByteToWideChar (cp, MB_PRECOMPOSED,
+  rc_uint_type conv_flags = MB_PRECOMPOSED;
+  /* MB_PRECOMPOSED is not allowed for UTF-7 or UTF-8.  
+     MultiByteToWideChar will set the last error to ERROR_INVALID_FLAGS
+     if we do anyways. */
+  if (cp == CP_UTF8 || cp == CP_UTF7)
+      conv_flags = 0;
+  ret = (rc_uint_type) MultiByteToWideChar (cp, conv_flags,
 					    mb, -1, u, u_len);
   /* Convert to bytes. */
   ret *= sizeof (unichar);
======================== END =========================
Comment 1 Tom Bramer 2009-05-17 22:49:48 UTC
Created attachment 3940 [details]
Proposed patch to windres
Comment 2 Tom Bramer 2009-05-17 22:51:13 UTC
Created attachment 3941 [details]
Test case input.
Comment 3 cvs-commit@gcc.gnu.org 2009-06-09 15:14:45 UTC
Subject: Bug 10165

CVSROOT:	/cvs/src
Module name:	src
Changes by:	nickc@sourceware.org	2009-06-09 15:14:23

Modified files:
	binutils       : ChangeLog winduni.c 

Log message:
	PR 10165
	* winduni.c (wind_MultiByteToWideChar): Do not pass MB_PRECOMPOSED
	to MultiByteToWideChar when using the CP_UTF8 or CO_UTF7 types.


Comment 4 Nick Clifton 2009-06-09 15:16:22 UTC
Hi Tom,

  Thanks for reporting this bug and supplying a patch to fix it.

  I have checked in your patch along with the changelog entry below.


2009-06-09  Tom Bramer  <tjb@postpro.net>

	PR 10165
	* winduni.c (wind_MultiByteToWideChar): Do not pass MB_PRECOMPOSED
	to MultiByteToWideChar when using the CP_UTF8 or CO_UTF7 types.
Comment 5 Jackie Rosen 2014-02-16 17:44:57 UTC Comment hidden (spam)