[Bug libc/19432] New: iconv rejects redundant escape sequences in IBM903, IBM905, IBM907, and IBM909

[msebor at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19432

            Bug ID: 19432
           Summary: iconv rejects redundant escape sequences in IBM903,
                    IBM905, IBM907, and IBM909
           Product: glibc
           Version: 2.23
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
          Assignee: unassigned at sourceware dot org
          Reporter: msebor at redhat dot com
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

In the problem report corresponding to bug 17197 submitted against the RHEL
GLIBC, even though the focus of the customer's complaint is on GLIBC emitting
redundant shift sequences, the provided test case implies the customer is
actually having a problem interpreting already encoded data (i.e., with iconv()
treating the duplicate/redundant escape sequences as errors).  If iconv()
accepted the duplicate sequences the customer wouldn't be affected.

The solution implemented in response to bug 17197, changing the converter to
avoid emitting the redundant escape sequences, while a worthwhile improvement,
is not sufficient to resolve the complaint because there may already be
IBM930-encoded data containing redundant shift sequences that need to be
converted.  The current fix will not help users avoid the errors when
converting such data.

Looking for guidance, I checked POSIX to see if it provides support for the
current behavior (rejecting redundant escape sequences) in POSIX.  I don't see
anything in POSIX to sanction rejecting such input.  Even if POSIX did allow
for such behavior, since other GLIBC converters accept redundant escape
sequences, it seems that the IBM930 converter should accept them as well for
consistency.  Doing otherwise makes the iconv interfaces needlessly error-prone
to use.

The test case below is an example of input with redundant escape sequences in
two state-dependent encodings to show the inconsistency mentioned above.

#include <iconv.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

static int
test_iconv (const char *from_set, char *input, size_t inbytesleft)
{
  const char to_set[] = "UTF-8";
  iconv_t cd = iconv_open (to_set, from_set);
  if (cd == (iconv_t) -1)
    {
      printf ("iconv_open(\"%s\", \"%s\"): %s\n",
              from_set, to_set, strerror (errno));
      return 1;
    }

  char output [32];
  size_t outbytesleft = sizeof output;

  char *inbuf = input;
  char *outbuf = output;

  printf ("iconv(cd, %p, %zu, %p, %zu)\n",
          inbuf, inbytesleft, outbuf, outbytesleft);

  errno = 0;
  size_t ret = iconv (cd, &inbuf, &inbytesleft, &outbuf, &outbytesleft);
  printf ("  ==> %td: %s\n"
          "  inbuf%+td, inbytesleft=%zu, outbuf%+td, outbytesleft=%zu\n",
          ret, strerror (errno),
          inbuf - input, inbytesleft, outbuf - output, outbytesleft);

  return ret == (size_t)-1;
}

int
main (void)
{
  int ret = 0;

  // The third escape sequence is redundant but is accepted
  // in Shift-JIS:
  //            <ESC-JIS><ESC-ASCII>G<ESC-ASCII>
  char sjis[] = "\x1b\x28\x42G\x1b\x28\x42";
  ret += test_iconv ("SHIFT-JIS", sjis, sizeof sjis - 1);

  // The analogous redundant escape sequence is rejected
  // in IBM930:
  //              <SO><SI>G<SI>
  char ibm930[] = "\x0e\x0f\xc7\x0f";
  ret += test_iconv ("IBM930", ibm930, sizeof ibm930 - 1);

  return ret;
}

-- 
You are receiving this mail because:
You are on the CC list for the bug.