From 2bc0ae17c582c6bdec03586166fbb25a043efa3a Mon Sep 17 00:00:00 2001
From: Sam Lantinga <slouken@libsdl.org>
Date: Fri, 11 Dec 2009 08:00:57 +0000
Subject: [PATCH] Fixed bug #896

 John Popplewell      2009-12-08 23:05:50 PST

Originally reported by AKFoerster on the mailing list.

Error decoding UTF8 Russian text to UTF-16LE on Windows, but specifically on
platforms without iconv support (the default on Windows).

Valid UTF8 characters are flagged as being overlong and then substituted by the
UNKNOWN_UNICODE character.

After studying the testiconv.c example program, reading the RFCs and putting
some printf statements in SDL_iconv.c the problem is in a test for 'Maximum
overlong sequences', specifically 4.2.1, which is carried out by the following
code:

      } else if ( p[0] >= 0xC0 ) {
        if ( (p[0] & 0xE0) != 0xC0 ) {
          /* Skip illegal sequences
            return SDL_ICONV_EILSEQ;
          */
          ch = UNKNOWN_UNICODE;
        } else {
          if ( (p[0] & 0xCE) == 0xC0 ) {    <<<<<<<< here
            overlong = SDL_TRUE;
          }
          ch = (Uint32)(p[0] & 0x1F);
          left = 1;
        }
      } else {

Here is the 2-byte encoding of a character in range 00000080 - 000007FF
    110xxxxx 10xxxxxx

The line in question is supposed to be checking for an overlong sequence which
would be less than
    11000001 10111111

which should be represented as a single byte.

BUT, the mask value (0xCE) is wrong, it isn't checking the top-most bit:
    11000001     value
    11001110     mask (incorrect)
       ^
and should be (0xDE):
    11000001     value
    11011110     mask (correct)

making the above code:

      } else if ( p[0] >= 0xC0 ) {
        if ( (p[0] & 0xE0) != 0xC0 ) {
          /* Skip illegal sequences
            return SDL_ICONV_EILSEQ;
          */
          ch = UNKNOWN_UNICODE;
        } else {
          if ( (p[0] & 0xDE) == 0xC0 ) {    <<<<<<<< here
            overlong = SDL_TRUE;
          }
          ch = (Uint32)(p[0] & 0x1F);
          left = 1;
        }
      } else {

I can supply a test program and/or a patch if required,

best regards,
John Popplewell
---
 src/stdlib/SDL_iconv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/stdlib/SDL_iconv.c b/src/stdlib/SDL_iconv.c
index de86c20d2..9e6889141 100644
--- a/src/stdlib/SDL_iconv.c
+++ b/src/stdlib/SDL_iconv.c
@@ -383,7 +383,7 @@ size_t SDL_iconv(SDL_iconv_t cd,
 						*/
 						ch = UNKNOWN_UNICODE;
 					} else {
-						if ( (p[0] & 0xCE) == 0xC0 ) {
+						if ( (p[0] & 0xDE) == 0xC0 ) {
 							overlong = SDL_TRUE;
 						}
 						ch = (Uint32)(p[0] & 0x1F);