UTF8 Conversion problem

Here are some sites you can bookmark:
http://unicodelookup.com/
http://www.fileformat.info/info/unicode/char/search.htm
http://demo.icu-project.org/icu-bin/ubrowse

For codepages:
http://msdn.microsoft.com/en-US/goglobal/bb964653.aspx
http://www.tachyonsoft.com/cpindex.htm

gg

With your 'szText1', i encoded in unicode and wrote to file and i can read the text successfully.

Now i test with thai text and still get the same bad output.

Code:

---

WCHAR wText[1000];
char* szTest1 = "\xE0\xB8\xAA\xE0\xB8\xB8\xE0\xB8\x94"; // "สุด" (3 characters)
MultiByteToWideChar(CP_UTF8, 0, szTest1, -1, wText, 1000);
CLogger::WriteToFileW("Test1.txt", wText);

---

Original Text: "สุด" (3 characters which forms 2 symbols)
Output in Test1.txt: "*8"
Test1.txt file size: 6 Bytes

Please note that WriteToFileW write the string without the null bytes terminator... if youre wondering about the file size.

I still dont understand why it doesnt work.

It just doesn't know how to render the characters. If you open the file in a hex editor, you will see "2A0E 380E 140E" for those 3 characters - which is the little endian encoding for those 3 characters.
http://www.fileformat.info/info/unic...0e2a/index.htm
http://www.fileformat.info/info/unic...0e38/index.htm
http://www.fileformat.info/info/unic...0e14/index.htm

gg

Because you saved your file in Unicode format already.
Mine is in UTF8.
UTF8 format takes from 1 to 3 bytes and in my case, all of my characters takes 3 bytes.
After the conversion they should all take 2 byte.. like in your case.

Which get me back to my start point.
How do i convert this UTF-8(1-3 bytes based) string into a Unicode(2 bytes based) string?

If you look in your web links that you gave me, you will see that the first character in UTF8 is 0xE0 0xB8 0xAA, just like i hardcoded in my code, and the UTF16 is 0E2A, just like you read from your file.
How do i convert this UTF8 char into a UTF16?

Code:

---

#include <windows.h>
#include <stdio.h>

int main()
{
    WCHAR wText[1000];
    const char* szTest1 = "\xE0\xB8\xAA\xE0\xB8\xB8\xE0\xB8\x94";
    int res = MultiByteToWideChar(CP_UTF8, 0, szTest1, -1, wText, 1000);
    if (!res)
    {
        printf("MultiByteToWideChar failed, %u\n", GetLastError());
        return 1;
    }

    FILE *f = fopen("WriteToFileW.txt", "w");
    if (!f)
    {
        printf("Failed to open file\n");
        return 1;
    }

    const char* BOM = "\xFF\xFE"; // UTF16-LE
    fwrite(BOM, 1, 2, f);
   
    fwrite(wText, 1, wcslen(wText) * 2, f);
    fclose(f);
   
    return 0;
}//main

---

gg