UTF8 Conversion problem

[well90]

Hi,

I am retrieving a string from the MySQL database as UTF8.
Now ofcourse i checked if the string itself is ok by writing the retrieved string to a file and opened it in notepad++ under UTF8 format and it reads it correctly.
After using this code:

res = stmt->executeQuery("SELECT * FROM `members` WHERE `member_id` = 50");
while (res->next())
{
    WCHAR wText[1000];
    MultiByteToWideChar(CP_UTF8, 0, res->getString("title").c_str(), -1, wText, 1000);
    WriteToFile("Test.txt", wText);
}
// res->getString("title").c_str() = UTF8 String which im talking about 

I open the file in notepad++ and i see gibberish characters, i tried opening it under every existing encoding formats and all show gibberish.
Am i doing something in the conversion API?
I am trying to convert a UTF8 string to a UNICODE string (That UNICODE string that is supported by wprintf %s).

Any help would be greatly appreciated.
Thank you.

[juanpast]

Hi.

If you stop the debugger in "WriteToFile("Test.txt", wText);": is correct the value of wText?

[VictorN]

What is res?
What is getString?
What does MultiByteToWideChar return?
What is WriteToFile?

[Codeplug]

>> I am trying to convert a UTF8 string to a UNICODE string (That UNICODE string that is supported by wprintf %s).
Here is code for that: http://forums.codeguru.com/showthrea...31#post1790131

gg

[Igor Vartanov]

99% the garbage is because of WriteToFile. Remember, you write unicode text to file. So, is the function unicode aware? Did you manage to start the file with proper BOM?

[well90]

res and GetString() are MySQL objects, they are used to retreive the UTF8 string and they works as i mentioned in my post, i saved the UTF8 string using WriteToFile() and then opened in notepad++ under UTF8 encoding, works perfect.

WriteToFile() & WriteToFileW():

inline void WriteToFile(char* pFileName, char* text)
{
    GetModuleFileNameA(GetModuleHandleA(NULL), dlldir, 512);
    for(int i = strlen(dlldir); i > 0; i--) { if(dlldir[i] == '\\') { dlldir[i+1] = 0; break; } }

    HANDLE hFile = CreateFileA(GetDirectoryFile(pFileName), GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (hFile)
    {
        SetFilePointer(hFile, 0, 0, FILE_END);

        DWORD szOut;
        WriteFile(hFile, (void*)text, strlen(text), &szOut, NULL);
        CloseHandle(hFile);
    }
}

inline void WriteToFileW(char* pFileName, wchar_t* text)
{
    GetModuleFileNameA(GetModuleHandleA(NULL), dlldir, 512);
    for(int i = strlen(dlldir); i > 0; i--) { if(dlldir[i] == '\\') { dlldir[i+1] = 0; break; } }

    HANDLE hFile = CreateFileA(GetDirectoryFile(pFileName), GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (hFile)
    {
        SetFilePointer(hFile, 0, 0, FILE_END);

        DWORD szOut;
        WriteFile(hFile, (void*)text, wcslen(text)*2, &szOut, NULL);
        CloseHandle(hFile);
    }
} 

I dont see why they would not work.

Updated code:

res = stmt->executeQuery("SELECT * FROM `members` WHERE `member_id` = 50");
while (res->next())
{
    WCHAR wText[1000];
    MultiByteToWideChar(CP_UTF8, 0, res->getString("title").c_str(), -1, wText, 1000);
    WriteToFileW("Test.txt", wText);
} 

UTF8 String: สุดสุดสุดสุดสุด
Output in Test1.txt: *8*8*8*8*8
File Size: 30 bytes

There is really something wrong somewhere i cant find what.

Thank you for your help.

[VictorN]

res and GetString() are MySQL objects

And "MySQL objects" return std::string?
Or what is

res->getString("title").c_str()

And please, don't use PHP tags for C++ code, use Code tags (#) instead.

[Paul McKenzie]

res and GetString() are MySQL objects,

Those are objects that do things in your code we still don't know enough about, for example c_str(). What is returned from that function?

To make this simple, replace the third parameter of that call to MultiByteToWideChar with a hard-coded string instead of those calls to functions we have little knowledge of. If you did that, does that hard-coded string convert correctly?

If it does output correctly, then the root of the problem are those getString() and c_str() calls you're making. The focus is then moved away from the Windows calls and is now moved to your MySQL functions calls. If that string doesn't output correctly, then we can forget about all of that MySQL stuff and concentrate solely on getting a seemingly very simple string conversion to work correctly.

Regards,

Paul McKenzie

[well90]

Those are objects that do things in your code we still don't know enough about, for example c_str(). What is returned from that function?

To make this simple, replace the third parameter of that call to MultiByteToWideChar with a hard-coded string instead of those calls to functions we have little knowledge of. If you did that, does that hard-coded string convert correctly?

If it does output correctly, then the root of the problem are those getString() and c_str() calls you're making. The focus is then moved away from the Windows calls and is now moved to your MySQL functions calls. If that string doesn't output correctly, then we can forget about all of that MySQL stuff and concentrate solely on getting a seemingly very simple string conversion to work correctly.

Regards,

Paul McKenzie

I cant write thai characters in a simple char[] object, the compiler replace those chars with questions marks.
This code:

char* szTest1 = "สุดสุดสุดสุดสุด";
CLogger::WriteToFile("Test1.txt", szTest1);

Output in Text1.txt: ???????????????
And the compiler complains about:
Warning 18 warning C4566: character represented by universal-character-name '\u0E14' cannot be represented in the current code page (1252) E:\VS\MySQL Client Test\MySQL Client Test\MySQL Client Test.cpp 52 1 MySQL Client Test


Thats why i had to paste the MySQL way.

Now about your doubts of the MySQL functions, as ive mentioned in my previous post, i tested the getString().c_str() function by writing its returned value directly into a file... and the file have the correct thai string... therefor all mysql functions works fine.

res->getString("title") returns a std::string object.

[well90]

Ok, I have better example so you can stop making doubts on MySQL.

char* szTest1 = "t\xE9st"; // tést
WriteToFile("Test1.txt", szTest1);

I open Test1.txt in notepad and I DO SEE "tést".

Now when i test this:

WCHAR wText[1000];
char* szTest1 = "t\xE9st"; // tést
MultiByteToWideChar(CP_UTF8, 0, szTest1, -1, wText, 1000);
WriteToFileW("Test1.txt", wText);

I open Test1.txt in notepad and i see "t�st".

Now its 100% not MySQL...
Anything that i convert get into gibberish... atleast UTF8 characters.

[Codeplug]

>> I open Test1.txt in notepad and i see "t�st".
The UTF8 encoding for small letter E with acute is "\xC3\xA9". You should be checking for error codes from functions that return them.
Next you need to add a UTF16-LE BOM at the front of your log file - since that is what you're writing. Or you can tell notepad++ that the encoding is "UCS-2 Little Endian".

gg

[well90]

After telling Notepad++ that the encoding is "UCS-2 Little Endian", i see this "t�st", except the � is a cube.

Another test:

wprintf(L"Member Name: %ls \n", wText); // Prints 't?st'

wprintf(L"Member Name: %s \n", wText); // Prints 't?st'

WTH is going on...

[Codeplug]

>> The UTF8 encoding for small letter E with acute is "\xC3\xA9".

const char* szTest1 = "t\xC3\xA9st"; // tést

>> Another test:
http://cboard.cprogramming.com/cplus...ml#post1086757

gg

[Paul McKenzie]

I cant write thai characters in a simple char[] object,

Yes you can if you know the escape codes. See how CodePlug writes string literals using the hex escape codes.

Regards,

Paul McKenzie

[well90]

>> The UTF8 encoding for small letter E with acute is "\xC3\xA9".

const char* szTest1 = "t\xC3\xA9st"; // tést

>> Another test:
http://cboard.cprogramming.com/cplus...ml#post1086757

gg

Weird, why this site encode that 'é' with 0xE9 ?
http://people.physics.anu.edu.au/~mx...pt/jsUTF8.html