-
June 24th, 2012, 06:38 AM
#1
UTF8 Conversion problem
Hi,
I am retrieving a string from the MySQL database as UTF8.
Now ofcourse i checked if the string itself is ok by writing the retrieved string to a file and opened it in notepad++ under UTF8 format and it reads it correctly.
After using this code:
PHP Code:
res = stmt->executeQuery("SELECT * FROM `members` WHERE `member_id` = 50");
while (res->next())
{
WCHAR wText[1000];
MultiByteToWideChar(CP_UTF8, 0, res->getString("title").c_str(), -1, wText, 1000);
WriteToFile("Test.txt", wText);
}
// res->getString("title").c_str() = UTF8 String which im talking about
I open the file in notepad++ and i see gibberish characters, i tried opening it under every existing encoding formats and all show gibberish.
Am i doing something in the conversion API?
I am trying to convert a UTF8 string to a UNICODE string (That UNICODE string that is supported by wprintf %s).
Any help would be greatly appreciated.
Thank you.
-
June 25th, 2012, 02:38 AM
#2
Re: UTF8 Conversion problem
Hi.
If you stop the debugger in "WriteToFile("Test.txt", wText);": is correct the value of wText?
-
June 25th, 2012, 03:05 AM
#3
Re: UTF8 Conversion problem
What is res?
What is getString?
What does MultiByteToWideChar return?
What is WriteToFile?
Victor Nijegorodov
-
June 25th, 2012, 02:39 PM
#4
Re: UTF8 Conversion problem
res and GetString() are MySQL objects, they are used to retreive the UTF8 string and they works as i mentioned in my post, i saved the UTF8 string using WriteToFile() and then opened in notepad++ under UTF8 encoding, works perfect.
WriteToFile() & WriteToFileW():
PHP Code:
inline void WriteToFile(char* pFileName, char* text)
{
GetModuleFileNameA(GetModuleHandleA(NULL), dlldir, 512);
for(int i = strlen(dlldir); i > 0; i--) { if(dlldir[i] == '\\') { dlldir[i+1] = 0; break; } }
HANDLE hFile = CreateFileA(GetDirectoryFile(pFileName), GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
if (hFile)
{
SetFilePointer(hFile, 0, 0, FILE_END);
DWORD szOut;
WriteFile(hFile, (void*)text, strlen(text), &szOut, NULL);
CloseHandle(hFile);
}
}
inline void WriteToFileW(char* pFileName, wchar_t* text)
{
GetModuleFileNameA(GetModuleHandleA(NULL), dlldir, 512);
for(int i = strlen(dlldir); i > 0; i--) { if(dlldir[i] == '\\') { dlldir[i+1] = 0; break; } }
HANDLE hFile = CreateFileA(GetDirectoryFile(pFileName), GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
if (hFile)
{
SetFilePointer(hFile, 0, 0, FILE_END);
DWORD szOut;
WriteFile(hFile, (void*)text, wcslen(text)*2, &szOut, NULL);
CloseHandle(hFile);
}
}
I dont see why they would not work.
Updated code:
PHP Code:
res = stmt->executeQuery("SELECT * FROM `members` WHERE `member_id` = 50");
while (res->next())
{
WCHAR wText[1000];
MultiByteToWideChar(CP_UTF8, 0, res->getString("title").c_str(), -1, wText, 1000);
WriteToFileW("Test.txt", wText);
}
UTF8 String: สุดสุดสุดสุดสุด
Output in Test1.txt: *8*8*8*8*8
File Size: 30 bytes
There is really something wrong somewhere i cant find what.
Thank you for your help.
-
June 25th, 2012, 03:01 PM
#5
Re: UTF8 Conversion problem
Originally Posted by well90
res and GetString() are MySQL objects
And "MySQL objects" return std::string?
Or what is
Code:
res->getString("title").c_str()
And please, don't use PHP tags for C++ code, use Code tags (#) instead.
Victor Nijegorodov
-
June 25th, 2012, 04:11 PM
#6
Re: UTF8 Conversion problem
Originally Posted by well90
res and GetString() are MySQL objects,
Those are objects that do things in your code we still don't know enough about, for example c_str(). What is returned from that function?
To make this simple, replace the third parameter of that call to MultiByteToWideChar with a hard-coded string instead of those calls to functions we have little knowledge of. If you did that, does that hard-coded string convert correctly?
If it does output correctly, then the root of the problem are those getString() and c_str() calls you're making. The focus is then moved away from the Windows calls and is now moved to your MySQL functions calls. If that string doesn't output correctly, then we can forget about all of that MySQL stuff and concentrate solely on getting a seemingly very simple string conversion to work correctly.
Regards,
Paul McKenzie
Last edited by Paul McKenzie; June 25th, 2012 at 04:18 PM.
-
June 25th, 2012, 04:35 PM
#7
Re: UTF8 Conversion problem
Originally Posted by Paul McKenzie
Those are objects that do things in your code we still don't know enough about, for example c_str(). What is returned from that function?
To make this simple, replace the third parameter of that call to MultiByteToWideChar with a hard-coded string instead of those calls to functions we have little knowledge of. If you did that, does that hard-coded string convert correctly?
If it does output correctly, then the root of the problem are those getString() and c_str() calls you're making. The focus is then moved away from the Windows calls and is now moved to your MySQL functions calls. If that string doesn't output correctly, then we can forget about all of that MySQL stuff and concentrate solely on getting a seemingly very simple string conversion to work correctly.
Regards,
Paul McKenzie
I cant write thai characters in a simple char[] object, the compiler replace those chars with questions marks.
This code:
Code:
char* szTest1 = "สุดสุดสุดสุดสุด";
CLogger::WriteToFile("Test1.txt", szTest1);
Output in Text1.txt: ???????????????
And the compiler complains about:
Warning 18 warning C4566: character represented by universal-character-name '\u0E14' cannot be represented in the current code page (1252) E:\VS\MySQL Client Test\MySQL Client Test\MySQL Client Test.cpp 52 1 MySQL Client Test
Thats why i had to paste the MySQL way.
Now about your doubts of the MySQL functions, as ive mentioned in my previous post, i tested the getString().c_str() function by writing its returned value directly into a file... and the file have the correct thai string... therefor all mysql functions works fine.
res->getString("title") returns a std::string object.
-
June 25th, 2012, 04:56 PM
#8
Re: UTF8 Conversion problem
Ok, I have better example so you can stop making doubts on MySQL.
Code:
char* szTest1 = "t\xE9st"; // tést
WriteToFile("Test1.txt", szTest1);
I open Test1.txt in notepad and I DO SEE "tést".
Now when i test this:
Code:
WCHAR wText[1000];
char* szTest1 = "t\xE9st"; // tést
MultiByteToWideChar(CP_UTF8, 0, szTest1, -1, wText, 1000);
WriteToFileW("Test1.txt", wText);
I open Test1.txt in notepad and i see "t�st".
Now its 100% not MySQL...
Anything that i convert get into gibberish... atleast UTF8 characters.
-
June 25th, 2012, 07:33 PM
#9
Re: UTF8 Conversion problem
Originally Posted by well90
I cant write thai characters in a simple char[] object,
Yes you can if you know the escape codes. See how CodePlug writes string literals using the hex escape codes.
Regards,
Paul McKenzie
Last edited by Paul McKenzie; June 25th, 2012 at 07:40 PM.
-
June 25th, 2012, 12:02 PM
#10
Re: UTF8 Conversion problem
>> I am trying to convert a UTF8 string to a UNICODE string (That UNICODE string that is supported by wprintf %s).
Here is code for that: http://forums.codeguru.com/showthrea...31#post1790131
gg
-
June 25th, 2012, 01:48 PM
#11
Re: UTF8 Conversion problem
99% the garbage is because of WriteToFile. Remember, you write unicode text to file. So, is the function unicode aware? Did you manage to start the file with proper BOM?
Best regards,
Igor
-
June 25th, 2012, 05:20 PM
#12
Re: UTF8 Conversion problem
>> I open Test1.txt in notepad and i see "t�st".
The UTF8 encoding for small letter E with acute is "\xC3\xA9". You should be checking for error codes from functions that return them.
Next you need to add a UTF16-LE BOM at the front of your log file - since that is what you're writing. Or you can tell notepad++ that the encoding is "UCS-2 Little Endian".
gg
-
June 25th, 2012, 06:39 PM
#13
Re: UTF8 Conversion problem
After telling Notepad++ that the encoding is "UCS-2 Little Endian", i see this "t�st", except the � is a cube.
Another test:
Code:
wprintf(L"Member Name: %ls \n", wText); // Prints 't?st'
Code:
wprintf(L"Member Name: %s \n", wText); // Prints 't?st'
WTH is going on...
-
June 25th, 2012, 07:23 PM
#14
Re: UTF8 Conversion problem
>> The UTF8 encoding for small letter E with acute is "\xC3\xA9".
Code:
const char* szTest1 = "t\xC3\xA9st"; // tést
>> Another test:
http://cboard.cprogramming.com/cplus...ml#post1086757
gg
-
June 25th, 2012, 08:39 PM
#15
Re: UTF8 Conversion problem
Originally Posted by Codeplug
Weird, why this site encode that 'é' with 0xE9 ?
http://people.physics.anu.edu.au/~mx...pt/jsUTF8.html
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|