Stream Reader/Writer Encoding

[mansoor_aziz]

Hello all,

I am having a problem while setting encoding for stream/writer reader using c#.

I have to download text files using ftp and http requets.

The problem is that i dont know the encoding of text files.

i am using code below.

Stream readStream = new MemoryStream();
readStream= FTPwebResponse.GetResponseStream();
or
readStream = HTTPwebResponse.GetResponseStream();

then i will read it through stream reader

StreamReader reader = new StreamReader(readStream, Encoding.?????);
String responce = reader.ReadToEnd();

StreamWriter writeStream = new StreamWriter(FileNameWithPath, true, Encoding.????????);

writeStream.Write(responce);

here i dont know which encoding to use. if i dont use encoding i am getting sone unknown characters in some files. that are italian, spanish, arabic etc. some of these are comming fine but in some places i am getting boxes and question marks.

is there any way to detect encoding of stream ? so that i can use it for in stream reader/writer??

hope some one will help me out.

thanks in advance.

Best Regards,
Mansoor.

[mansoor_aziz]

Any one who can help????

any guru there ?

[torrud]

Well the problem is that you have to know which kind of encoding the other side of your communication use. If you use the HTTP protocol you can hope that the header lines are correct. Then you will get something like:

Content-Type: text/html; charset=utf-8

.

[riscutiavlad]

If you want to send/receive files, try using a BinaryReader and a BinaryWriter for this.

StreamReader and StreamWriters work on text files and only on specific encodings - that's why some text files might not transfer correctly if you specify the wrong encoding. If you don't care about the content and just want to transfer them, binary readers/writers, as their name implies, will treat files as binary files, reading and writing chunks of bytes which guarantees that the file won't be misinterpreted.

[cjard]

Unless the files youre downloading have byte order marks in them then there isnt much you can do to get the machine to read them and guess what encoding was used.

If you think about it a little deeper: Why are you asking us what the encoding of the files is? Thats like coming here and saying "right, ive got this document froma guy called Wilhelm Schmitz and its in some language that I dont know. what language is it?"

Er... We might guess that it's a German name so it might be written in german, but Wilhelm might be fluent Latin and wrote the document in that. We havent seen the document and if we did, there would be some guessing but we wouldnt know for definite. If noone here spoke Latin, then we wouldnt be able to answer..

Go ask the person who wrote the document!

Similarly, if the files youre retrieving have a specific encoding then find the person who created the files and ask them what encoding they used!

[mansoor_aziz]

Unless the files youre downloading have byte order marks in them then there isnt much you can do to get the machine to read them and guess what encoding was used.

If you think about it a little deeper: Why are you asking us what the encoding of the files is? Thats like coming here and saying "right, ive got this document froma guy called Wilhelm Schmitz and its in some language that I dont know. what language is it?"

Er... We might guess that it's a German name so it might be written in german, but Wilhelm might be fluent Latin and wrote the document in that. We havent seen the document and if we did, there would be some guessing but we wouldnt know for definite. If noone here spoke Latin, then we wouldnt be able to answer..

Go ask the person who wrote the document!

Similarly, if the files youre retrieving have a specific encoding then find the person who created the files and ask them what encoding they used!

riscutiavlad:
well i havent tried binary reader/writer yet, i will try it thanks alot for advise.

torrud :
yes i can get encoding for files downloaded through http rewuest as below
Encoding en = Encoding.GetEncoding(HTTPwebResponse.CharacterSet);
but it is not working with ftp it does not have any functionality loke that or may be it has but i could find it, please advise me if any one knows?

cjard :
it would be so easy for me if i could know the encoding of file before downloading. and even i could its not a better idea because i have different users uploading their files and number of users is also not fixed they can grow. so if i say to users tell me what encoding you are using then i have to hard code encodings for different users, thats not seems good to me. there should be one gereral way of downloading any file.

anyway thanks all for help. I will try it using binary reader hope it works.

[mansoor_aziz]

finnally got the soultion, binary reader works yepeeee..
thanks all who helped.

here is the code, I think the way i used to read from stream and write is crap, if any one knows better then please tell me...as readStream is memory stream i canot perform any kind of seek operation on it.

i done it like that
BinaryReader br = new BinaryReader(readStream);
BinaryWriter bw = new BinaryWriter(System.IO.File.Open(FileNameWithPath, FileMode.Create), Encoding.UTF8);
byte[] buff = new byte[1024];
int c=1;
while(c > 0 )
{
c = br.Read(buff, 0, 1024);
for(int i=0;i<c;i++)
bw.Write(buff[i]);

}

bw.Close();






here is complete code might help someone else.

FtpWebRequest FTPRequest;
HttpWebRequest HTTPRequest;
WebResponse FTPwebResponse;
HttpWebResponse HTTPwebResponse = null;
Stream readStream = new MemoryStream();
Encoding Enc;
StreamReader reader;
string responce;
Uri uri = new Uri(url);
if (url.Contains("ftp"))
{
FTPRequest = (FtpWebRequest)WebRequest.Create(uri);
FTPRequest.Timeout = (1000 * 60 * 30);
FTPwebResponse = FTPRequest.GetResponse();
readStream = FTPwebResponse.GetResponseStream();
}
else
if (url.Contains("http"))
{
HTTPRequest = (HttpWebRequest)WebRequest.Create(uri);
HTTPRequest.Timeout = (1000 * 60 * 30);
HTTPwebResponse = (HttpWebResponse)HTTPRequest.GetResponse();
readStream = HTTPwebResponse.GetResponseStream();
}
BinaryReader br = new BinaryReader(readStream);
BinaryWriter bw = new BinaryWriter(System.IO.File.Open(FileNameWithPath, FileMode.Create), Encoding.UTF8);
byte[] buff = new byte[1024];
int c=1;
while(c > 0 )
{
c = br.Read(buff, 0, 1024);
for(int i=0;i<c;i++)
bw.Write(buff[i]);

}

bw.Close();

[torrud]

BinaryReader br = new BinaryReader(readStream);
BinaryWriter bw = new BinaryWriter(System.IO.File.Open(FileNameWithPath, FileMode.Create), Encoding.UTF8);
byte[] buff = new byte[1024];
int c=1;
while(c > 0 )
{
c = br.Read(buff, 0, 1024);
for(int i=0;i<c;i++)
bw.Write(buff[i]);

}

bw.Close();

Hm, during the instantiation of the BinaryWriter you use always UTF8 encoding. Works this correct if the read file is encoded ISO 8859-12 for instance?