[RESOLVED] 'System.OutOfMemoryException'

[mansoor_aziz]

Hi All,

i am getting 'System.OutOfMemoryException' while executing code below

StreamReader readerFile = new StreamReader(sReadResultPath, System.Text.Encoding.UTF8);
stringFile= readerFile.ReadToEnd()

file is of big size it is of 250 MB , is there any wayaround to read it quickly, than i have to replace certian strings in it like

stringReplaced = stringFile.Replace("&lt;", "<");
stringReplaced = stringReplaced.Replace("&gt;", ">");

thanks in advance.

best regards,
Mansoor.

[Tischnoetentoet]

You can read it by blocks instead of at once. That will use much lesser memory.

See the ReadBlock function documentation.

[mansoor_aziz]

i have tried readline() method but it was v slow,

lets try read block method too.

[mansoor_aziz]

got the solution. it might help some one

so here is the code


StreamReader readerFile = new StreamReader(sReadResultPath, System.Text.Encoding.UTF8);

while (!readerFile.EndOfStream)
{
StreamWriter writerFile = new StreamWriter(sWriteNewXMLPath + "\\" + OrignalFileName, true);
StringBuilder sb = new StringBuilder();
char[] buff = new char[10 * 1024 * 1024];
int xx = readerFile.ReadBlock(buff, 0, buff.Length);
sb.Append(buff);
stringReplaced = sb.ToString();
stringReplaced = stringReplaced.Replace("&lt;", "<");
stringReplaced = stringReplaced.Replace("&gt;", ">");
writerFile.WriteLine(stringReplaced);
writerFile.Close();
writerFile.Dispose();
stringReplaced = null;
sb = null;
}


readerFile.Close();
readerFile.Dispose();

[Mutant_Fruit]

Why read in 10MB blocks? Reading in 256kB blocks would be just as efficient.

char[] buff = new char[10 * 1024 * 1024];
int xx = readerFile.ReadBlock(buff, 0, buff.Length);
sb.Append(buff);
stringReplaced = sb.ToString();
stringReplaced = stringReplaced.Replace("&lt;", "<");
stringReplaced = stringReplaced.Replace("&gt;", ">");
writerFile.WriteLine(stringReplaced);

So first you allocate a 10mb char[], then you allocate a 10mb StringBuilder. Next you allocate a 10 mb string, then another 10mb string, then another 10mb string and finally write that string to disk. Why? Isn't that a tad excessive? Wouldn't it make more sense to do:

char[] buff = new char[10 * 1024 * 1024];
int xx = readerFile.ReadBlock(buff, 0, buff.Length);
sb.Append(buff);
sb.Replace("&lt;", "<");
sb.Replace("&gt;", ">");
writerFile.WriteLine(sb.ToString());

Allocate a 10MB char array, allocate 10MB stringbuilder then allocate a 10MB string and write it to disk. That's 30mb as opposed to the 50/60MB the first version did.

Also, you allocate a new 10MB char buffer, 10MB string builder and a streamwriter every time you loop. This is very inefficient. Reuse the char array, stringbuilder and streamwriter! Finally, read in chunks of 64kB, or 128kB. There's no need to read 10MB at a time. It's no more efficient than reading 64kB.

[opedog]

I don't think your latest comment is the correct solution. It may fix your OOM exception problem, but in the event a "&lt;" spans more than 1 block, or:


block 1: "...&"
block 2: "lt;..."

You're going to miss this case.

What you have in your first comment will probably work, but what you need to keep in mind is the immutability of strings. This means that every string you use is a brand new instance of string in memory. From your code:

StreamReader readerFile = new StreamReader(sReadResultPath, System.Text.Encoding.UTF8);
stringFile= readerFile.ReadToEnd(); // stringFile is now 250 megs

stringReplaced = stringFile.Replace("&lt;", "<");  // stringReplaced is now 250 megs
stringReplaced = stringReplaced.Replace("&gt;", ">");

So all total, just the strings in the above section of code totals at least 500 megs in memory, not counting what the unclosed stream is using (which may be negligible, I don't know). And not counting the spikes of memory during the Replace() calls. Who knows how many new strings that sucker creates during the loop. Definitely close that stream before you start your Replace() just to see and if possible only do operations on one string if stringReplaced isn't necessary, change it to stringFile so the stringFile instance isn't hanging around out there taking up useless chunks of memory.

Sorry I don't have The Answer for you, but I'm at work and only get a few minutes a day to try and help. But I thought I'd try and provide some things to think about.

[mansoor_aziz]

thats the reason i used 10 MB block..there is less chance of missing event like you explained

block 1: "...&"
block 2: "lt;..."

most of files are under 10 MB