[RESOLVED] 'System.OutOfMemoryException'
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 7 of 7

Thread: [RESOLVED] 'System.OutOfMemoryException'

  1. #1
    Join Date
    Aug 2007
    Posts
    30

    [RESOLVED] 'System.OutOfMemoryException'

    Hi All,

    i am getting 'System.OutOfMemoryException' while executing code below

    StreamReader readerFile = new StreamReader(sReadResultPath, System.Text.Encoding.UTF8);
    stringFile= readerFile.ReadToEnd()

    file is of big size it is of 250 MB , is there any wayaround to read it quickly, than i have to replace certian strings in it like

    stringReplaced = stringFile.Replace("&lt;", "<");
    stringReplaced = stringReplaced.Replace("&gt;", ">");

    thanks in advance.

    best regards,
    Mansoor.

  2. #2
    Join Date
    Nov 2003
    Posts
    2,185

    Re: 'System.OutOfMemoryException'

    You can read it by blocks instead of at once. That will use much lesser memory.

    See the ReadBlock function documentation.

  3. #3
    Join Date
    Aug 2007
    Posts
    30

    Re: 'System.OutOfMemoryException'

    i have tried readline() method but it was v slow,

    lets try read block method too.

  4. #4
    Join Date
    Aug 2007
    Posts
    30

    Re: 'System.OutOfMemoryException'

    got the solution. it might help some one

    so here is the code


    StreamReader readerFile = new StreamReader(sReadResultPath, System.Text.Encoding.UTF8);

    while (!readerFile.EndOfStream)
    {
    StreamWriter writerFile = new StreamWriter(sWriteNewXMLPath + "\\" + OrignalFileName, true);
    StringBuilder sb = new StringBuilder();
    char[] buff = new char[10 * 1024 * 1024];
    int xx = readerFile.ReadBlock(buff, 0, buff.Length);
    sb.Append(buff);
    stringReplaced = sb.ToString();
    stringReplaced = stringReplaced.Replace("&lt;", "<");
    stringReplaced = stringReplaced.Replace("&gt;", ">");
    writerFile.WriteLine(stringReplaced);
    writerFile.Close();
    writerFile.Dispose();
    stringReplaced = null;
    sb = null;
    }


    readerFile.Close();
    readerFile.Dispose();

  5. #5
    Join Date
    May 2007
    Posts
    1,546

    Re: 'System.OutOfMemoryException'

    Why read in 10MB blocks? Reading in 256kB blocks would be just as efficient.

    Code:
    char[] buff = new char[10 * 1024 * 1024];
    int xx = readerFile.ReadBlock(buff, 0, buff.Length);
    sb.Append(buff);
    stringReplaced = sb.ToString();
    stringReplaced = stringReplaced.Replace("&lt;", "<");
    stringReplaced = stringReplaced.Replace("&gt;", ">");
    writerFile.WriteLine(stringReplaced);
    So first you allocate a 10mb char[], then you allocate a 10mb StringBuilder. Next you allocate a 10 mb string, then another 10mb string, then another 10mb string and finally write that string to disk. Why? Isn't that a tad excessive? Wouldn't it make more sense to do:

    Code:
    char[] buff = new char[10 * 1024 * 1024];
    int xx = readerFile.ReadBlock(buff, 0, buff.Length);
    sb.Append(buff);
    sb.Replace("&lt;", "<");
    sb.Replace("&gt;", ">");
    writerFile.WriteLine(sb.ToString());
    Allocate a 10MB char array, allocate 10MB stringbuilder then allocate a 10MB string and write it to disk. That's 30mb as opposed to the 50/60MB the first version did.

    Also, you allocate a new 10MB char buffer, 10MB string builder and a streamwriter every time you loop. This is very inefficient. Reuse the char array, stringbuilder and streamwriter! Finally, read in chunks of 64kB, or 128kB. There's no need to read 10MB at a time. It's no more efficient than reading 64kB.
    Last edited by Mutant_Fruit; April 8th, 2008 at 11:25 AM.
    www.monotorrent.com For all your .NET bittorrent needs

    NOTE: My code snippets are just snippets. They demonstrate an idea which can be adapted by you to solve your problem. They are not 100% complete and fully functional solutions equipped with error handling.

  6. #6
    Join Date
    Mar 2008
    Location
    Atlanta, GA
    Posts
    49

    Re: 'System.OutOfMemoryException'

    I don't think your latest comment is the correct solution. It may fix your OOM exception problem, but in the event a "&lt;" spans more than 1 block, or:


    block 1: "...&"
    block 2: "lt;..."

    You're going to miss this case.

    What you have in your first comment will probably work, but what you need to keep in mind is the immutability of strings. This means that every string you use is a brand new instance of string in memory. From your code:

    Code:
    StreamReader readerFile = new StreamReader(sReadResultPath, System.Text.Encoding.UTF8);
    stringFile= readerFile.ReadToEnd(); // stringFile is now 250 megs
    
    stringReplaced = stringFile.Replace("&lt;", "<");  // stringReplaced is now 250 megs
    stringReplaced = stringReplaced.Replace("&gt;", ">");
    So all total, just the strings in the above section of code totals at least 500 megs in memory, not counting what the unclosed stream is using (which may be negligible, I don't know). And not counting the spikes of memory during the Replace() calls. Who knows how many new strings that sucker creates during the loop. Definitely close that stream before you start your Replace() just to see and if possible only do operations on one string if stringReplaced isn't necessary, change it to stringFile so the stringFile instance isn't hanging around out there taking up useless chunks of memory.

    Sorry I don't have The Answer for you, but I'm at work and only get a few minutes a day to try and help. But I thought I'd try and provide some things to think about.

  7. #7
    Join Date
    Aug 2007
    Posts
    30

    Re: 'System.OutOfMemoryException'

    thats the reason i used 10 MB block..there is less chance of missing event like you explained

    block 1: "...&"
    block 2: "lt;..."

    most of files are under 10 MB

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center