How to prevent outofmemory exeption?

[Red Squirrel]

I have an app that has to load large amounts of data in chunks (maybe like, half a MB in a string) but for some reason I'll get out of memory errors even though there is TONS of ram left + page file. The memory used is then cleared once the string has been processed.

Is there a way to make the program less sensible to large amounts of ram being used? If the ram is there to use, then it should use it instead of crapping out.

[Shuja Ali]

OutOfMemory exception usually occurs when you are trying to save something whereas the variable that you are trying to save it in does not have enough memory. It does not mean that if you declare a string object, if there is 1 GB memory free on your system, it will not use 1 GB, it will only use the specified size.

I would suggest using stringbuilder instead of a string if your data is too large.

[TheCPUWizard]

I have an app that has to load large amounts of data in chunks (maybe like, half a MB in a string) but for some reason I'll get out of memory errors even though there is TONS of ram left + page file. The memory used is then cleared once the string has been processed.

Is there a way to make the program less sensible to large amounts of ram being used? If the ram is there to use, then it should use it instead of crapping out.

Remember objects over 80,000 bytes (that is 40,000 chars in Unicode) get allocated on the LArgeObject Heap, which is sucbject to fragmentation.
Use your debugger tools to look at how the memory is being used at the time of the OOM.

[Red Squirrel]

How would I go about ensuring it does not get fragmented?

I've also read I should try to reuse objects instead of redeclaring, will that help? Right now the string that holds all the data gets redeclared for each object (over 800k) some are small while some are very large.

[TheCPUWizard]

How would I go about ensuring it does not get fragmented?

Dont allocate and release blocks of differig sizes that are over 80,000 bytes in size.

[qupte]
I've also read I should try to reuse objects instead of redeclaring, will that help?
[/quote]
Re-Read above...

[/quote]Right now the string that holds all the data gets redeclared for each object (over 800k) some are small while some are very large.[/QUOTE]
Strings are immutable, and ALWAYs involve reallocation.

If you combine these, irt means one very simple thing...A robust application should not have strings over 40,000 (actually slightly smaller). PERIOD.

[Mutant_Fruit]

I have an app that has to load large amounts of data in chunks (maybe like, half a MB in a string)

Do you actually need all that in memory at the same time? If so, do you actually need it to be a single chunk of memory? As TheCPUWizard said, you're more than likely hitting issues with the large object heap. If you avoided having to have all that data in-memory in one single chunk of memory, you'd completely avoid the issue.

[Red Squirrel]

Dont allocate and release blocks of differig sizes that are over 80,000 bytes in size.

I've also read I should try to reuse objects instead of redeclaring, will that help?

Re-Read above...

Right now the string that holds all the data gets redeclared for each object (over 800k) some are small while some are very large.

Strings are immutable, and ALWAYs involve reallocation.

If you combine these, irt means one very simple thing...A robust application should not have strings over 40,000 (actually slightly smaller). PERIOD.
[/quote]
I don't have a choice since I need to load that data into memory, and it comes from a SQL database so it needs to be taken in one chunk, processed, then move to next row. Thing is, it's freed up after.

What's the point of having 3GB of ram FREE if I can't even use it? If I had the choice this app would be written in C++ which has less limitations but I don't have a choice here. There's got to be a way to get around these errors without "not using that much ram in one variable" since I'm not even using close to what there is available.

[Red Squirrel]

Do you actually need all that in memory at the same time? If so, do you actually need it to be a single chunk of memory? As TheCPUWizard said, you're more than likely hitting issues with the large object heap. If you avoided having to have all that data in-memory in one single chunk of memory, you'd completely avoid the issue.

Yes these strings have to be in single chunks. The way the app was engineered at first was not very SQL friendly, but I could not keep it using files as it was getting inefficient. So now each Item, Mobile, etc is stored as a string of data into a SQL row. This row is then read and put in a string, this string is then processed and the individual variables of the object are populated, then the string is discarded. This process only happens when the world has to load.

The program is for a game server so we're talking TONS of data here, of various lengths. I will be optimizing things as I go. The biggest culpit now is house data so I may find another way to store these but I can't see how this method can't work considering I have enough ram anyway.

Eventually I may convert this app to C++ to overcome .net / C# limitations, but that's a long way ahead. I prefer dealing with pointers and stuff as at least I have full control over memory management.

[TheCPUWizard]

I sincerely doubt that you will be able to get a stable solution without a complete re-design. This one symptom alone indicates that the developer(s) had no clue as to how to write a proper .NET application.

Translation....

If I was approached to take this on as a paid contract, the ONLY way I would approach the job is as a complete re-design and re-write. REPEATED experience has shown that unless an application is designed from the ground up following best practices, that attempting to retro-fit it, is unlikely to succeed and will almost always result in major cost over runs, and a low probability of success...

[Red Squirrel]

I sincerely doubt that you will be able to get a stable solution without a complete re-design. This one symptom alone indicates that the developer(s) had no clue as to how to write a proper .NET application.

Translation....

If I was approached to take this on as a paid contract, the ONLY way I would approach the job is as a complete re-design and re-write. REPEATED experience has shown that unless an application is designed from the ground up following best practices, that attempting to retro-fit it, is unlikely to succeed and will almost always result in major cost over runs, and a low probability of success...

I agree with that, and that is actually plan B, to redo the whole thing from scratch, but it will take a very long time, and I wish to end this project by 2010. So I'm hoping there's a way around this issue such as a function of some sort to make it defragment the memory (probably very often) durring the load process.

An idea I also had was to have a C++ app read the DB and dump the contents to file, then have the program read from the file this way I can control my buffer size, but this will add more complexity as well.

I was testing and I can hold a few GB of data in a single string. I had a loop that just kept concatenating a char.

[TheCPUWizard]

I'll bet you that if you take the "lets fix it incrementally" approach that you will NOT have a robust working product by the end of 2010.....

There is NO way to defragment the LOH. EVERY application developer should be using a performance monitoring tool while they are writing and testing code and be measuring the number of LOH allocations, and their size distribution, ESPECIALLY if they have a program that they intend to be continously executing (ie a "sever" program).

[Red Squirrel]

Hmm so guess I'll have to have a secondary app that reads the data from the DB then outputs to file then. This way I can read it in smaller buffers. Thing is, it's not like I'm keeping those strings filled, it's just a buffer that is temp storage for a SQL row and then formatted properly in the program. a proper DB structure is not an option for this app unless I recode it from scratch which I'm trying to avoid.

Yes if I do it from scratch it would be 10 times more efficient, but I just don't want to spend that much time on it.

I've asked myself many times if I should though as I keep running into road blocks and it's usually a limitation of C# or just something that is horribly coded.

[TheCPUWizard]

I've asked myself many times if I should though as I keep running into road blocks and it's usually a limitation of C# or just something that is horribly coded.

I have yet to find a SINGLE instance of the former (limitation of the language), but have professionally encountered thousands of the latter (horrible code).

[Mutant_Fruit]

The program is for a game server so we're talking TONS of data here, of various lengths..

Eventually I may convert this app to C++ to overcome .net / C# limitations, but that's a long way ahead. I prefer dealing with pointers and stuff as at least I have full control over memory management.

C# should not be a limitation when writing a game server. There are some massive games out there using C# which have no scalability issues. SecondLife would be the biggest one that I can think of right now. Bear in mind that if decide to rewrite in C++, you'll be starting from scratch. Maybe it'd be better to just start from scratch and rewrite it properly in C#.

I've asked myself many times if I should though as I keep running into road blocks and it's usually a limitation of C# or just something that is horribly coded.

If performance is a concern and the current architecture does not allow for performance to scale, then the current architecture design needs to revisited and altered. If that requires a rewrite, then so be it. It's better to do it now rather than 3 months before the release. This is a question you need to answer asap.

[Zaccheus@Work]

This row is then read and put in a string

Does the whole row need to be read in one go?
Is there no way to read part of a row?