How to avoid OutOfMemoryException?

I am working on a game that needs to work with large data sets, basically it will process 1000 of them and store it's raw data into memory temporarily because it's fast and will not affect game play. Another thread will be called, then process each data set individually, storing them to the SQL database. These can be anywhere from a few bytes, to a few MB each.

So basically, there is lot of allocation/deallocation going on, and this is all happening in seconds, in a continuous loop. This is to save any changed data, as it gets changed (items, mobs etc).

Often, I will run into the dreaded OOME even though there is tons of ram available. According to my research, windows apps have a limit of about 1.5GB to 2GB. The app can reach near that at times. Is there not a way to circumvent this limit? I'm sure there are much larger apps out there that need to allocate much larger chunks of data, and need to hold much more data into memory, such as 3D CAD apps, games, etc... so what gives? How do they do it?

Another thing I was reading is that memory can become fragmented, and when allocating, if there is not a single chunk of space big enough, it will also crash, instead of allocating different parts, or doing some kind of defrag operation. So how would I go about dealing with that part? Is there maybe a defrag function of some sort I can call?

Thanks in advance for the help!

When running on a 32bit setup, .NET have a limit of around 1.5 GB. (Address space properly).
Run on a 64bit setup if you need more.

Will I really get more then 1.5GB in 64-bit though? I was reading somewhere that it's still a limitation. (on a per app basis).

Yes - .NET can address a larger space in x64 and therefore you can use more memory.
Now, whether it allocates more physical memory or just uses swapping or what not, I've not looked in to, though.
But I ran some tests at one point because I encountered this problem while doing some XSLT via .NET and it crashed consistently with OutOfMemory at around 1.4-1.5 GB on a 32 bit setup.

However there might also be an issue with your code containing memory leaks and what not which might be fixed and helps, but well - as said - x32 bit crashes when using that much.

Quote:

---

I was reading somewhere that it's still a limitation. (on a per app basis).

---

There is still a limit, but I doubt you could afford the buy all that ram or even find a way of actually hooking it all up into one computer. It's a very very very large number ;)

What you should probably do is run your app in a heap profiler and see if you're retaining objects in memory which you aren't expecting to retain.

Are you closing the SqlConnection and SqlCommand objects as outlined in this thread from last year?

http://www.codeguru.com/forum/showth...=483366&page=2