I am working on a game that needs to work with large data sets, basically it will process 1000 of them and store it's raw data into memory temporarily because it's fast and will not affect game play. Another thread will be called, then process each data set individually, storing them to the SQL database. These can be anywhere from a few bytes, to a few MB each.

So basically, there is lot of allocation/deallocation going on, and this is all happening in seconds, in a continuous loop. This is to save any changed data, as it gets changed (items, mobs etc).

Often, I will run into the dreaded OOME even though there is tons of ram available. According to my research, windows apps have a limit of about 1.5GB to 2GB. The app can reach near that at times. Is there not a way to circumvent this limit? I'm sure there are much larger apps out there that need to allocate much larger chunks of data, and need to hold much more data into memory, such as 3D CAD apps, games, etc... so what gives? How do they do it?

Another thing I was reading is that memory can become fragmented, and when allocating, if there is not a single chunk of space big enough, it will also crash, instead of allocating different parts, or doing some kind of defrag operation. So how would I go about dealing with that part? Is there maybe a defrag function of some sort I can call?

Thanks in advance for the help!