I have an application in which I do some pre-processing of a flat file. The result of the pre-processing is a ton of data. Next, the application works on this data. This part is very time sensitive and is the reason I pre-process the flat file.... so that I can simply blow through the data. Each piece of data is stored as a structure consisting of a float and 2 chars. After pre-processing a typical file I could potentially have 40,800,000 dynamically allocated structures.

I initially started off using vectors. It worked decent with smaller files. As the file size increased this became extremely slow.

I don't know exactly how memory is allocated when requesting such large amounts. I assume that at some point (when?) the contents of physical memory are paged out to virtual memory.

My next thought is to simply output the results of the pre-processing to another flat file. So, all I would have to do when the time comes to execute, is open the file and read each chunk of data and act on it.

Any suggestions?