How to handle big file memory usage-Help urgently

[VC123start]

I need to get the record infomation from a big text file about 50M. If I load the file to a 2 dimension string Array, eg sInfo[record][detaileLine], each line will be add into array. It will take a lot of memory and the program become very slow.

Any one has idea how to handle the situation? I need the get the record of the customer from the file frequently.

I read some post of the site and don't know whether the page mapping idea is sutiable for this? Should I mapping the original file and the search the file to get the record each time or should I put the big string arry into some place?

Thanks!!

[Darth Hacker]

Some more information would be helpful. What are you planning to do with this data? Do you just need to access certain records, or do you need access to all the data at once? What does the data look like? Is it ASCII or binary? Are the records fixed length or variable?

For example, if your data is organized into fixed length records, and you need random access capability to read certain records, the solution you need is pretty straightforward. Just calculate the byte offset into the file for the desired record, do a seek, then do a read. Otherwise, some other approach might be needed.

Reading all 50MB into memory isn't necessarily a bad option. It might slow you down, but virtual memory managers on today's operating systems are really good. Depending on how much RAM you have, how much disk swap space is available, and what other processes are already running on your PC, you may not notice too much of a slowdown.

[kirants]

If your file is a flat structure which you know well, you could try memory mapped file approach. Here you simply load the file and then access the data as if it were in memory. I don't know if it applies to your situation, but worth looking at.

MapViewOfFile

[Eli Gassert]

I would also suggest a memory mapped file like kirants suggested.

However, if this is not an option, then I would recommend taking a single pass through the file to index where data starts (I assume you have records in your file and it's not just a big blob of text?). At that point, if you know, say, record 1234 starts at byecount 38392, then you can open the file, seek to that position, read the record, and close the file.

But if your data doesn't allow the above suggestion, then I'd go with memory mapped file approach

[VC123start]

Thanks for the reply.

My flat file is ASCII text like

<begin character>record1<end charater>record2<end character>..

The record length is variable. when customer need record1, my jot is to check the file and to find the <begin character> and <end charater> to get the record and put it in a view and print it out for customer.

Now I load the file into memory and put record into to string array and also other infomation, so 50M flat file maybe take 200M memory. In my situation, it is too slow to meet customer requirement.

So In this situation, should I use 1. memory mapped file or 2.taking a single pass through the file to index where data starts?

rgds,

[kirants]

Have you considered using a database ?
There is also a jet text driver one can use to use csv type text files, I guess. I may be wrong here..

[VC123start]

Good idea.

I also consider sepearate the big file into small piece that each piece is on record. Then put it into small file or database as you suggested.

This is feasible. But I'm not sure this is the best way in my situation.

[kirants]

Using DBs make it easier to index records, query on different criteria, allows scalability in case you want to add more data etc..

However, note that you need to look at it from a end user perspective, licensiing perspective etc. , installation perspective

[Eli Gassert]

If you'll be updating your data often, I'd go database for sure. But if you just need to get data from it and don't wanna spend a lot of dev time, then the easiest option, in my opinion, is to take a pass through the file and remember the start and end position of each record

std::map< int, std::pair<int, int> > m_mapData;

your key would be the recordID (I assume it's an int?) and the std:air would be startPos and endPos, respectively.

STL has it spelled out for you... in a quick, easy, loop, you could have all the data mapped out for you.

But, I would also seriuosly consider Kirants's suggestion of a DB

good luck, and happy coding

[VC123start]

I just think my brain have a little clear understanding about it from Gassert' reply.

But kirants, what it mean

Do a clean build. Do you see any warnings ?

Now I still don't change my program (definitely I'll do it later) and just read record and put it into CString array, so it is no warning.

[kirants]

sorry posted in wrong thread
Will try to delete post..