|
-
January 10th, 2009, 09:37 PM
#7
Re: Excessively slow file sequence access
First of all, thanks for your assistance! You are going above and beyond what I expected.
But, your test is flawed:
1) stdio will buffer the file operations. FILE_FLAG_NO_BUFFERING | FILE_FLAG_SEQUENTIAL_SCAN when using the windows API file IO makes a massive difference in the performance of the IO. It is a 50-100% increase in performance to use this pair of flags.
2) Since stdio is buffering, it is likely your tests are being affected by windows caching its buffering. I have noticed this affect when I am using smaller sequences (~150 files or less), the second time passing over a sequence it is extremely fast. In your case, the files/file table info are probably still in windows' cache because you wrote the files recently. However, I can't rely on the windows cache to help me out because a 2 gig/200 file sequence is relatively very tiny compared to the real world data this app must process. My real files/file sequences will be in the 100's of gig range. Also I need it to be fast the first time through the sequence, not just after caching it.
3) I don't think this will make a big difference in the relative performance between the two methods, but you are reading in 4 kb blocks, whereas I typically read in multi-megabyte blocks at once.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|