|
-
March 19th, 2008, 07:32 AM
#1
Improve data loading time, pre-caching?
Hi there.
I have a question on improving data loading times. The data sets that I am reading into my program consists of 300-500 files each of size 512 kB.
I have noticed that loading times vary greatly, normal values range from 10s to 30s for loading the ca 200 MB data. However, when reading a data set that has recently been used (not necessarily the previous data set), loading time is around 1s.
- How come?
- Are these files in some kind of cache?
- Since the hard drive cache is 16MB I would not expect several hundred MB of files to be in the disk cache.
- Is there some other kind of cache around that helps improving the reading speed?
- If so, would it be possible to order the OS to do some pre-caching if the next set of data files to be read is known in advance?
And finally, a code question: now I am reading the files using
Code:
BinaryReader binReader = new BinaryReader(File.Open(fileName, FileMode.Open, FileAccess.Read));
byte[] rawBytes = binReader.ReadBytes(512*1024)
- is this the best way to do it or is there any other faster command to read lots of data?
____
Edit: Running VS2005 on a machine with WinXP SP2
Last edited by Cyanide; March 19th, 2008 at 09:34 AM.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|