Reading text files quickly?

[DinoVaught]

Does anyone know the fastest way to read the entire contents of a text file into a single String variable?

[JeffB]

By using FileLen() method, you can achieve this, but this might not be really good for huge file

'fast reading txt files from a "Cakkie" tip 

'strFile is the name of the file
'strString is the string that will contains the file value
Open strFile For Input As #1
    strString = Input(FileLen(strFile), #1)
Close #1

JeffB

[Cimperiali]

Balena solution:

Function FileText (filename$) As String
    Dim handle As Integer
    handle = FreeFile
    Open filename$ For Input As #handle
    FileText = Input$(LOF(handle), handle)
    Close #handle
End Function

Balena second solution

Function FileText(ByVal filename As String) As String
    Dim handle As Integer
     
    ' ensure that the file exists
    If Len(Dir$(filename)) = 0 Then
        Err.Raise 53  ' File not found
    End If
     
    ' open in binary mode
    handle = FreeFile
    Open filename$ For Binary As #handle
    ' read the string and close the file
    FileText = Space$(LOF(handle))
    Get #handle, , FileText
    Close #handle
End Function

Chirs Eastwood solution

Dim iFile As Integer 
On Error Resume Next 
iFile = FreeFile 
GetFileContents = Space(FileLen(FileName)) 
Open FileName For Binary As #iFile 
Get #iFile, , GetFileContents 
Close #iFile 
' 
End Function

Cakkie solution

Dim FFile as Integer
Dim strFile as string 
FFile = FreeFile
Open "yourfile.txt" for input as #FFile
	strFile = input(FileLen("yourfile.txt"),FFile)
Close #FFile

[DinoVaught]

I tested all the samples provided in this thread. I ran them against a 12 meg text file 4 times. The fastest snippet of code was (Balena second solution) coming in at an average of 343 milliseconds . (Chirs Eastwood solution) was pretty quick too. It averaged 355.5 milliseconds for the 4 executions. I think opening them with (For Binary) has something to do with them being so quick??

These times are about 200 milliseconds faster than the API calls GetFileSize, ReadFile, CreateFile and CloseHandle I’ve been using to read text files. Does anyone know how to use these API calls to get a faster read time than VB’s Open statement?

BTW, these approaches are noticeable quicker than the File Systems Object / TextStream’s ReadAll method

[Cimperiali]

(maybe he said it in another forum) about Api being slower for a matter of initialization (but I could have misunterstood the all, and may be ClearCode, TheMasterOfApi said nothing about). In any case, Fso is an object with properties and methods, thus shurely slower - but that handles many things for you, making your life easier.
It would be nice to know also results you get about average time of Cakkie method and Fso and Api...

Cesare

[JeffB]

Yes, the first time you use an API, it is loaded into memory, thus taking some time. I believe once it's loaded, it is much quicker. You can see that behaviour on WinNt. Just look at the TaskManager and the memory usage of your application, when you'll hit command button that use API, you'll see memory usage increase (and keep higher after the code is executed), maybe you could try TWO times in a row and look for the time it takes the second time

JeffB

[DinoVaught]

I use the GetTickCount API to determine how fast certain sections of code run. From my understanding GetTickCount is refreshed every 16 cycles of the PC’s internal clock. Or something like that. Anyway, I believe it to be accurate or close enough for me. I’m not launching satellites or anything. Anyone know a more accurate way of timing how fast code executes?

Sample below times how long it takes to iterate a loop 500,000 times. Remove the DoEvents from the loop to see the difference.

Option Explicit
Private Declare Function GetTickCount Lib "kernel32.dll" () As Long

Private Sub Form_Load()
Dim lSpeedTestCounter As Long
    
    SpeedTest True
    ' Code to test the speed of. . .
    Do Until lSpeedTestCounter = 500000
        lSpeedTestCounter = lSpeedTestCounter + 1
        DoEvents
    Loop
    SpeedTest False

End Sub


Private Sub SpeedTest(bTimerStart As Boolean)
Static iStartCount As Long
Dim iStopCount As Long

    On Error Resume Next
    If bTimerStart Then
        iStartCount = GetTickCount()
        Exit Sub
    Else
        iStopCount = GetTickCount()
    End If

    MsgBox "Code executed in" & Str$(iStopCount - iStartCount) & " milliseconds"

End Sub

[JeffB]

You can also use TIMER:

Private Sub Command1_Click()
    Dim lSpeedTestCounter As Long
    Dim sngStartTime As Single
    Dim sngStopTime As Single
     
    sngStartTime = Timer
    Do Until lSpeedTestCounter = 5000
        lSpeedTestCounter = lSpeedTestCounter + 1
    Loop
    sngStopTime = Timer
     
    MsgBox "Duration (s):" & sngStopTime - sngStartTime
End Sub

GetTickCount is probably more accurate.

Calling a sub is time consuming, so that's why you must use GetTickCount as soon as possible, in your case, if you want to be more accurate (or if you want to launch a satellite), you should use the GetTickCount API just before the DO and just after the LOOP (like the way I use Timer), this prevent the call of a sub, the declaration of some variable and the evaluation of the parameters

JeffB

[Cimperiali]

..and I am sure you know launching it twice may give
different results, as CPU may be differetly free...
In any case, you did the test four times for each method suggested, and the average should be a good indicator.
I am really lazy, now, but maybe in next days will do the test
myself to learn about times of various methods.
From now on, and thanks to you, I will remember binary is faster

Have happy coding,

Cesare Imperiali

[Cakkie]

Originally posted by JeffB
if you want to be more accurate (or if you want to launch a satellite), you should use the GetTickCount API just before the DO and just after the LOOP

If you were to launch a sattelite, you wouldn't be using VB would you?

Anyway, GetTickCount and Timer have both about the same accuracy, nl somewhere around 10ms. However, I've written a solution using high frequency timers, which had an accuracy of approx 0.0078 ms, which is more than 1000 times as acurate. Try beating that .
I've attached a class that implements that. It has a few minor problems when the counter exceeds it's maximum, but it does the job really good for small measurements.

[Andrex]

I need to read (and then write) line by line using different variables, how can I do this?

[Thread1]

Files opened in binary are really fast!

Just a simple reminder from the audience

[yolip]

Dear all,

I tried the above code of Balena second solution, but VB prompts error... said
Run-time error '458'
Variable uses in Automation type not supported in VB

the program stuck on this syntax:

Get #handle, , FileText



please give me a hand..i am new to it...


thanks,
Yolip

[cjard]

Originally posted by DinoVaught
I tested all the samples provided in this thread. I ran them against a 12 meg text file 4 times. The fastest snippet of code was (Balena second solution) coming in at an average of 343 milliseconds . (Chirs Eastwood solution) was pretty quick too. It averaged 355.5 milliseconds for the 4 executions. I think opening them with (For Binary) has something to do with them being so quick??

These times are about 200 milliseconds faster than the API calls GetFileSize, ReadFile, CreateFile and CloseHandle I’ve been using to read text files. Does anyone know how to use these API calls to get a faster read time than VB’s Open statement?

BTW, these approaches are noticeable quicker than the File Systems Object / TextStream’s ReadAll method

Your performance test may be near meaningless, because 12 megabytes in 343 milliseconds is approximately 35 megs/second - which would be near the maximum transfer rate of your hard disk, depnding on what mood it was in, what temperature the room was, what else was being loaded or paged etc..

i.e., i would say that the figures are fairly artificial, especially for repeated reads of the same file, bcause it is than partly an issue for how much the operating system has biffered and where etc etc..

[DinoVaught]

I have to disagree when you say my tests are meaningless. I can still see consistently faster reads using the method described by Balena. I can see it on my PC / the folks that use the applications develop get faster results too. Give it a try. Balena’s method, simple put, is the fastest. I posted this msg back in 2002. You must be digging deep. Ha ha