Reading text files quickly?
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 1 of 2 12 LastLast
Results 1 to 15 of 17

Thread: Reading text files quickly?

  1. #1
    Join Date
    Nov 2002
    Posts
    278

    Reading text files quickly?

    Does anyone know the fastest way to read the entire contents of a text file into a single String variable?

  2. #2
    Join Date
    Sep 2001
    Location
    Québec, Canada
    Posts
    1,923

    fast reading txt files from a "Cakkie" tip

    By using FileLen() method, you can achieve this, but this might not be really good for huge file

    Code:
    'fast reading txt files from a "Cakkie" tip 
    
    'strFile is the name of the file
    'strString is the string that will contains the file value
    Open strFile For Input As #1
        strString = Input(FileLen(strFile), #1)
    Close #1
    JeffB
    CodeGuru VB FAQ Visual Basic Frequently Asked Questions
    VB Code color Tool to color your VB code on CodeGuru
    Before you post Importants informations to know before posting

  3. #3
    Join Date
    Jul 2000
    Location
    Milano, Italy
    Posts
    7,726

    Oh, yes...

    Balena solution:
    Code:
    Function FileText (filename$) As String
        Dim handle As Integer
        handle = FreeFile
        Open filename$ For Input As #handle
        FileText = Input$(LOF(handle), handle)
        Close #handle
    End Function
    Balena second solution
    Code:
    Function FileText(ByVal filename As String) As String
        Dim handle As Integer
         
        ' ensure that the file exists
        If Len(Dir$(filename)) = 0 Then
            Err.Raise 53  ' File not found
        End If
         
        ' open in binary mode
        handle = FreeFile
        Open filename$ For Binary As #handle
        ' read the string and close the file
        FileText = Space$(LOF(handle))
        Get #handle, , FileText
        Close #handle
    End Function
    Chirs Eastwood solution
    Code:
    Dim iFile As Integer 
    On Error Resume Next 
    iFile = FreeFile 
    GetFileContents = Space(FileLen(FileName)) 
    Open FileName For Binary As #iFile 
    Get #iFile, , GetFileContents 
    Close #iFile 
    ' 
    End Function
    Cakkie solution
    Code:
    Dim FFile as Integer
    Dim strFile as string 
    FFile = FreeFile
    Open "yourfile.txt" for input as #FFile
    	strFile = input(FileLen("yourfile.txt"),FFile)
    Close #FFile
    ...at present time, using mainly Net 4.0, Vs 2010



    Special thanks to Lothar "the Great" Haensler, Chris Eastwood , dr_Michael, ClearCode, Iouri and
    all the other wonderful people who made and make Codeguru a great place.
    Come back soon, you Gurus.

  4. #4
    Join Date
    Nov 2002
    Posts
    278
    I tested all the samples provided in this thread. I ran them against a 12 meg text file 4 times. The fastest snippet of code was (Balena second solution) coming in at an average of 343 milliseconds . (Chirs Eastwood solution) was pretty quick too. It averaged 355.5 milliseconds for the 4 executions. I think opening them with (For Binary) has something to do with them being so quick??

    These times are about 200 milliseconds faster than the API calls GetFileSize, ReadFile, CreateFile and CloseHandle I’ve been using to read text files. Does anyone know how to use these API calls to get a faster read time than VB’s Open statement?

    BTW, these approaches are noticeable quicker than the File Systems Object / TextStream’s ReadAll method

  5. #5
    Join Date
    Jul 2000
    Location
    Milano, Italy
    Posts
    7,726

    Wink ClearCode once said something

    (maybe he said it in another forum) about Api being slower for a matter of initialization (but I could have misunterstood the all, and may be ClearCode, TheMasterOfApi said nothing about). In any case, Fso is an object with properties and methods, thus shurely slower - but that handles many things for you, making your life easier.
    It would be nice to know also results you get about average time of Cakkie method and Fso and Api...

    Cesare
    ...at present time, using mainly Net 4.0, Vs 2010



    Special thanks to Lothar "the Great" Haensler, Chris Eastwood , dr_Michael, ClearCode, Iouri and
    all the other wonderful people who made and make Codeguru a great place.
    Come back soon, you Gurus.

  6. #6
    Join Date
    Sep 2001
    Location
    Québec, Canada
    Posts
    1,923
    Yes, the first time you use an API, it is loaded into memory, thus taking some time. I believe once it's loaded, it is much quicker. You can see that behaviour on WinNt. Just look at the TaskManager and the memory usage of your application, when you'll hit command button that use API, you'll see memory usage increase (and keep higher after the code is executed), maybe you could try TWO times in a row and look for the time it takes the second time

    JeffB
    CodeGuru VB FAQ Visual Basic Frequently Asked Questions
    VB Code color Tool to color your VB code on CodeGuru
    Before you post Importants informations to know before posting

  7. #7
    Join Date
    Nov 2002
    Posts
    278
    I use the GetTickCount API to determine how fast certain sections of code run. From my understanding GetTickCount is refreshed every 16 cycles of the PC’s internal clock. Or something like that. Anyway, I believe it to be accurate or close enough for me. I’m not launching satellites or anything. Anyone know a more accurate way of timing how fast code executes?

    Sample below times how long it takes to iterate a loop 500,000 times. Remove the DoEvents from the loop to see the difference.

    Code:
    Option Explicit
    Private Declare Function GetTickCount Lib "kernel32.dll" () As Long
    
    Private Sub Form_Load()
    Dim lSpeedTestCounter As Long
        
        SpeedTest True
        ' Code to test the speed of. . .
        Do Until lSpeedTestCounter = 500000
            lSpeedTestCounter = lSpeedTestCounter + 1
            DoEvents
        Loop
        SpeedTest False
    
    End Sub
    
    
    Private Sub SpeedTest(bTimerStart As Boolean)
    Static iStartCount As Long
    Dim iStopCount As Long
    
        On Error Resume Next
        If bTimerStart Then
            iStartCount = GetTickCount()
            Exit Sub
        Else
            iStopCount = GetTickCount()
        End If
    
        MsgBox "Code executed in" & Str$(iStopCount - iStartCount) & " milliseconds"
    
    End Sub

  8. #8
    Join Date
    Sep 2001
    Location
    Québec, Canada
    Posts
    1,923
    You can also use TIMER:

    Code:
    Private Sub Command1_Click()
        Dim lSpeedTestCounter As Long
        Dim sngStartTime As Single
        Dim sngStopTime As Single
         
        sngStartTime = Timer
        Do Until lSpeedTestCounter = 5000
            lSpeedTestCounter = lSpeedTestCounter + 1
        Loop
        sngStopTime = Timer
         
        MsgBox "Duration (s):" & sngStopTime - sngStartTime
    End Sub
    GetTickCount is probably more accurate.

    Calling a sub is time consuming, so that's why you must use GetTickCount as soon as possible, in your case, if you want to be more accurate (or if you want to launch a satellite), you should use the GetTickCount API just before the DO and just after the LOOP (like the way I use Timer), this prevent the call of a sub, the declaration of some variable and the evaluation of the parameters

    JeffB
    CodeGuru VB FAQ Visual Basic Frequently Asked Questions
    VB Code color Tool to color your VB code on CodeGuru
    Before you post Importants informations to know before posting

  9. #9
    Join Date
    Jul 2000
    Location
    Milano, Italy
    Posts
    7,726

    Cool Nice...

    ..and I am sure you know launching it twice may give
    different results, as CPU may be differetly free...
    In any case, you did the test four times for each method suggested, and the average should be a good indicator.
    I am really lazy, now, but maybe in next days will do the test
    myself to learn about times of various methods.
    From now on, and thanks to you, I will remember binary is faster

    Have happy coding,

    Cesare Imperiali
    ...at present time, using mainly Net 4.0, Vs 2010



    Special thanks to Lothar "the Great" Haensler, Chris Eastwood , dr_Michael, ClearCode, Iouri and
    all the other wonderful people who made and make Codeguru a great place.
    Come back soon, you Gurus.

  10. #10
    Join Date
    Jan 2000
    Location
    Olen, Belgium
    Posts
    2,477
    Originally posted by JeffB
    if you want to be more accurate (or if you want to launch a satellite), you should use the GetTickCount API just before the DO and just after the LOOP
    If you were to launch a sattelite, you wouldn't be using VB would you?

    Anyway, GetTickCount and Timer have both about the same accuracy, nl somewhere around 10ms. However, I've written a solution using high frequency timers, which had an accuracy of approx 0.0078 ms, which is more than 1000 times as acurate. Try beating that .
    I've attached a class that implements that. It has a few minor problems when the counter exceeds it's maximum, but it does the job really good for small measurements.
    Attached Files Attached Files
    Tom Cannaerts
    email: cakkie@cakkie.be.remove.this.bit
    www.tom.be (dutch site)

  11. #11
    Join Date
    Feb 2003
    Location
    AR
    Posts
    228
    I need to read (and then write) line by line using different variables, how can I do this?

  12. #12
    Join Date
    Jan 2003
    Location
    7,107 Islands
    Posts
    2,487
    Files opened in binary are really fast!

    Just a simple reminder from the audience
    Busy

  13. #13
    Join Date
    Oct 2002
    Location
    Australia
    Posts
    207

    Runtime error '458'

    Dear all,

    I tried the above code of Balena second solution, but VB prompts error... said
    Run-time error '458'
    Variable uses in Automation type not supported in VB

    the program stuck on this syntax:

    Get #handle, , FileText



    please give me a hand..i am new to it...


    thanks,
    Yolip

  14. #14
    Join Date
    Oct 2003
    Location
    .NET2.0 / VS2005 Developer
    Posts
    7,104
    Originally posted by DinoVaught
    I tested all the samples provided in this thread. I ran them against a 12 meg text file 4 times. The fastest snippet of code was (Balena second solution) coming in at an average of 343 milliseconds . (Chirs Eastwood solution) was pretty quick too. It averaged 355.5 milliseconds for the 4 executions. I think opening them with (For Binary) has something to do with them being so quick??

    These times are about 200 milliseconds faster than the API calls GetFileSize, ReadFile, CreateFile and CloseHandle I’ve been using to read text files. Does anyone know how to use these API calls to get a faster read time than VB’s Open statement?

    BTW, these approaches are noticeable quicker than the File Systems Object / TextStream’s ReadAll method
    Your performance test may be near meaningless, because 12 megabytes in 343 milliseconds is approximately 35 megs/second - which would be near the maximum transfer rate of your hard disk, depnding on what mood it was in, what temperature the room was, what else was being loaded or paged etc..

    i.e., i would say that the figures are fairly artificial, especially for repeated reads of the same file, bcause it is than partly an issue for how much the operating system has biffered and where etc etc..
    "it's a fax from your dog, Mr Dansworth. It looks like your cat" - Gary Larson...DW1: Data Walkthroughs 1.1...DW2: Data Walkthroughs 2.0...DDS: The DataSet Designer Surface...ANO: ADO.NET2 Orientation...DAN: Deeper ADO.NET...DNU...PQ

  15. #15
    Join Date
    Nov 2002
    Posts
    278
    I have to disagree when you say my tests are meaningless. I can still see consistently faster reads using the method described by Balena. I can see it on my PC / the folks that use the applications develop get faster results too. Give it a try. Balena’s method, simple put, is the fastest. I posted this msg back in 2002. You must be digging deep. Ha ha

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Azure Activities Information Page

Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center