looping txt file line by line.. startegy

[sal21]

I need to read line by line a txt file with approx 2.600.000 lines!!!!!!!!!!!!!!!!

How to with a fast way?

You can know:

All days i recive from other apps this txt file in a server dir:-(

[DataMiser]

Normally you would use Line Input # within a do while loop to read a file line by line but depending on what you are doing with the file it could be better to read it in much larger blocks or perhaps even all at once in binary mode and then use Split() function to break it down to an array of lines in memory and then loop through the resulting array.

[dglienna]

Probably want to upgrade to .Net, to utilize PARALLEL processors and tasks. You didn't say if 2.6 was a total size for a month, a week, or an HOUR. That is the main reason to process a file quickly.

Processing time CANNOT take longer than it takes to create a new file.

[DataMiser]

I got the impression that the file was a end of day type of process.

Good point about processing time vs creation time. If it is a simple batch update that happens once a day then time is not a big deal but still is good to code it to run as quickly as one can in a reasonable amount of time.

Parrel processors and multi thread would work well if doing more than one file at the same time but if simply reading one file line by line very little would be gained if any at all. Most modern PCs are bottlenecked by the drive and network speed rather than that of the CPU.

[dglienna]

If each thread processed 50K lines, it would be considerable faster...

[sal21]

If each thread processed 50K lines, it would be considerable faster...

If each thread processed 50K lines????

Example... peraphs with Mod statement?

[DataMiser]

First you need to move to VB.Net if you want to go the multithreaded route.

[sal21]

First you need to move to VB.Net if you want to go the multithreaded route.

VB.Net I need to install wath? (in italian link please):-)

[DataMiser]

Google VB Express

[sal21]

Google VB Express

ok... 2010, 2011 or?

[dglienna]

VS2010 has 5 version. Express is FREE. You'll also want to download the SP's for it. You can also visit http://www.CODEPLEX.COM to find open-source projects that work on VS Express.

[sal21]

VS2010 has 5 version. Express is FREE. You'll also want to download the SP's for it. You can also visit http://www.CODEPLEX.COM to find open-source projects that work on VS Express.

Installed! now.... example on fly to resolve my prob?

[dglienna]

Still need to know the answers to the original questions regarding the file. Explain what it is, how often it comes in, if it's always in the EXACT same format, etc...

You might not need to use threads at all. VB.Net is a LOT faster than VB6 is (when it comes to processing time).

[DataMiser]

A lot faster? Depends on what you are doing and how you go about it. In most cases they are about the same when coded properly and in others VB6 may be faster where in others .Net may be faster. In most cases the only real speed differences I have noticed are that the .net IDE is much slower, graphics are faster in .net and of course the option to use multithreaded approach can make your apps faster on multicore pcs.

[dglienna]

I had a web parser that accepted two parameters. I could only fire off about 20 instances on the Windows 7 box with 3gb RAM, using the compiled VB6 application.

Rewriting it, to combine the two programs into one, that looped into a separate THREAD, guess which was faster? And, it ran 40 instances at once! Still took LESS time.