-
July 5th, 2011, 03:49 PM
#1
looping txt file line by line.. startegy
I need to read line by line a txt file with approx 2.600.000 lines!!!!!!!!!!!!!!!!
How to with a fast way?
You can know:
All days i recive from other apps this txt file in a server dir:-(
-
July 5th, 2011, 04:36 PM
#2
Re: looping txt file line by line.. startegy
Normally you would use Line Input # within a do while loop to read a file line by line but depending on what you are doing with the file it could be better to read it in much larger blocks or perhaps even all at once in binary mode and then use Split() function to break it down to an array of lines in memory and then loop through the resulting array.
Always use [code][/code] tags when posting code.
-
July 5th, 2011, 04:51 PM
#3
Re: looping txt file line by line.. startegy
Probably want to upgrade to .Net, to utilize PARALLEL processors and tasks. You didn't say if 2.6 was a total size for a month, a week, or an HOUR. That is the main reason to process a file quickly.
Processing time CANNOT take longer than it takes to create a new file.
-
July 5th, 2011, 05:46 PM
#4
Re: looping txt file line by line.. startegy
I got the impression that the file was a end of day type of process.
Good point about processing time vs creation time. If it is a simple batch update that happens once a day then time is not a big deal but still is good to code it to run as quickly as one can in a reasonable amount of time.
Parrel processors and multi thread would work well if doing more than one file at the same time but if simply reading one file line by line very little would be gained if any at all. Most modern PCs are bottlenecked by the drive and network speed rather than that of the CPU.
Always use [code][/code] tags when posting code.
-
July 5th, 2011, 06:04 PM
#5
Re: looping txt file line by line.. startegy
If each thread processed 50K lines, it would be considerable faster...
-
July 6th, 2011, 01:09 PM
#6
Re: looping txt file line by line.. startegy
Originally Posted by dglienna
If each thread processed 50K lines, it would be considerable faster...
If each thread processed 50K lines????
Example... peraphs with Mod statement?
-
July 6th, 2011, 01:58 PM
#7
Re: looping txt file line by line.. startegy
First you need to move to VB.Net if you want to go the multithreaded route.
Always use [code][/code] tags when posting code.
-
July 6th, 2011, 02:00 PM
#8
Re: looping txt file line by line.. startegy
Originally Posted by DataMiser
First you need to move to VB.Net if you want to go the multithreaded route.
VB.Net I need to install wath? (in italian link please):-)
-
July 6th, 2011, 02:06 PM
#9
Re: looping txt file line by line.. startegy
Always use [code][/code] tags when posting code.
-
July 6th, 2011, 02:09 PM
#10
Re: looping txt file line by line.. startegy
Originally Posted by DataMiser
Google VB Express
ok... 2010, 2011 or?
-
July 6th, 2011, 02:29 PM
#11
Re: looping txt file line by line.. startegy
VS2010 has 5 version. Express is FREE. You'll also want to download the SP's for it. You can also visit http://www.CODEPLEX.COM to find open-source projects that work on VS Express.
-
July 6th, 2011, 02:51 PM
#12
Re: looping txt file line by line.. startegy
Originally Posted by dglienna
VS2010 has 5 version. Express is FREE. You'll also want to download the SP's for it. You can also visit http://www.CODEPLEX.COM to find open-source projects that work on VS Express.
Installed! now.... example on fly to resolve my prob?
-
July 6th, 2011, 02:59 PM
#13
Re: looping txt file line by line.. startegy
Still need to know the answers to the original questions regarding the file. Explain what it is, how often it comes in, if it's always in the EXACT same format, etc...
You might not need to use threads at all. VB.Net is a LOT faster than VB6 is (when it comes to processing time).
-
July 6th, 2011, 03:05 PM
#14
Re: looping txt file line by line.. startegy
A lot faster? Depends on what you are doing and how you go about it. In most cases they are about the same when coded properly and in others VB6 may be faster where in others .Net may be faster. In most cases the only real speed differences I have noticed are that the .net IDE is much slower, graphics are faster in .net and of course the option to use multithreaded approach can make your apps faster on multicore pcs.
Always use [code][/code] tags when posting code.
-
July 6th, 2011, 09:39 PM
#15
Re: looping txt file line by line.. startegy
I had a web parser that accepted two parameters. I could only fire off about 20 instances on the Windows 7 box with 3gb RAM, using the compiled VB6 application.
Rewriting it, to combine the two programs into one, that looped into a separate THREAD, guess which was faster? And, it ran 40 instances at once! Still took LESS time.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|