Opening very large textfiles. -> Asigning more memory...

[tbscope]

If I open a file wich is larger than approx. 2MB my program crashes, it doesn't respond anymore.

I think it has to do with memory assignment.

If it is the case of memory. How can I expand it?
Anyway...
How can I open very large (approx 50MB) textfiles in VB?

Any help would be appreciated.
Thanks...

[John G Duffy]

There are not many restrictions on processing large files. Two might be physical and virtual memory size related to Windows processing instead of VB.
You might be a little more explicit about how you are processig this file. Opening a file does very little. Reading records does a lot more. Are you trying to read the entire file into memory at onec? If yes how are you attempting to do this?
What kind of error are you getting? Error Message? Simple hang? VB crash? system crash? Hard loop?


John G

[tbscope]

I resolved the crashes but stumbled on a bizare fenomenon...

After reading some texts and books, explaining how to open and manage large files like bitmaps and textfiles, I used some api functions like globalalloc and copymemory. But when I started the program and opened a textfile for the second time, visual basic quits. The errormsg says something like: a read or write error to a memorypointer didn't succeed (Not the exact words...)

Increasing virual memory helps a little. But why should I do that if I have about 250 MB of free physical memory. That is insane...

Also after using the functions globalunlock and globalfree (to get rid of the reserved memory), the taskmanager showed that no memory was actualy released.

The problem wasn't so much on opening a large file, it was more on processing the file.

To get rid of the memoryfaillures I removed the api functions and just opened the text into a richtextbox.

After opening the file i let the program split the text in parts (using selstart,sellength,seltext) and copy them to stringvariables. This is fast.

I have to do that because when trying to do operations on the file (like searching for a word and replace or remove it) the program didn't respond. This means it didn't hang but it will take half an hour to find the word. (The text hasn't got more than 1000000 characters)

By splitting the large file, i can process the small parts after eachother. This will increase the speed.

Now here's the weird part.
I designed an algorithm to split the text in a predefined number of parts. I designed it on paper, put it on the computer and it didn't work.

I let the algorithm run automatically to the end of the textfile. The result was that some words got splitted.

I copied the code to another program and made it so that by pressing a button i run through the algorithm line by line.
The result stunned me: it actually worked.
There where no splitted words...

I made the second program run through the large textfile automatically (like the first prg).
Amazingly, no splitted words.

The code of the 2 programs are EXACTLY the same.
Only the second program does it right and the first messed it up.

(You wouldn't send a rocket to the moon with something like this...)

Anyway, thanks for your reply...

[John G Duffy]

Using the memory management APIs can get a bit ugly if it ain't perfect.

John G

[Cakkie]

The reason your program hangs is probably because you are reading the file as a string. As you might know, strings are slow, very slow. Also, the function you use to read the file can make a big difference. Using Get will increase performance a lot, this function is much much much better than Input(), or Line Input. also, if possible, use a byte array to do the reading, which is faster than string. Of course, this might be a bit hard to process if you actually need to compare two values.
I had a similar problem a few days ago, I used Input and strings, which took me 40 seconds to read a 4 MB file into memory. When using Get and a byte array, it took me 200 ms to read the entire file.

Tom Cannaerts
slisse@planetinternet.be

Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning -- Rich Cook

[tbscope]

Bedankt...

By working with bytes if have to consider ANSI and Unicode formats.
I haven't tested it yet but I think that it will slow down some processes if I have to convert from ANSI to Unicode. Of course Unicode is a lot faster than ANSI but only on NT Computers and not on standard Windows computers.

Anyway, I'm going to try it and see what effects it will have on the performance.

Thanks