Performance Issue

[greatchap]

Hi Guys,

I have developed a vb.net dll (.net 3.5) which does the following :

- I have data of several stock market companies.
- They are stored in several folders under a main folder.
- These files are random access files.
- There can be around 4000 files in total for example.

I wrote a program which reads a current csv file and stores contents in array. Then I store name and path of companies in folders (which I was talking about). Then I match one company from csv (stored in array) to companies in user's folder (again reading from array). If found I insert data in the file (random access file).

This whole process takes for eg. 2 min my system. A similar program was written years ago by someone in powerbasic which takes slight less time if run one but if run for multiple csv files it takes only a few seconds doing the same thing for other files.

When I do the same thing (by writing code to .net dll) and making a dummy app to run it, no matter how many times I run it takes same time. And is way slower than the earlier one (PB) one I was talking about. But if I paste the same code in my win (.net) app and not use .net dll then it runs a bit faster.
Why??

I know all this may sound a little unfocussed or ambigious. But I have been in a fix for few days. I am trying to improve the performance of this applicaiton which I made. What important things should I keep in mind?

I need to show status so if I pass my form's label reference to .net dll and it updates it does that slow down process a lot. As I mentioned I am trying to process the files and it takes longer in .net. What should I do to get best performance?

Unfortunately due to unexplainable reasons I cannot change the format or use a db. The files which I am reading/writing (random access files) are use by other products so I have to stick to that format.

What I am doing is storing the location of those files in array so that I dont have to search folder again. that is not taking time. But opening reading and writing files is time consuming if i am doing it for few thousand of them. A similar program in powerbasic (developed years ago) runs fast.

Running code thru .net dll is slower than running it thru program. Why?
[Image attached (see attachment) which explains a bit]

Any commens will be welcome.

Thanks a lot,

Cheers,
GR

[HanneSThEGreaT]

Are you making use of ant Threads / BackgroundWorker objects in your program ?

[DataMiser]

Hrd to say without seeing any of the code but are you running in debug mode or are you using a release build without the debugging?

I have not actually compared the two but I would imagion you would see a difference in speed.

I doubt that we will see .net run as fast as a native exe in most cases though.

[greatchap]

Ya I am but its not helping me much. The size of each file (random access) can range from 10 - 200 KB (approx). and the size of the csv file can range from 500KB to 1 MB. Well thats not much right. But there can be around 3000 -4000 of those random access files.

I use one thread which calls my function and it does the following:

- Reads entire CSV file in memory (array)
- Scans directory and stores names of each company (by reading header file). So after entire directory is scanned my array maybe like this -

Symbol: Microsoft
Name : Microsoft Corp
Dat file: Fxx.dat
Startdate & End Date
Folder Path: C:\xyzpath\abc\

So around 4K of those entries.

- Then I start a loop number 1, I read each entry of csv file.
- Loop No 2, I search in another array (which I showed above) and try to determine files where that symbol's data is present.

- If found in x no. of locations I then call a sub which opens random access file, checks for record, matches date and inserts at end or where ever required. Closes file. Updated header file. Closes header file.

-The whole process is repeated again.

[dglienna]

Rather than text files, you should be using a DB to search quickly