[RESOLVED] Multiple thread question
We have a program that sequentially processes a large number of files (currently about 700 expected to increase to about 1500). The program performs the same processing on each file (and doesn't involve any other file) which is io-bound and not cpu-bound. This process takes several hours and it is normally performed overnight.
I've refactored the program so that the processing for each file is done within its own thread (ie one thread created for the processing of one file). This gives rise to many hundreds of io-bound threads. This refactored program is working with no errors reported and has reduced the total processing time down to about 10 minutes.
Does any guru know of any problems that might arise having this number of threads (700 to 1500) created/running?
All advice is offered in good faith only. You are ultimately responsible for effects of your programs and the integrity of the machines they run on.