Threading Problem

[davewolfs]

Hey all,

Quick question.

I'm designing something where I have multiple threads between 100 and 2000 which are constantly processing data.

I'd like the ability to take snap shots of these calculations and aggregate them into a single value i.e. the entire sum of all data calculated by each thread over time x. Basically I'd like to take snapshots of the data currently processed by all threads and sum this information into a single value over a fix amount of time i.e. every 10 seconds.

If I'd like to do something like this in tandem, do you have any recommendations on what type of pattern or method I could use to accomplish this efficiently.

I appreciate your input.

[Arjay]

I'd start by taking another overall look at your design. While 100 threads may not be that bad, 2000 threads is excessive (unless you are running on a heavy multi-proc/multi-core machine). You may look into using a thread pool mechanism.

For the aggregation, consider having each thread average it's data, then use a shared queue to send the data from each thread to a 'aggregation' thread. The 'aggregation' thread just pulls the data out of the queue and aggregates it at a specified interval.