Multi-threading Problem

[kenjo]

Hi there,

I am executing a call to a function contained within a 3rd Party .dll and it is running extremely slowly when executed by a thread.

Calling the function using a single thread works fine, executing in 20ms as expected. When I create and execute two threads, the time for the function to execute is about 40-60 ms for each thread. It varies greatly but never close to 20ms.


Basically I have the following:

ThirdPartObject obj1 = new ThirdPartObject();
ThirdPartObject obj2 = new ThirdPartObject();

public void Run()
{

ThreadStart worker1 = delegate { RunObj1(); };
new Thread(worker1).Start();


ThreadStart worker2 = delegate { RunObj2(); };
new Thread(worker2).Start();

}

public void RunObj1()
{
this.obj1.Execute();
}


public void RunObj2()
{
this.obj2.Execute();
}


if I replace the code in RunObj1 and RunObj2 with say a for-loop which adds numbers, it behaves as expected and both RunObj1 and RunObj2 run in parallel.

Any ideas?

Thanks.

[DeepT]

Without knowing what the function does, you can't really know. However, if we assume each function takes 20ms then what you are experiencing makes sense.

Your application is allocated a 20ms time slice from the CPU for each cycle. If your application has two threads which consume all the time they can, they will take twice as long to run. For example:

Cycle 1: 20ms allocated
Main App thread consumes 2ms
Thread 1 consumes 9ms
Thread 2 consumes 9ms.

Cycle 2:
Main App thread consumes 2ms
Thread 1 consumes 9ms
Thread 2 consumes 9ms.

At this point you are 40ms into the operation but each worker thread still has only gotten 18ms of CPU time to complete a 20ms task.

You see, 40ms of work is always going to be 40ms of work no matter how many threads are involved. Your application will always get 20ms chunks of CPU time which will be shared by all threads. The only thing you can hope for is that other applications do not use much or any of their CPU time so that they yield the CPU to the OS. This will result in your program getting 20ms CPU blocks more frequently.

Anyway, the point I am trying to make is that threads != free work. The CPU cost is still paid in full, it just spread out over more time slices. In a multi-core environment, different threads may end up on different cores, which simply allows the OS to load balance the CPU's more effectively.

The only way you will ever make a function that takes 20ms 'faster' is by improving the function or improving the hardware the computer runs on.

[Mutant_Fruit]

Are you on a multi-core system?

[kenjo]

Are you on a multi-core system?

Sorry, I forgot to mention that I'm using a Quad Core.

Replacing the 3rd party library function with my own function which executes a for loop that takes 20 ms works fine. Both loops execute in 20ms and the two threads executing within 20ms.

So i'm wondering if it's a problem with the third party library, which is performing a very CPU intensive operation.

Again, executing the function with one thread only works fine, it's when I run a second thread I get the crazy drop in performance.


Thanks.

[Mutant_Fruit]

My guess is that the third party lib performs locking so that what you think are two completely independent objects aren't actually independent. Can you not contact the creator of the third party lib or read it's docs to see if anything is mentioned? They could have written a thread-safe library, which could easily have the performance problems you've mentioned.

[kenjo]

Very poor documentation with the third party library and their support team have basically said it's a .net threading issue and not their problem. Worse still we pay these guys an annual support fee!

What you mentioned probably makes sense. I'll contact them again armed with a bit more information.

[Mutant_Fruit]

I'd be interested in hearing their reply. The other possibility is you are timing the calls incorrectly. Are you checking how long the actual method takes to complete like this:

public void RunObj2()
{
StartTimer();
this.obj2.Execute();
Console.WriteLine(timer.ElapsedTime);
}

Or are you measuring the time between calling thread.Start() and the thread exiting. If so, there are quite a few areas where delays could happen which would inflate the execution time.

[boudino]

Isn't the 3rd party dll a COM component? Maybe setting thread apartment to single could help:

[STAThread]
public static void Main()
{
...
}

[DeepT]

I have been doing some digging and I am not sure the following is correct, but it seems to be right. I can't find any detailed documentation on exactly how windows time slices an application and it's threads with respect to multiple cores.

The first thing I found out is that Vista is much more 'fair' to threads then previous NT kernels. Pre-Vista, threads could actually starve for CPU time, where as Vista guarantees each thread will get it's fair share of CPU time.

It would seem that each core has it's pool of time slices. Each thread in Vista is allocated a full time-slice. When a thread is ready to run, it gets the next available time-slice which can be on any core. So it would seem that an application with 3 threads could have all the work done simultaneously. However, that is not a guarantee because it is up to the discretion of Vista. It may assign time slices all on the same core or split them out on 2 of 4 cores, etc...

The end result is, 4 threads that take 20ms on a quad core might take 20ms to run or 80ms to run, or something in between. I suspect that there is some overhead for switching a thread from one core to another so it may tend to stack them on a single core.


There is a real-world example of this that I just thought of. There is a game that came out several months ago. It is called Supreme Commander. It featured multi-core support. The heavy-weight work was threaded (rendering) so it could take advantage of multiple cores. There was a problem though. Vista (and XP) was not very wise in it's choice of spreading the work out. People would notice that one core was maxed out, but others were nearly asleep.

Some programmer made a simple watchdog application that looked at each core's usage and would re-balance the threads every 10 seconds. This made an immediate, huge gain in the performance of Supreme Commander.

The moral of the story is: Just because you have multiple cores doesn't mean windows will use them wisely. Microsoft really needs to improve their task scheduler to keep things more balanced. The utility made for Supreme Commander should have been unnecessary.

[Mutant_Fruit]

It would seem that each core has it's pool of time slices. Each thread in Vista is allocated a full time-slice. When a thread is ready to run, it gets the next available time-slice which can be on any core. So it would seem that an application with 3 threads could have all the work done simultaneously. However, that is not a guarantee because it is up to the discretion of Vista. It may assign time slices all on the same core or split them out on 2 of 4 cores, etc...

True, but i doubt that's the issue here. I think you can figure out the operating system thread ID's that are used by your managed app, so that might allow you to check if you are using more than one physical thread. The best way of testing might be to run 'this.obj2.Execute();' in a while loop and print how long each iteration takes inside each thread when you are running 1, 2, 3 and 4 threads. That'd definitely help.

If it's 20ms with 1 thread and then >> 20ms with more than 1 thread, it's a library issue for definite. Running in a while loop removes thread creation overhead and whatnot.

Some programmer made a simple watchdog application that looked at each core's usage and would re-balance the threads every 10 seconds. This made an immediate, huge gain in the performance of Supreme Commander.

That's kinda interesting. Do you have a link to any threads about it? I couldn't find anything myself.

[DeepT]

Link

After reading that, I was wondering how to write my own monitoring tool to watch the CPU load over a long period of time. I can't find anything in the windows API for it, either C++ or C#. There are functions that will get the generic CPU time for a thread of process, but there doesn't seem to be a way to tell which CPU they used time on.


Ideally I would like to create a tool which can know exactly how busy each core is and when there is a significant imbalance, to be able to see which threads are consuming the most time on each core and then change their affinity. It would be nice to create an all-purpose core management utility which kept the workload as even as possible. Does anyone know of the API calls that will give me these two pieces of information? That would be, how much CPU time a Thread/Proc consumes on which CPU (its very important to know which CPU it was using) and then to figure out how much work each CPU is doing.

Even without the ability to change the CPU affinity of something, it would be interesting to find out how good or bad Window's task scheduler actually is.

[Allan Olesen]

There was a problem though. Vista (and XP) was not very wise in it's choice of spreading the work out. People would notice that one core was maxed out, but others were nearly asleep.

This description may not be entirely fair to Windows. There were some indications that the problem was caused by Supreme Commander doing a bad job of trying to force its own distribution of threads to cores instead of relying on Windows to do the job.

Before MadBoris made his utility, some players tricked the game by starting it on one core (which probably turned the game's own thread distribution off so Windows could do the distribution without intervention from the game) and then setting affinity to all cores afterwards. This resulted in a more even distribution of threads