Mutex used in Thread calling recv (TCP connection)

[aparaske]

Hi,

I was wondering if someone could help me figure this out as I am quite stuck.

I have threads called NetworkConnection that connect to servers and constantly receive data via the recv function. Once they receive a message, they store it in their buffers. There's another thread called NetworkController which manages all the NetworkConnection threads and processes messages whenever there's something in their buffers.

I would like to have the NetworkController thread use WaitForMultipleObjects where I pass an array of all the Mutex handles of the NetworkConnection threads. NetworkConnection threads should only release their Mutex when there's something in their buffer.

My problem is that once a NetworkConnection thread is in the recv funtion and the Mutex is owned by NetworkController because there's still messages in the buffer, once the NetworkController processes all messages in the buffer, NetworkConnection won't be able to lock the Mutex because it's stuck in the recv function until there's new data. As a result, the NetworkController will end up constantly polling the buffer while there's no more data to process.

If a Mutex could be used in a way where it doesn't matter which thread locks it or releases it, it would solve my problem (ie NetworkController locks it whenever there's no more data in the buffer and NetworkConnection releases it whenever there's new data in the buffer), however, as I understand that's not possible. A thread should only lock a Mutex once and always be the one to release it.

Does any of you have any ideas how I can go around this problem?

Many thanks,

Alex

[D_Drmmr]

What you need is a condition variable. I find this quite difficult to get right using the synchronization primitives in the Win API. If you are able to use the Boost Thread library, I can certainly recommend it. Also, this will be adopted in the upcoming C++0x standard. No support in VS2010 for this yet, unfortunately.

[Arjay]

Can you post the code inside one of the NetworkConnection threads?

[aparaske]

@ Arjay: Unfortunately, NetworkConnection is just a Base class and part of a major project; is there something specific that you'd want to see?

@D_Drmmr: There's something that confuses me. How exactly do condition variables work?

Here's the example from the link you posted:

boost::unique_lock<boost::mutex> lock(mut);
while(!data_ready)
{
        cond.wait(lock);
}

What I don't understand is how it gets signalled that data_ready becomes true and when it does, will it immediately come out of sleep? Also, to quote "When the thread is woken from the wait, then it checks to see if the appropriate condition is now true, and continues if so." How long does each wait last?

Thanks,

Alex

[aparaske]

Oh, also, NetworkController manages multiple NetworkConnection threads so would it be possible to use condition variables in a similar way as WaitForMultipleObjects?

[JohnW@Wessex]

I've solved something a little similar to this. One 'mutex free' way to approach it is to construct a fixed length lock-free FIFO (using atomic flags and memory barriers).

When the producer thread receives a block of data it adds it to the FIFO and sets a 'Data Available' event.

The consumer thread wakes up on receiving this event and reads as many blocks as it can from the FIFO. The consumer would wait on as many events as there were producers.

Each producer would have its own FIFO.
A little calculation and empirical testing would be required to determine the best size of the FIFO.

[aparaske]

I've solved something a little similar to this. One 'mutex free' way to approach it is to construct a fixed length lock-free FIFO (using atomic flags and memory barriers).

When the producer thread receives a block of data it adds it to the FIFO and sets a 'Data Available' event.

The consumer thread wakes up on receiving this event and reads as many blocks as it can from the FIFO. The consumer would wait on as many events as there were producers.

Each producer would have its own FIFO.
A little calculation and empirical testing would be required to determine the best size of the FIFO.

In my case, I use an stl list since I would like the queue to grow as much as needed and protect it using Critical Sections (critical sections work well with condition variables from what I understand). I have one list for each NetworkConnection as well.

How did you get your consumer thread to manage multiple producer threads?

Regards,

Alex

[JohnW@Wessex]

In my case, I use a vector since I would like the queue to grow as much as needed

What I had was a FIFO of vectors. Each element in the FIFO was a variable sized packet of data. You have to make sure that the FIFO is big enough to buffer any latency in the consumer responding to the event.

How did you get your consumer thread to manage multiple producer threads?

It didn't really manage them. All of the threads were created at start-up.
I can't use mutexes in my code (tight timing constraints) so I tend to use lock-free buffers, state machines and events to coordinate the various threads.

[D_Drmmr]

What I don't understand is how it gets signalled that data_ready becomes true and when it does, will it immediately come out of sleep?

The wait member function returns after cond.notify_one(); is called from another thread. The thread that was waiting then checks again whether the condition is satisfied (in the example when data_ready is true). So the condition variable only does the signaling, you have to check the condition yourself, but it's easy to do that in a way that it is protected by a mutex.

Instead of having a bool as a condition, you can also check if a list of jobs is non-empty e.g.

Also, to quote "When the thread is woken from the wait, then it checks to see if the appropriate condition is now true, and continues if so." How long does each wait last?

In the example, the thread waits until cond.notify_one() gets called. If it doesn't get called, the thread will wait until the process ends (with a 'wild' thread). You can also let the wait time out after a certain duration or at a specific system time.

[aparaske]

Hi D_Drmmr,

I tried using Windows CONDITION_VARIABLE that seemed pretty simple enough and worked well with my CRITICAL_SECTION code, only to realise that it's supported by Vista/7/2008.

I'll use boost, however, I don't want to include all of boost in my project. Is there a way to figure out the dependencies of the condition_variable class and just include that?

Regards,

Alex

[Lindley]

Since Boost Thread is a header-only library, your compiler will automatically figure out which portions of Boost it actually needs.

If you're concerned about distribution size, then there is a tool out there which can generate subsets of the Boost distribution given only what you need, but I don't recall its name right now.