Multiple-readers, single-writer synchronization lock between processes

There's a well-known algorithm that employs readers/writer lock synchronization between threads of a single process on a Windows platform using pure WinAPIs/C++:
http://www.glennslayden.com/code/win...er-writer-lock

In my case I need to do this between several processes, i.e. the writer is in one process, and readers are in other processes. Any idea how to do that?

PS. I need this for an already developed project, so I can't use anything other than C++/MFC or pure WinAPIs. In other words I can't use Boost or the like extensions.

Named mutex of a name in global namespace (single reader/writer access). Or a dedicated read/write broker.

Oh, actually named semaphore could be used for counting readers. And named event to indicate writing in progress.

Quote:

---

Originally Posted by Igor Vartanov

Oh, actually named semaphore could be used for counting readers. And named event to indicate writing in progress.

---

Thanks, Igor. Replacing the critical section with global named mutex should work. The issue is in the counter of readers. I actually thought to use named semaphores for it too, but couldn't come up with a working code. The issue is that semaphores count down and this algorithm needs to count up & down.

So can I ask you to demonstrate what you meant in code?

The idea actually was for reader to inverse the logic behind semaphore object and use ReleaseSemaphore(by 1) to indicate read operation start and WaitForSingleObject to indicate read completion. The semaphore should be created with maximum allowed count but initial count is zero. Of course, reader should catch the mutex and immediately release it after incrementing semaphore. Symmetrically, writer should not grab the mutex until semaphore becomes non-signalled.

One potential danger thing to be confirmed that died reader never blocks the whole mechanism.

Quote:

---

Originally Posted by Igor Vartanov

The idea actually was for reader to inverse the logic behind semaphore object and use ReleaseSemaphore(by 1) to indicate read operation start and WaitForSingleObject to indicate read completion. The semaphore should be created with maximum allowed count but initial count is zero. Of course, reader should catch the mutex and immediately release it after incrementing semaphore. Symmetrically, writer should not grab the mutex until semaphore becomes non-signalled.

One potential danger thing to be confirmed that died reader never blocks the whole mechanism.

---

But how would you know that there're no readers reading the data?

No readers is indicated by semaphore non-signalling state.

Let's put it into code:

Code:

---

//For simplicity I'm writing it for a single process
class MultiReaderSingleWriter
{
private:
        CRITICAL_SECTION m_csWrite;
        CRITICAL_SECTION m_csReaderCount;
        //long m_cReaders;
        HANDLE m_semCounter;
        HANDLE m_hevReadersCleared;

public:
        MultiReaderSingleWriter()
        {
                m_semCounter = CreateSemaphore(NULL, 0, 1000, NULL);
                InitializeCriticalSection(&m_csWrite);
                InitializeCriticalSection(&m_csReaderCount);
                m_hevReadersCleared = CreateEvent(NULL,TRUE,TRUE,NULL);
        }

        ~MultiReaderSingleWriter()
        {
                WaitForSingleObject(m_hevReadersCleared,INFINITE);
                CloseHandle(m_semCounter);
                CloseHandle(m_hevReadersCleared);
                DeleteCriticalSection(&m_csWrite);
                DeleteCriticalSection(&m_csReaderCount);
        }


        void EnterReader(void)
        {
                EnterCriticalSection(&m_csWrite);
                EnterCriticalSection(&m_csReaderCount);
                //if (++m_cReaders == 1)
                //        ResetEvent(m_hevReadersCleared);

                ReleaseSemaphore(m_semCounter);  //Can we call it here with the count being 0?
                //So how do you check the semaphore count here?
               
                LeaveCriticalSection(&m_csReaderCount);
                LeaveCriticalSection(&m_csWrite);
        }

        void LeaveReader(void)
        {
                EnterCriticalSection(&m_csReaderCount);
                //if (--m_cReaders == 0)
                //        SetEvent(m_hevReadersCleared);

                WaitForSingleObject(m_semCounter, 0);
                //Same problem here??

                LeaveCriticalSection(&m_csReaderCount);
        }

        void EnterWriter(void)
        {
                EnterCriticalSection(&m_csWrite);
                WaitForSingleObject(m_hevReadersCleared,INFINITE);
        }

        void LeaveWriter(void)
        {
                LeaveCriticalSection(&m_csWrite);
        }
};

---

It doesn't seem to work. I don't know that semaphores can be used as counters? I might be wrong. If so, correct me.

The code below is just an idea that needs to be proved, or disproved, or corrected. And no, I never did anything like this before. :)

Code:

---

#include <windows.h>
#include <crtdbg.h>

class MultiReaderSingleWriter
{
        HANDLE m_hSemRead;  // counts active readers
        HANDLE m_hEvtNoRead;  // prevents read operation

public:
        MultiReaderSingleWriter()
        {
                m_hSemRead = CreateSemaphore(NULL, 0, 1000, NULL);
                m_hEvtNoRead = CreateEvent(NULL, TRUE, FALSE, NULL);
        }

        ~MultiReaderSingleWriter()
        {
                CloseHandle(m_hSemRead);
                CloseHandle(m_hEvtNoRead);
        }

        BOOL EnterRead()
        {
                DWORD waitRes = WaitForSingleObject(m_hEvtNoRead, INFINITE); // blocks until write completion
                _ASSERT(WAIT_OBJECT_0 == waitRes);       
                LONG count = 0;
                BOOL res = ReleaseSemaphore(m_hSemRead, 1, &count); // increments semaphore counter
                // here you can log count to see internal semaphore counter before you incremented it
                _ASSERT(res);
                return res;
        }

        BOOL LeaveRead()
        {
                DWORD waitRes = WaitForSingleObject(m_hSemRead, 0); // decrement semaphore counter
                BOOL res = WAIT_TIMEOUT == waitRes;  // here not sure about the exact retcode :)
                _ASSERT(res);
                return res;
        }

        BOOL EnterWrite()
        {
                BOOL res = SetEvent(m_hEvtNoRead); // prevents new reads
                _ASSERT(res);
                DWORD waitRes = 0;
                do
                {
                        // loop until last reader gone
                        waitRes = WaitForSingleObject(m_hSemRead, 0);
                        if (WAIT_OBJECT_0 == waitRes)
                                Sleep(10);
                        else if (WAIT_TIMEOUT == waitRes)
                                return TRUE;
                        else
                                return FALSE;
                } while (TRUE);

                return FALSE;
        }

        BOOL LeaveWrite()
        {
                BOOL res = ResetEvent(m_hEvtNoRead); // release waiting readers
                _ASSERT(res);
                return res;
        }
};

---

Thanks, Igor.

Quote:

---

Code:

---

DWORD waitRes = WaitForSingleObject(m_hSemRead, 0); // decrement semaphore counter
BOOL res = WAIT_TIMEOUT == waitRes;  // here not sure about the exact retcode :)

---

---

Yes, I'm not sure about this one either. That's what I meant in my previous post that a semaphore object was never intended as just a counter. Although I do appreciate your "bold" approach to it.

Quote:

---

One potential danger thing to be confirmed that died reader never blocks the whole mechanism.

---

Yes, that becomes an issue in case I put your code into several processes. For instance, if one reader crashes and doesn't decrement the semaphore, it will indefinitely deadlock the writer.

Also this, I guess is just nitpicking:

Quote:

---

Code:

---

// loop until last reader gone
waitRes = WaitForSingleObject(m_hSemRead, 0);
if (WAIT_OBJECT_0 == waitRes)
        Sleep(10);
else if (WAIT_TIMEOUT == waitRes)
        return TRUE;
else
        return FALSE;

---

---

Wouldn't it be better to do:

Code:

---

waitRes = WaitForSingleObject(m_hSemRead, 10);

---

and drop Sleep(10); to save those 10ms?

Actually, the initial code is too dirty. Apparently, 2 AM isn't the best hour for writing clean code. :D

Attached please find the reviewed and debugged variant. (.sln file is not a normal solution but just a debugging session)

The change: writer should release semaphore too, like reader does, otherwise wait distorts the semaphore counter. BTW, this way it peeks into the semaphore counter as well. ;)

Code:

---

    BOOL EnterWrite()
    {
        PutLog(CString(TEXT("+ ENTER + ")) + TEXT(__FUNCTION__));
        BOOL res = ResetEvent(m_hEvtNoRead); // prevents new reads
        _ASSERT(res);

        DWORD waitRes = 0;
        do
        {
            // loop until last reader gone
            LONG count = 0;
            res = ReleaseSemaphore(m_hSemRead, 1, &count);
            _ASSERT(res);
            waitRes = WaitForSingleObject(m_hSemRead, 0);
            if (count)
                Sleep(10);
            else if (WAIT_OBJECT_0 == waitRes)
            {
                res = TRUE;
                break;
            }
            else
            {
                res = FALSE;
                break;
            }
        } while (TRUE);

        PutLog(CString(TEXT("- RETURN - ")) + CString(__FUNCTION__) + TEXT(", res = ") + ToString(res));
        return res;
    }

    BOOL LeaveWrite()
    {
        PutLog(CString(TEXT("+ ENTER + ")) + TEXT(__FUNCTION__));
        BOOL res = SetEvent(m_hEvtNoRead); // release waiting readers
        _ASSERT(res);
        PutLog(CString(TEXT("- RETURN - ")) + CString(__FUNCTION__) + TEXT(", res = ") + ToString(res));
        return res;
    }

---

Thanks again. I truly appreciate your unconventional approach to using semaphores.

I reworked your code to use two separate processes (for readers and a writer.) Sorry, I can't write concise solutions like you do, so I opted for a default console app template instead. To test it run the writer first (only one instance is allowed) and then start as many readers as you want. I also removed your log file output and instead redirected it to the screen to achieve faster output (to test synchronization mechanism) and also to give a tester visual clue that threads are running.

I had to remove your CRT I/O file functions in favor of lower-level APIs. Maybe I don't know much about fopen() but it seems to have its own built-in synchronization, which mixed with the results of the synchronization mechanism in this example.

Then, first off, let me correct a bug. When you create m_hEvtNoRead event, it must be set to signaled state. Otherwise the first reader will deadlock without the first writer call.

Then a couple of issues that I ran into (or maybe I should name them as questions):

1. The sleep(10) part of the EnterWrite() method seems to be a bottleneck:

Code:

---

waitRes = WaitForSingleObject(m_hSemRead, 0);
if (count)
        Sleep(10);

---

but if I rewrite it into:

Code:

---

waitRes = WaitForSingleObject(m_hSemRead, 10);
if (count)
        /*Sleep(10)*/;

---

it really speeds things up, but unfortunately I get occasional "read fail". Why do you think would that be?

PS. While typing this reply, I let the "writer" run on the background with Sleep(10) line... And it also gave me "read fail" after about 5-10 minutes of continuous running... There still must be an issue somewhere in the this lock... :(

After a quick overview it seems that the part of the code that changes the semaphore count must be enclosed in its own critical section. Don't you think?

2. The second issue is more challenging to fix. As I brought up before, if one of the readers crashes inside its critical section, the writer mechanism will also deadlock. To illustrate, search for the following line in the attached solution, and uncomment it:

Code:

---

#define INCLUDE_READER_CRASH        //Uncommect this line to introduce an artifical "Reader" crash

---

This will make the reader process crash after 200 reads. When that happens watch the writer ... it will deadlock.

I think I fixed the issue #1 by enclosing the semaphore in a critical section. Attached is an updated project. There's still the issue #2 remaining though ...

Quote:

---

unconventional approach to using semaphores

---

Well, the idea was got from MSDN article after reading about start with zero counter. So it's not that unconventional. :)

Quote:

---

let me correct a bug. When you create m_hEvtNoRead event, it must be set to signaled state. Otherwise the first reader will deadlock without the first writer call.

---

It might be a feature, as long as without first write it might be senseless to read the void. You know this better, as I'm totally unaware of your ultimate design.

Quote:

---

The sleep(10) part of the EnterWrite() method seems to be a bottleneck

---

I said you before that all my experience with synchronization was mostly about dealing with processes where hundreds of milliseconds never matter. Typically for me in business logic keeping precedence matters much more than super timings. :)

And in case when few milliseconds really is a bottleneck, I'd say that either you need some other synchronization solution, or, which is more probable, your synchronized multi-process reader-writer approach itself is a bottleneck.

Quote:

---

but unfortunately I get occasional "read fail". Why do you think would that be?

---

Like any other pure architectural thing it needs to be analyzed for flaws (code review, sequence diagrams, etc.) and properly debugged (by offline logs or interactively). Seems like I just was able to invest an idea, but have no time to purify it, sorry. Maybe other guys here will help. :)

Yes. Thank you. I appreciate it.