Managing an array shared by multiple threads?

[mrgr8avill]

Hello - thanks for taking the time to read this. I am developing a program in which work is queued into either a DB or an array, and multiple (about 100) threads perform the work by obtaining the next item in the list and processing it.

Obviosuly, there is a huge risk of threads deadheading or getting the same entry from the array or db when a second thread queries for the next item to process before the first thread has had an opportunity to remove the item from the queue.

My thoughts are to use a static class method and boolean to lock the queue:
public static List<string> dataQueue = new List<string>;
public static bool isQLocked = false;
public static void lockQ(){
while(isQLocked==true){
Thread.sleep(100);
}
isQLocked=true;
}

Then, in the instance threads, each will first lock the queue by calling the static lockQ() method before processing. Once the method returns, it would be assumed that the calling instance had exclusive control, as all other instances would be stuck waiting for isQLocked to be made false again:

staticClass.lockQ();
//no other instance of this method should be able to get here now -- this instance should have exclusive control
string nextInLine = dataQueue[0];
dataQueue.RemoveAt(0);
//done with getting the next item in the queue - release the lock to the next instance
staticClass.isQLocked=false;

Question is -- does this work? I have tried other methods and keep failing. It's hours and hours to clear out my garbage and work something like this, so I'm hoping for some insight and advice before I have to try it again.

Am I right in thinking that the first instance class calling the static method would get the lock, and then all other calling instances would be queued and released one by one? Or would it happen like other things I've tried where as soon as the first calling instance releases the lock, there is a massive jumble and the risk of releasing two of the same queue items still exists?

Thanks so much in advance for your help and any infromation you can provide as to how might be the best way to "feed" a single array to 100 threads such that they don't get duplicates.

I'm wondering if this concept

[Lindley]

Just use a mutex from your threading library. There's little point in trying to invent your own locking mechanism (which won't work as shown above, since even boolean assignments are not guaranteed to be atomic) when there are perfectly good locks readily available.

[mrgr8avill]

Dude...you rock! Such is the ignorance of being self-taught that I didn't know it existed! THANKS!!!

[Lindley]

Arguably the two most important things to be familiar with in any threading library are mutexes and condition variables. The former allow you to protect important data structures and variables from simultaneous access; the latter allows you to efficiently cause one thread to wait for a particular event to happen in another thread.

Both will probably be needed to define a reader/writer queue mechanism correctly.

[dvyukov]

Arguably the two most important things to be familiar with in any threading library are mutexes and condition variables. The former allow you to protect important data structures and variables from simultaneous access; the latter allows you to efficiently cause one thread to wait for a particular event to happen in another thread.

Both will probably be needed to define a reader/writer queue mechanism correctly.

mrgr8avill, you need something along the lines of:

class request_queue
{
public:
  void enqueue(request* req)
  {
    bool notify = false;
    {
      std::lock_guard lock (mtx);
      notify = queue.empty();
      queue.push_back(req);
    }      
    if (notify)
      cv.notify_one();
  }
  
  request* dequeue()
  {
      std::lock_guard lock (mtx);
      while (queue.empty())
        cv.wait(lock);
      request* req = queue.front();
      queue.pop_front();
      return req;
  }
  
private:
  std::mutex mtx;
  std::condition_variable cv;
  std::queue<request*> queue;
};

[dvyukov]

There's little point in trying to invent your own locking mechanism ... when there are perfectly good locks readily available.

Yeah, until your own locking mechanism does not outperform those "perfectly good locks" by several orders of magnitude.

[dvyukov]

how might be the best way to "feed" a single array to 100 threads such that they don't get duplicates.

There is not best way for that, because the idea is busted in itself.
What is the best way to manage 100 programmers to work on a single computer? Sorry, you just need more computers.

[ClayBreshears]

I can't add anything except to repeat the old saw: Don't reinvent the wheel. I find that (practically) every time I ask myself "Why hasn't anyone ever thought of this?", invariably someone has already thought of it.

[mgullapalli]

I can't add anything except to repeat the old saw: Don't reinvent the wheel. I find that (practically) every time I ask myself "Why hasn't anyone ever thought of this?", invariably someone has already thought of it.

dsdsdsdsd-Ravi

[Lindley]

??