CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 1 of 2 12 LastLast
Results 1 to 15 of 22
  1. #1
    Join Date
    Jan 2012
    Location
    USA
    Posts
    91

    [RESOLVED] Questions about a sample code on shared memory.

    Hi,

    I attached a sample code of shared memory that I downloaded from somewhere. However there are few things about it:

    1. Producer must be started first, o.w. consumer gives an error: "shm_open: No such file or directory". -- I want to be able to start the processes in any order.
    2. If Producer restarts, then Consumer must be restarted as well. -- That's very inconvenient, if my producer goes down then I'd have to restart Consumer as well.
    3. Consumer takes 100% of CPU time of one core. -- I think that's because Consumer is polling the buffer, can it be resolved using semaphores?

    Someone could comment how to resolve those?

    THX
    Attached Files Attached Files

  2. #2
    Join Date
    Nov 2003
    Posts
    1,902

    Re: Questions about a sample code on shared memory.

    >> ... that I downloaded from somewhere.
    That code is garbage. Don't even try to learn something from it.

    This looks better for learning: https://github.com/risingape/producer-consumer
    It's a double-buffer technique instead of a ring-buffer. Once you understand how to synchronize with semaphores, applying it to a ring-buffer approach isn't too difficult (there are examples of that on the inter-web too).

    gg

  3. #3
    Join Date
    Jan 2012
    Location
    USA
    Posts
    91

    Re: Questions about a sample code on shared memory.

    Thank you for the code, I might have some questions later so I am not going to close it for now.

  4. #4
    Join Date
    Jul 2013
    Posts
    576

    Re: Questions about a sample code on shared memory.

    Quote Originally Posted by vincegata View Post
    Someone could comment how to resolve those?
    If you're using C++ 11 it's pretty easy to turn an ordinary std::queue into a thread-safe blocking queue that can be used in a producer-consumer scenario,

    Code:
    	template<typename T>
    	class blocking_queue {
    		typedef std::mutex Mutex;
    	public:
    		void push(const T& value) {
    			std::lock_guard<Mutex> lock(mutex);
    			queue.push(value);
    			condition.notify_one();
    		}
    
    		bool try_pop(T& value) {
    			std::lock_guard<Mutex> lock(mutex);
    			if (queue.empty()) return false;
    			value = queue.front();
    			queue.pop();
    			return true;
    		}
    
    		T pop() { // blocking
    			std::unique_lock<Mutex> lock(mutex);
    			condition.wait(lock,[&]{return !queue.empty();});
    			T value = queue.front();
    			queue.pop();
    			return value;
    		}
    
    	private:
    		Mutex mutex;
    		std::queue<T> queue;
    		std::condition_variable condition;
    	};
    The producer pushes items on the queue using push() and if the consumer calls pop() it will sleep waiting for an item to arrive. The try_pop() method returns immediately with or without an item depending on whether there was one.
    Last edited by razzle; October 15th, 2013 at 03:17 AM.

  5. #5
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626

    Re: Questions about a sample code on shared memory.

    "shared memory" does not require an order of who starts first, although if you want a consumer to start before a producer, this may involve additional synchronisation and code to properly initialise the shared memory.

    It appears your code sample isn't about shared memory, but about using a memorymapped file to achieve more or less the same thing.
    Again, this can be changed to consumer first, but it may require additional synchronisation to have the consumer "start up" the file mapping (possibly creating the file) if it doesn't already exist.

    I agree with codeplug, the ipc.zip sample is not the way to do it.

  6. #6
    Join Date
    Jan 2012
    Location
    USA
    Posts
    91

    Re: Questions about a sample code on shared memory.

    @OReubens, that code sample is about using shared memory without locking so to reduce latency. I am looking at the codeplug's sample now.

    @razzle, thanks for the code snippet, I'll re-use it somewhere. However, in this particular case I essentially have two applications - two executables that do not share any code, not sure if I can use container.

  7. #7
    Join Date
    Jul 2013
    Posts
    576

    Re: Questions about a sample code on shared memory.

    Quote Originally Posted by vincegata View Post
    However, in this particular case I essentially have two applications - two executables that do not share any code, not sure if I can use container.
    No then you cannot use a C++ container. You could check out a Boost library called Interprocess.

    http://www.boost.org/doc/libs/1_54_0...erprocess.html

  8. #8
    Join Date
    Nov 2003
    Posts
    1,902

    Re: Questions about a sample code on shared memory.

    >> that code sample is about using shared memory without locking so to reduce latency.
    Don't try to solve a latency issue that doesn't exist (premature optimization).

    The code in that zip was an attempt to use the Linux Kernel circular buffer macro's: https://www.kernel.org/doc/Documenta...ar-buffers.txt

    Implementing correct lock-free code is *extremely* hard (even in C++11). Stick with the synchronization primitives that Posix provides.

    What is the data that is being produced/consumed? Why use shared memory? If there is only a single producer and single consumer, consider using a pipe or socket and simply stream the data.

    gg

  9. #9
    Join Date
    Jan 2012
    Location
    USA
    Posts
    91

    Re: Questions about a sample code on shared memory.

    @razzle - I am staying away from the third parties.

    @codeplug - these are some financial data and there are a lot of them. I am using FIFO right now (with your help too btw ) but I found it to be somewhat too slow.

    So it's an event-driven application with Data Adapter (producer) that either reads data from the internet over sockets in real-time or it reads stored data from a file / database. I have Data Analyzer (consumer) that processes those data. Right now producer sends serialized data to consumer over FIFO. It works alright when producer reads the data from the Internet but when producer reads the data from the database then FIFO becomes a bottleneck. Note, that my consumer cannot go directly into a file / database to read the data, the data must go from producer to consumer - that's the requirement.

    To replace FIFO I am considering to use shared memory. I've also just started looking at the Observer Pattern.

  10. #10
    Join Date
    Jul 2013
    Posts
    576

    Re: Questions about a sample code on shared memory.

    Quote Originally Posted by vincegata View Post
    @razzle - I am staying away from the third parties.
    Well, Boost isn't just any third party. It's almost as good as the C++ standard library. Many, even most of the features you find in the C++ standard library have started out in Boost.

    I would pick a library from Boost any time over some company specific alternative. It's peer tested, portable and may even be part of C++ after a while.

  11. #11
    Join Date
    Apr 2000
    Location
    Belgium (Europe)
    Posts
    4,626

    Re: Questions about a sample code on shared memory.

    Quote Originally Posted by vincegata View Post
    @OReubens, that code sample is about using shared memory without locking so to reduce latency. I am looking at the codeplug's sample now.
    Now you're confusing a lock free algorithm
    with shared memory
    and with memory mapping.

    if you really have to, you can implement a lock-free single linked list in shared memory which is slightly more complex as a regular implementation as you need to account for the fact that the list may be located at a different memory location in each process. I'm not aware of any other containers that simultaneously work lock-free and in shared memory.
    There's a couple "gotacha's" with it though.

    The recommended way it to stick with regular synchronisation if you need shared memory.

  12. #12
    Join Date
    Jan 2012
    Location
    USA
    Posts
    91

    Re: Questions about a sample code on shared memory.

    @OReubens - so I have a producer and a consumer, the point is to send the data as fast as possible. Since synchronization takes time, a person who developed the code that I posted, used polling instead of semaphores.

    Thx about linked list, is it possible to have a lock-free linked list in a memory of a process instead of a shared memory? Can I use it to exchange data between threads in a lock-free manner?

  13. #13
    Join Date
    Nov 2003
    Posts
    1,902

    Re: Questions about a sample code on shared memory.

    >> the point is to send the data as fast as possible. Since synchronization takes time, a person who developed the code that I posted, used polling instead of semaphores.
    Premature optimization.

    First you have to implement something. Then you profile it. Once the bottleneck is identified, then you can start thinking about how to squeeze that bottleneck.

    gg

  14. #14
    Join Date
    Jan 2012
    Location
    USA
    Posts
    91

    Re: Questions about a sample code on shared memory.

    @Codeplug - I am not sure if you read my post above the one that you quoted, but I do already have a running application using FIFO and it is too slow for me.

  15. #15
    Join Date
    Nov 2003
    Posts
    1,902

    Re: Questions about a sample code on shared memory.

    I was referring to shared-mem implementations only.

    But since you mention it - how was the FIFO determined to be the bottle neck?

    What is the raw messages-per-second (mps) that the consumer code can handle? (measure a hard loop that delivers test messages directly to the consumer function)

    What is the raw mps that the FIFO could deliver? (measure hard loop that adds test data to FIFO with hard consumption loop that reads each message and discards it)

    Knowing these kinds of characteristics can help in deciding what kind of shared-mem impl. to use - double/triple buffers, ring buffer, etc...

    gg

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured