IPC Suggestions

[Monolith]

I need help determining what methods of interprocess communication would be best to use for my situation.

I am adding on to a large existing application, and I have determined that I need to create a small external process to do some slave operations. The main application will start the external process when it's needed, and close/terminate it when it's no longer needed.

The simplest and almost ideal method of communicating between these two applications would be to use Windows messages. The problem with that though is that the external process will need to send real-time streaming data back to the main application. I don't want to be flooding the main application's message pump with hundreds of messages a second as data comes through the external process, so I need another way of handling this volume of data.

Windows sockets would be a nice solution, except I cannot use this because we believe we have determined that using local asynchronous sockets in combination with certain firewalls can cause serious crashes.

Currently my best option seems to be to use named pipes. The problem I have with this though is that it's not so easy to use pipes asynchronously (at least so far as I've been able to determine). The main application that's going to be receiving this data is primarily single threaded, and that's not going to change. The user interface and data processing happen in the same thread, so I can't use any blocking calls. Overlapped I/O doesn't really work either because I can't be sitting in a loop continually checking if there's data to be read. I need to be notified when there is data available. Idealy I'd like to be able to receive notifications that new data is available by Windows messages, like I can do with asynchronous sockets. I have not found a way to do this with pipes/file handles though. I'd really like to stay away from threads because I don't want to have to start putting synchronization code everywhere, but right now it seems like the only option I have left. If I must use a thread, I want to try to keep it as simple as possible, i.e. do no processing in the thread, but just send Windows messages to the main thread notifying it that there's something to do.

If anybody has any thoughts or other ideas of what might be good to try for my situation, please let me know.

[Arjay]

I'd really like to stay away from threads because I don't want to have to start putting synchronization code everywhere, but right now it seems like the only option I have left.

If you do it properly, you won't have 'sync code everywhere'. When properly done threading code can be encapsulated, logical and easy to maintain.

If I must use a thread, I want to try to keep it as simple as possible, i.e. do no processing in the thread, but just send Windows messages to the main thread notifying it that there's something to do.

You really don't need to involve messaging for this. Let the UI handle the UI stuff and use a second thread to handle the incoming data. This thread can notify the UI when it has data. You can also use a queue mechanism to aide in the processing of the data.

A couple of questions. How large is the data stream? Is is possible to retrieve the data in chunks if the stream is large?

[Monolith]

You really don't need to involve messaging for this. Let the UI handle the UI stuff and use a second thread to handle the incoming data. This thread can notify the UI when it has data. You can also use a queue mechanism to aide in the processing of the data.

All the data processing has to be done in the main thread, which is the same thread as the UI. This is not going to change because many other parts of the project rely on this. The reason I keep going back to messages is because it's the best way I know of safely and simply notifying the main thread of some event from another thread.

A couple of questions. How large is the data stream? Is is possible to retrieve the data in chunks if the stream is large?

Individual complete packets of data are about 300 bytes, and it could easily get up to 100 of those packets in a second. I want to process the data as soon as possible because the timestamp of when I process the data is important. I'm avoiding using a timer to trigger reading and processing data because of this. I need to process the data within the second that it's sent.

[Igor Vartanov]

While both processes run on the same system, the best solution as for me is memory mapped file and a number of dedicated named events and guarding mutex. The "writer" process acquires the mutex, fills the buffer in memory mapped region, releases the mutex and sets event. The "reader" process thread waits on the event, acquires the mutex, reads the data. Releasing mutex it signals that another data chunk may be put into mapped memory.

On the other hand, pipes may be the same way effective, admit this. So it's the matter of preferences.

[TheCPUWizard]

The most accepted model for current professional applications [as a generalization] is to use SOA. One of the best implementations is WCF.

[Arjay]

All the data processing has to be done in the main thread, which is the same thread as the UI. This is not going to change because many other parts of the project rely on this. The reason I keep going back to messages is because it's the best way I know of safely and simply notifying the main thread of some event from another thread.

You can still process the data in the main thread, but it doesn't have to be retrieved from the main thread. Or it could be. Keep in mind that when processing/retreiving data in the main UI thread you risk the loss of data because you can't control what a user is doing in the UI. For example, a user could interrupt the data retrieval by moving the mouse around or moving the application window.

Individual complete packets of data are about 300 bytes, and it could easily get up to 100 of those packets in a second. I want to process the data as soon as possible because the timestamp of when I process the data is important. I'm avoiding using a timer to trigger reading and processing data because of this. I need to process the data within the second that it's sent.[/QUOTE]It's not a great deal of data so you could use the MMF technique Igor mentioned. You could also, queue up the data by putting them into a queue as you received an event from the MMF. From there they would be removed from the queue by a separate thread (UI thread) where they could be processed. The advantage to this would be external processes writing to the MMF wouldn't be blocked waiting for the data to be processed. This technique is illustrated in the sample code listed in the articles in my subject. See the LogSend/LogRcv projects.