"Safe" FIFO/Pipe

[reset-leo]

Hmm, I see what you mean. Could the producer keep a copy of each record it puts in the pipe, deleting it when the consumer indicates it has been received and processed? That way, if the pipe collapses (because the consumer has died), the producer can re-stuff the pipe once it's been re-established.

good idea! but where should i put the record copies? if they're in non-persistant memory, they're lost if producer AND consumer die at once... if i put them into a file, i would have to remove them from the file, which is almost the same operations as when each record uses it's own file.

Another possibility to ensure data stability is to borrow a trick from the aircraft industry: run multiple copies of each thread and have a "sanity check" mechanism to determine which producer is correct and which consumer is correct. This gets complicated, but you can extend it to have several producer / consumer threads running on several computers....

i think i will first check if there are enough system resources to do the one-file-per-record thing because this sounds like "work"... i don't need "rocket science" in the moment

@krzemo: i was thinking about using a database as buffer between producer and consumer, but that's too complicated and it's also too unsafe for me because i'm not sure how the DB reacts on low system resources.
i must be absolutely sure that my producer can immediately put the records somewhere where they can not get lost.


@

[mike200]

Would MSMQ do what you are looking to do? MSMQ was designed to be a middleware component for fast, reliable asynchronous electronic communication. It also ensures guaranteed message delivery, receipt notification and transaction control. Here is the link in MSDN.

[TSYS]

... they're lost if producer AND consumer die at once... if i put them into a file, i would have to remove them from the file, which is almost the same operations as when each record uses it's own file.

Well, I think if the producer dies you can do nothing (other than shut down the bottle conveyor, as you indicated in an earlier posting). As I said before, if the producer dies you have no knowledge of whether or not it was in the midst of writing data to persistent storage.

The "aircraft" methodology wasn't intended seriously. I did work on such a system once, and it was very, very complicated - but it did keep the helicopters flying!

[Krzemo]

but that's too complicated and it's also too unsafe for me because i'm not sure how the DB reacts on low system resources.

Ms SQL will react better than access. And it is less complicated (and safer) than FIFO/Pipe solution...

Let me put it this way:
U are spending a lot of time trying to protect data from being lost on the way to its destination. And finally (sooner or later) U'll do it... but is access database a safe place?
How will U find yourself in situation when destination database will be corrupt, and U loose all information collected with such a big affort ...

Just think about it..

Best regards,
Krzemo.

[reset-leo]

ok. databases usually are great programs which are fast, safe, reliable, ...
but databases usually are also big programs with lots of features and consuming much system resources and so on.

i don't want to use a whole "solution" only as a "buffer". well, maybe it simplifies the development a little, but i will have to set that all up and support it at the customer's site, which is 250km away from our office! i feel much more confident keeping this feature as small and simple as i can and knowing how everything works and that it works as i want it to.
maybe i'm a bit too cautious, but i also don't want to rely on the database tha i can really store the data when it should, as it may be too busy, crashed or whatever (of course, if i get an error from the database, i can stop the conveyer, but the customer won't be too happy if it stops). i want to rely on the file system.

[reset-leo]

Well, I think if the producer dies you can do nothing (other than shut down the bottle conveyor, as you indicated in an earlier posting). As I said before, if the producer dies you have no knowledge of whether or not it was in the midst of writing data to persistent storage.

an element of risk will remain. i have to accept that.

[reset-leo]

Would MSMQ do what you are looking to do? MSMQ was designed to be a middleware component for fast, reliable asynchronous electronic communication. It also ensures guaranteed message delivery, receipt notification and transaction control. Here is the link in MSDN.

interesting!