"Safe" FIFO/Pipe

Hello!

I've already searched the forum but didn't find an answer to the following.

The system I'm working on includes some datasource that produces records which have to be transferred to a MS Access Database (append to a table). To make sure that none of these records get lost if the database is too busy, i would like to pass them through a "safe" FIFO: the producer-thread puts the record into the fifo, then a consumer thread fetches them out of the fifo and puts them into the DB.

In case of a system crash (producer, consumer, both, ... however), it must be guaranteed, that no records in the FIFO get lost! I remember that in Linux there were file-based FIFOs called "Pipes". Is there something similar in Win32/VC++?

Any help will be appreciated!

Why write something to persistent storage twice?

Quote:

---

Originally Posted by Sahir

Why write something to persistent storage twice?

---

i want to guarantee, that the data to is written to the database. if the database is busy, so that writing fails, i must guarantee, that the data is buffered until writing is possible! as i can't "wait" for the database to be available, i have to put new records into some (FIFO) buffer.

i want this FIFO to be persistent, so that if the processes crash before all buffered records are processed, no records get lost. I MUST NOT LOSE ONE SINGLE RECORD!

The Microsoft facilities for inter-thread (or inter-process) communication do not involve persistent storage. You've probably already searched MSDN for "IPC" (inter-process communication) and seen what's available.

It's not clear to me how your scheme would work; if the consumer thread fails, the data would be in the FIFO (as you want), but if the producer thread fails, the data may or may not be in the FIFO. I think what you need is not so much persistent intermediate storage as much as messages between the producer and consumer threads telling each other the state of the data transfer. In this case, if either thread fails, the absence of any "sanity" message would inform the other, and the data transfer could be re-initiated. If both threads fail - in other words, the computer crashes - the producer thread would need some method of knowing what its last-processed record was (eg., writing to a disk file).

(I'm assuming that your producer thread will have the ability to request a retransmission of records from your data source).

Quote:

---

Originally Posted by TSYS

...

(I'm assuming that your producer thread will have the ability to request a retransmission of records from your data source).

---

this is an important assumption, which is wrong. a retransmission to the producer thread is not possible. my data source is a scanner device that registers bottles on a conveyer. if they're past the scanner, they're past.

i will implement some "dead-man's safety feature" which stops the conveyer when the producer thread crashes. but i don't want to stop the conveyer only because the database is busy!

OK, if worrying about the producer thread crashing is not important (in this context), then you don't need persistent storage. A pipe (as you originally surmised) will work, as will any method that allows the producer to queue up data which then awaits processing by the consumer. You may find that a CMemFile will do what you want.

You'll still need some sort of sanity handshake, though, so the producer knows when to restart the consumer thread. And, of course, you'll need to figure out what happens if the producer thread pushes data into the FIFO faster than the consumer can take it out (a situation we hope cannot arise).

Quote:

---

Originally Posted by TSYS

... You may find that a CMemFile will do what you want.

---

thank you for the sugestion!
i'm not sure if a file - on hard disk or in memory - will do what i want. my producer will continuously write records to the end of that file. the consumer will read that records from the file one after the other. after the consumer has processed one record, is should be removed from the file.
now the problem is, that as long as the file is opened for writing by the producer, it will be hard for the consumer to delete something from the file, right? so the file will grow a little, i assume ;)

maybe i'm just not getting the idea... please tell me

Quote:

---

Originally Posted by TSYS

You'll still need some sort of sanity handshake, though, ...

---

of course.

I HAD AN IDEA!

i could create one separate file for each record, serially numbered. then the consumer can process file by file, deleting each file after it was processed!

performance? bad, but it will work.

I would say - forget about access database!

U really don't need any pipes here!
If U install MSDE (for example) U can make your application more robust and failsafe.

Any buffering system will only add more unnecesery complexity to your code. Persistent storage in buffer will do acctually the same thing as database do .

Best regards,
Krzemo.

Quote:

---

Originally Posted by reset-leo

now the problem is, that as long as the file is opened for writing by the producer, it will be hard for the consumer to delete something from the file, right? so the file will grow a little, i assume ;)

---

You assume correctly, and I wasn't thinking. The file will grow more than "a little"!

Have a look at "anonymous pipes". I think an anonymous pipe will do what you want.

[Edit]
Yes, what you propose with multiple files would work. I have two reservations, though:

- The time overhead with creating / deleting files will be considerable
- There is a chance of data corruption. What will you do if the consumer gets a "read" error?

Quote:

---

Originally Posted by TSYS

Have a look at "anonymous pipes". I think an anonymous pipe will do what you want.

---

i'm afraid the standard anonymous pipe will be closed as soon as the threads die (freeing the memory containing unprocessed records).

@krzemo: i'll tell our customer to EAT all the access licenses he has and spend some money on a hi-tec database. of course, i'll add that he won't need the database for anything else than this pilot installation my company does, but he needn't care, because it is good as long as it reduces my effort :wave:

Quote:

---

and spend some money on a hi-tec database

---

MSDE is for free ;) .

Quote:

---

Originally Posted by Krzemo

MSDE is for free ;) .

---

however... what i wanted to say is that if the customer wants access, he will get access.
NEVER argue with customers on such things.

Quote:

---

Originally Posted by reset-leo

i'm afraid the standard anonymous pipe will be closed as soon as the threads die (freeing the memory containing unprocessed records).

---

Hmm, I see what you mean. Could the producer keep a copy of each record it puts in the pipe, deleting it when the consumer indicates it has been received and processed? That way, if the pipe collapses (because the consumer has died), the producer can re-stuff the pipe once it's been re-established.

Another possibility to ensure data stability is to borrow a trick from the aircraft industry: run multiple copies of each thread and have a "sanity check" mechanism to determine which producer is correct and which consumer is correct. This gets complicated, but you can extend it to have several producer / consumer threads running on several computers....

Quote:

---

NEVER argue with customers on such things

---

However U can give them an advise.... it is for theirs own good, maybe they understand it ;) .

If not U can replicate MSDE tables to access database in any time. So MSDE can work as FIFO/Pipe for U. Than U can create tables optimised for writting only and set dedlock priority to "producer" task, so it will never encounter bussy state.

Best regards,
Krzemo.

Quote:

---

Originally Posted by TSYS

Hmm, I see what you mean. Could the producer keep a copy of each record it puts in the pipe, deleting it when the consumer indicates it has been received and processed? That way, if the pipe collapses (because the consumer has died), the producer can re-stuff the pipe once it's been re-established.

---

good idea! but where should i put the record copies? if they're in non-persistant memory, they're lost if producer AND consumer die at once... if i put them into a file, i would have to remove them from the file, which is almost the same operations as when each record uses it's own file.

Quote:

---

Originally Posted by TSYS

Another possibility to ensure data stability is to borrow a trick from the aircraft industry: run multiple copies of each thread and have a "sanity check" mechanism to determine which producer is correct and which consumer is correct. This gets complicated, but you can extend it to have several producer / consumer threads running on several computers....

---

:eek: i think i will first check if there are enough system resources to do the one-file-per-record thing because this sounds like "work"... i don't need "rocket science" in the moment ;)

@krzemo: i was thinking about using a database as buffer between producer and consumer, but that's too complicated and it's also too unsafe for me because i'm not sure how the DB reacts on low system resources.
i must be absolutely sure that my producer can immediately put the records somewhere where they can not get lost.


@

Would MSMQ do what you are looking to do? MSMQ was designed to be a middleware component for fast, reliable asynchronous electronic communication. It also ensures guaranteed message delivery, receipt notification and transaction control. Here is the link in MSDN.

Quote:

---

Originally Posted by reset-leo

... they're lost if producer AND consumer die at once... if i put them into a file, i would have to remove them from the file, which is almost the same operations as when each record uses it's own file.

---

Well, I think if the producer dies you can do nothing (other than shut down the bottle conveyor, as you indicated in an earlier posting). As I said before, if the producer dies you have no knowledge of whether or not it was in the midst of writing data to persistent storage.

The "aircraft" methodology wasn't intended seriously. I did work on such a system once, and it was very, very complicated - but it did keep the helicopters flying!

Quote:

---

but that's too complicated and it's also too unsafe for me because i'm not sure how the DB reacts on low system resources.

---

Ms SQL will react better than access. And it is less complicated (and safer) than FIFO/Pipe solution...

Let me put it this way:
U are spending a lot of time trying to protect data from being lost on the way to its destination. And finally (sooner or later) U'll do it... but is access database a safe place?
How will U find yourself in situation when destination database will be corrupt, and U loose all information collected with such a big affort ...

Just think about it..

Best regards,
Krzemo.

ok. databases usually are great programs which are fast, safe, reliable, ...
but databases usually are also big programs with lots of features and consuming much system resources and so on.

i don't want to use a whole "solution" only as a "buffer". well, maybe it simplifies the development a little, but i will have to set that all up and support it at the customer's site, which is 250km away from our office! i feel much more confident keeping this feature as small and simple as i can and knowing how everything works and that it works as i want it to.
maybe i'm a bit too cautious, but i also don't want to rely on the database tha i can really store the data when it should, as it may be too busy, crashed or whatever (of course, if i get an error from the database, i can stop the conveyer, but the customer won't be too happy if it stops). i want to rely on the file system.

Quote:

---

Originally Posted by TSYS

Well, I think if the producer dies you can do nothing (other than shut down the bottle conveyor, as you indicated in an earlier posting). As I said before, if the producer dies you have no knowledge of whether or not it was in the midst of writing data to persistent storage.

---

an element of risk will remain. i have to accept that.

Quote:

---

Originally Posted by mike200

Would MSMQ do what you are looking to do? MSMQ was designed to be a middleware component for fast, reliable asynchronous electronic communication. It also ensures guaranteed message delivery, receipt notification and transaction control. Here is the link in MSDN.

---

interesting!