"Safe" FIFO/Pipe

Hello!

I've already searched the forum but didn't find an answer to the following.

The system I'm working on includes some datasource that produces records which have to be transferred to a MS Access Database (append to a table). To make sure that none of these records get lost if the database is too busy, i would like to pass them through a "safe" FIFO: the producer-thread puts the record into the fifo, then a consumer thread fetches them out of the fifo and puts them into the DB.

In case of a system crash (producer, consumer, both, ... however), it must be guaranteed, that no records in the FIFO get lost! I remember that in Linux there were file-based FIFOs called "Pipes". Is there something similar in Win32/VC++?

Any help will be appreciated!

Why write something to persistent storage twice?

Quote:

---

Originally Posted by Sahir

Why write something to persistent storage twice?

---

i want to guarantee, that the data to is written to the database. if the database is busy, so that writing fails, i must guarantee, that the data is buffered until writing is possible! as i can't "wait" for the database to be available, i have to put new records into some (FIFO) buffer.

i want this FIFO to be persistent, so that if the processes crash before all buffered records are processed, no records get lost. I MUST NOT LOSE ONE SINGLE RECORD!

The Microsoft facilities for inter-thread (or inter-process) communication do not involve persistent storage. You've probably already searched MSDN for "IPC" (inter-process communication) and seen what's available.

It's not clear to me how your scheme would work; if the consumer thread fails, the data would be in the FIFO (as you want), but if the producer thread fails, the data may or may not be in the FIFO. I think what you need is not so much persistent intermediate storage as much as messages between the producer and consumer threads telling each other the state of the data transfer. In this case, if either thread fails, the absence of any "sanity" message would inform the other, and the data transfer could be re-initiated. If both threads fail - in other words, the computer crashes - the producer thread would need some method of knowing what its last-processed record was (eg., writing to a disk file).

(I'm assuming that your producer thread will have the ability to request a retransmission of records from your data source).

Quote:

---

Originally Posted by TSYS

...

(I'm assuming that your producer thread will have the ability to request a retransmission of records from your data source).

---

this is an important assumption, which is wrong. a retransmission to the producer thread is not possible. my data source is a scanner device that registers bottles on a conveyer. if they're past the scanner, they're past.

i will implement some "dead-man's safety feature" which stops the conveyer when the producer thread crashes. but i don't want to stop the conveyer only because the database is busy!

OK, if worrying about the producer thread crashing is not important (in this context), then you don't need persistent storage. A pipe (as you originally surmised) will work, as will any method that allows the producer to queue up data which then awaits processing by the consumer. You may find that a CMemFile will do what you want.

You'll still need some sort of sanity handshake, though, so the producer knows when to restart the consumer thread. And, of course, you'll need to figure out what happens if the producer thread pushes data into the FIFO faster than the consumer can take it out (a situation we hope cannot arise).

Quote:

---

Originally Posted by TSYS

... You may find that a CMemFile will do what you want.

---

thank you for the sugestion!
i'm not sure if a file - on hard disk or in memory - will do what i want. my producer will continuously write records to the end of that file. the consumer will read that records from the file one after the other. after the consumer has processed one record, is should be removed from the file.
now the problem is, that as long as the file is opened for writing by the producer, it will be hard for the consumer to delete something from the file, right? so the file will grow a little, i assume ;)

maybe i'm just not getting the idea... please tell me

Quote:

---

Originally Posted by TSYS

You'll still need some sort of sanity handshake, though, ...

---

of course.

I HAD AN IDEA!

i could create one separate file for each record, serially numbered. then the consumer can process file by file, deleting each file after it was processed!

performance? bad, but it will work.

I would say - forget about access database!

U really don't need any pipes here!
If U install MSDE (for example) U can make your application more robust and failsafe.

Any buffering system will only add more unnecesery complexity to your code. Persistent storage in buffer will do acctually the same thing as database do .

Best regards,
Krzemo.

Quote:

---

Originally Posted by reset-leo

now the problem is, that as long as the file is opened for writing by the producer, it will be hard for the consumer to delete something from the file, right? so the file will grow a little, i assume ;)

---

You assume correctly, and I wasn't thinking. The file will grow more than "a little"!

Have a look at "anonymous pipes". I think an anonymous pipe will do what you want.

[Edit]
Yes, what you propose with multiple files would work. I have two reservations, though:

- The time overhead with creating / deleting files will be considerable
- There is a chance of data corruption. What will you do if the consumer gets a "read" error?

Quote:

---

Originally Posted by TSYS

Have a look at "anonymous pipes". I think an anonymous pipe will do what you want.

---

i'm afraid the standard anonymous pipe will be closed as soon as the threads die (freeing the memory containing unprocessed records).

@krzemo: i'll tell our customer to EAT all the access licenses he has and spend some money on a hi-tec database. of course, i'll add that he won't need the database for anything else than this pilot installation my company does, but he needn't care, because it is good as long as it reduces my effort :wave:

Quote:

---

and spend some money on a hi-tec database

---

MSDE is for free ;) .

Quote:

---

Originally Posted by Krzemo

MSDE is for free ;) .

---

however... what i wanted to say is that if the customer wants access, he will get access.
NEVER argue with customers on such things.

Quote:

---

Originally Posted by reset-leo

i'm afraid the standard anonymous pipe will be closed as soon as the threads die (freeing the memory containing unprocessed records).

---

Hmm, I see what you mean. Could the producer keep a copy of each record it puts in the pipe, deleting it when the consumer indicates it has been received and processed? That way, if the pipe collapses (because the consumer has died), the producer can re-stuff the pipe once it's been re-established.

Another possibility to ensure data stability is to borrow a trick from the aircraft industry: run multiple copies of each thread and have a "sanity check" mechanism to determine which producer is correct and which consumer is correct. This gets complicated, but you can extend it to have several producer / consumer threads running on several computers....

Quote:

---

NEVER argue with customers on such things

---

However U can give them an advise.... it is for theirs own good, maybe they understand it ;) .

If not U can replicate MSDE tables to access database in any time. So MSDE can work as FIFO/Pipe for U. Than U can create tables optimised for writting only and set dedlock priority to "producer" task, so it will never encounter bussy state.

Best regards,
Krzemo.