intermediate file multi-process safety
Hi
I have a compressor that takes a file and first compresses it to an intermediate file "temp.lz", before compressing it to the final format. Immediately after that the file be removed by calling c's remove(char*). The problem I am afraid of is if calling this compressor from different processes is safe because of the intermediate file created which has the same name for all. (say temp.lz). Will I have problems when callinga sytem call from different process something like: system ("compress -i test.txt -o test.z") ?
I was thinking of rewriting the compressor to avoid the use of the temporary file but that is a bit awkward. I just need to use a temporary stream (file) different for each process to avoid race conditions.
thanks
Re: intermediate file multi-process safety
Where's the temporary file created?
Maybe each process could create a uniquely named dictionary and somehow direct the compressor utility to put its intermediate file there. Then there would be no clashes because although the intermediate file is called the same it will be in a different dictionary for each process.
Re: intermediate file multi-process safety
Why is that awkward? Use GetTempFileName instead of a fix file name.
Re: intermediate file multi-process safety
The temporary file is created in the same directory and it could stay alive for 10 seconds where there is conflict. I think I can solve it by giving different names to each temporary file (eg. appending the output file name by other characters). But I wonder if I can just create an "anonymous stream" to use temporarily for each process and then discard on exit.
Re: intermediate file multi-process safety
Quote:
Originally Posted by
dshawul
The temporary file is created in the same directory
So the temporary file is always created in a fixed absolute place in the file system? It's not created in a relative place such as the "current" directory or something?
Re: intermediate file multi-process safety
Quote:
Why is that awkward? Use GetTempFileName instead of a fix file name.
That seems to be a windows only solution. Is there something in c like remove?
update: I got about 10 or so corrupt files so indeed the problem is there. Somehow I thought system would be 'process safe'..
Re: intermediate file multi-process safety
Quote:
Originally Posted by
nuzzle
So the temporary file is always created in a fixed absolute place in the file system? It's not created in a relative place such as the "current" directory or something?
Yes I want to do a parallel compression of many files on linux cluster all residing in the same current directory. Creating a folder just to put in the temporary file would be problematic.
Re: intermediate file multi-process safety
The "system" is process safe if you make it so. If you open a file as r/w shared well it's going to be that...
Yes GetTempFilename is windows. In Linux you can use mkstemp or you can use boost::filesystem::unique_path for both systems.
Re: intermediate file multi-process safety
Quote:
Originally Posted by
S_M_A
The "system" is process safe if you make it so. If you open a file as r/w shared well it's going to be that...
Yes GetTempFilename is windows. In Linux you can use mkstemp or you can use boost::filesystem::unique_path for both systems.
Yes another alternative would be to use an flock() but why use it when synchronization can be avoided. Anyway I like appending the output file name with an extension. And it works just fine.
Thanks all
