CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 9 of 9
  1. #1
    Join Date
    Sep 2010
    Posts
    39

    intermediate file multi-process safety

    Hi
    I have a compressor that takes a file and first compresses it to an intermediate file "temp.lz", before compressing it to the final format. Immediately after that the file be removed by calling c's remove(char*). The problem I am afraid of is if calling this compressor from different processes is safe because of the intermediate file created which has the same name for all. (say temp.lz). Will I have problems when callinga sytem call from different process something like: system ("compress -i test.txt -o test.z") ?
    I was thinking of rewriting the compressor to avoid the use of the temporary file but that is a bit awkward. I just need to use a temporary stream (file) different for each process to avoid race conditions.
    thanks

  2. #2
    Join Date
    May 2009
    Posts
    2,413

    Re: intermediate file multi-process safety

    Where's the temporary file created?

    Maybe each process could create a uniquely named dictionary and somehow direct the compressor utility to put its intermediate file there. Then there would be no clashes because although the intermediate file is called the same it will be in a different dictionary for each process.

  3. #3
    Join Date
    Oct 2006
    Location
    Sweden
    Posts
    3,654

    Re: intermediate file multi-process safety

    Why is that awkward? Use GetTempFileName instead of a fix file name.
    Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it.
    - Brian W. Kernighan

    To enhance your chance's of getting an answer be sure to read
    http://www.codeguru.com/forum/announ...nouncementid=6
    and http://www.codeguru.com/forum/showthread.php?t=366302 before posting

    Refresh your memory on formatting tags here
    http://www.codeguru.com/forum/misc.php?do=bbcode

    Get your free MS compiler here
    https://visualstudio.microsoft.com/vs

  4. #4
    Join Date
    Sep 2010
    Posts
    39

    Re: intermediate file multi-process safety

    The temporary file is created in the same directory and it could stay alive for 10 seconds where there is conflict. I think I can solve it by giving different names to each temporary file (eg. appending the output file name by other characters). But I wonder if I can just create an "anonymous stream" to use temporarily for each process and then discard on exit.

  5. #5
    Join Date
    May 2009
    Posts
    2,413

    Re: intermediate file multi-process safety

    Quote Originally Posted by dshawul View Post
    The temporary file is created in the same directory
    So the temporary file is always created in a fixed absolute place in the file system? It's not created in a relative place such as the "current" directory or something?

  6. #6
    Join Date
    Sep 2010
    Posts
    39

    Re: intermediate file multi-process safety

    Why is that awkward? Use GetTempFileName instead of a fix file name.
    That seems to be a windows only solution. Is there something in c like remove?

    update: I got about 10 or so corrupt files so indeed the problem is there. Somehow I thought system would be 'process safe'..
    Last edited by dshawul; July 21st, 2012 at 04:09 PM.

  7. #7
    Join Date
    Sep 2010
    Posts
    39

    Re: intermediate file multi-process safety

    Quote Originally Posted by nuzzle View Post
    So the temporary file is always created in a fixed absolute place in the file system? It's not created in a relative place such as the "current" directory or something?
    Yes I want to do a parallel compression of many files on linux cluster all residing in the same current directory. Creating a folder just to put in the temporary file would be problematic.

  8. #8
    Join Date
    Oct 2006
    Location
    Sweden
    Posts
    3,654

    Re: intermediate file multi-process safety

    The "system" is process safe if you make it so. If you open a file as r/w shared well it's going to be that...
    Yes GetTempFilename is windows. In Linux you can use mkstemp or you can use boost::filesystem::unique_path for both systems.
    Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it.
    - Brian W. Kernighan

    To enhance your chance's of getting an answer be sure to read
    http://www.codeguru.com/forum/announ...nouncementid=6
    and http://www.codeguru.com/forum/showthread.php?t=366302 before posting

    Refresh your memory on formatting tags here
    http://www.codeguru.com/forum/misc.php?do=bbcode

    Get your free MS compiler here
    https://visualstudio.microsoft.com/vs

  9. #9
    Join Date
    Sep 2010
    Posts
    39

    Re: intermediate file multi-process safety

    Quote Originally Posted by S_M_A View Post
    The "system" is process safe if you make it so. If you open a file as r/w shared well it's going to be that...
    Yes GetTempFilename is windows. In Linux you can use mkstemp or you can use boost::filesystem::unique_path for both systems.
    Yes another alternative would be to use an flock() but why use it when synchronization can be avoided. Anyway I like appending the output file name with an extension. And it works just fine.
    Thanks all

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured