FileStream buffered output
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 5 of 5

Thread: FileStream buffered output

  1. #1
    Join Date
    Jul 2007
    Location
    Illinois
    Posts
    517

    FileStream buffered output

    I have some pretty large text files in excess of 600 MB on average that I need to open, read line by line, do some processing with that data, and then close the file. How does the FileStream object handle files of that size? Does it open them and store a buffered Stream (a cache maybe?) of their contents in memory? Or does it only buffer the input to memory when the input is read?

    I guess I'm trying to figure out how the FileStream class handles read and write operation without modifying the source file until you Flush() or Close() the stream. If you can "write" to the FileStream object without it modifying the original source file, how is that achieved?

    Thanks for the help
    R.I.P. 3.5" Floppy Drives
    "I know not with what weapons World War III will be fought, but World War IV will be fought with sticks and stones." - Albert Einstein

  2. #2
    Join Date
    Feb 2005
    Location
    Denmark
    Posts
    742

    Re: FileStream buffered output

    I've done something similar recently with files around 800-1000 MB and in one of the situations where I had to do some replace work on the file, then the way I did it was create a FileStreamReader and FileStreamWriter and effectively copied the file while doing my transformation work.

    So I had a FileStreamReader where I called ReadLine (I could use ReadLine, as it contained line breaks) and then did the work on the line and wrote it back using my FileStreamWriter with the method WriteLine.
    Alternative (if ReadLine isn't an option) I could have used MemoryStream as a buffer, but I would properly still write it back into another file using FileStreamWriter just to keep the source file intact until my transformation was done.

  3. #3
    Join Date
    Jul 2007
    Location
    Illinois
    Posts
    517

    Re: FileStream buffered output

    So you copied the file into memory? Or just to a "temp" location on disk so you could modify it? My main concern is that the application reading these files is going out to clients and I don't want to eat all of their RAM up by reading these files.
    R.I.P. 3.5" Floppy Drives
    "I know not with what weapons World War III will be fought, but World War IV will be fought with sticks and stones." - Albert Einstein

  4. #4
    Join Date
    Feb 2005
    Location
    Denmark
    Posts
    742

    Re: FileStream buffered output

    Quote Originally Posted by RaleTheBlade View Post
    So you copied the file into memory? Or just to a "temp" location on disk so you could modify it? My main concern is that the application reading these files is going out to clients and I don't want to eat all of their RAM up by reading these files.
    I copied the file as I was worked on it, not prior to working. So only a small segment was read into memory at the time.

    I read a line into memory.
    Did my work on that line.
    Wrote the line back to another file.

    This left a small memory footprint because I only had one line in memory, and the IO didn't seem to be a significant factor either.
    If you can't "read line", due to for example not having line breaks or similar, I would read a segment of a predetermined size into a memorystream, work on it and write it to disk.

    Then when done you can delete the first file and rename the second or how you want to handle it the result.

  5. #5
    Join Date
    Jul 2007
    Location
    Illinois
    Posts
    517

    Re: FileStream buffered output

    I gotcha... That was an option for us and that is basically how Im doing it now I was just curious as to how the FileStream class handled IO as far as opening a Stream, writing to the Stream, and saving the Stream. Because I know when you open a FileStream, you can make changes to the contents of the stream without changing the original source file unless Flush() or Close() is called.
    R.I.P. 3.5" Floppy Drives
    "I know not with what weapons World War III will be fought, but World War IV will be fought with sticks and stones." - Albert Einstein

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Azure Activities Information Page

Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center