FileStream buffered output

[RaleTheBlade]

I have some pretty large text files in excess of 600 MB on average that I need to open, read line by line, do some processing with that data, and then close the file. How does the FileStream object handle files of that size? Does it open them and store a buffered Stream (a cache maybe?) of their contents in memory? Or does it only buffer the input to memory when the input is read?

I guess I'm trying to figure out how the FileStream class handles read and write operation without modifying the source file until you Flush() or Close() the stream. If you can "write" to the FileStream object without it modifying the original source file, how is that achieved?

Thanks for the help

[Alsvha]

I've done something similar recently with files around 800-1000 MB and in one of the situations where I had to do some replace work on the file, then the way I did it was create a FileStreamReader and FileStreamWriter and effectively copied the file while doing my transformation work.

So I had a FileStreamReader where I called ReadLine (I could use ReadLine, as it contained line breaks) and then did the work on the line and wrote it back using my FileStreamWriter with the method WriteLine.
Alternative (if ReadLine isn't an option) I could have used MemoryStream as a buffer, but I would properly still write it back into another file using FileStreamWriter just to keep the source file intact until my transformation was done.

[RaleTheBlade]

So you copied the file into memory? Or just to a "temp" location on disk so you could modify it? My main concern is that the application reading these files is going out to clients and I don't want to eat all of their RAM up by reading these files.

[Alsvha]

So you copied the file into memory? Or just to a "temp" location on disk so you could modify it? My main concern is that the application reading these files is going out to clients and I don't want to eat all of their RAM up by reading these files.

I copied the file as I was worked on it, not prior to working. So only a small segment was read into memory at the time.

I read a line into memory.
Did my work on that line.
Wrote the line back to another file.

This left a small memory footprint because I only had one line in memory, and the IO didn't seem to be a significant factor either.
If you can't "read line", due to for example not having line breaks or similar, I would read a segment of a predetermined size into a memorystream, work on it and write it to disk.

Then when done you can delete the first file and rename the second or how you want to handle it the result.

[RaleTheBlade]

I gotcha... That was an option for us and that is basically how Im doing it now I was just curious as to how the FileStream class handled IO as far as opening a Stream, writing to the Stream, and saving the Stream. Because I know when you open a FileStream, you can make changes to the contents of the stream without changing the original source file unless Flush() or Close() is called.