Advice On Altering a 5 Gig .CSV File Via C#

Hello,

I have a 5 gig .CSV files of financial quote data that I tried to alter programmatically in Microsoft Access. Microsoft Access kept freezing when I tried this (even when using a Quadcore computer with 24gigs of ram) so I have decided to try doing this C#, which I understand is much more robust and better suited to handle large amounts of data.

I am pretty new to C#, so any advice or code examples of how to do the following conversion would be much appreciated. I'd like to do this within Visual Studio (any version), but am open to using a different IDE if you think it will give better results.

Anyway, here is the situation:

I have a 5 gig .CSV of financial quote data that looks as follows. Note: The
“type” column signifies whether the quote was a Bid(B) or an Ask(A).

date,timestamp,price,type,volume
20151130,110342316,208650,B,287
20151130,110342360,208675,A,281
20151130,110342364,208650,B,275
20151130,110342366,208650,B,273
20151130,110342408,208675,A,280

I need to convert this data so that each row shows both the
Bid price/volume and the Ask price/volume (with a new row being
created each time the Bid and/or Ask has a change in price and/or volume).

For instance, the example above would look as
follows after the conversion (note: the first row from the example
above could not be included below because it did not have a previous
row to get the Ask price and volume from).

date,timestamp,bidprice,bidvolume,askprice,askvolume
20151130,110342360,208650,287,208675,281
20151130,110342364,208650,275,208675,281
20151130,110342366,208650,273,208675,281
20151130,110342408,208650,273,208675,280

I look forward to your thoughts.

Thanks!

Hi.

5GB is huge...

Here is an excellent article that might help you with the logic. The article is in VB.NET though, but the coding concepts should be easy enough to understand.

http://www.codeguru.com/cpp/w-d/doc_...File-Limit.htm

Look at System.IO.FileInfo

You can read the file line by line using:
System.IO.StreamReader

You can create a new file line by line using:
System.IO.StreamWriter