Build an intelligent CSV parser?

Hi everyone,

I'm a little new to C++, so I hope I'm posing a relatively simplier problem here... Essentially, I am writing a program which will input a CSV file with multiple lines of data, 30 data values per line. I want to extract each individual data value and store it in a smaller string for analysis later. The format of the file is:

==============================================
Value01,Value02,Value03, ... Value29,Value30
Value01,Value02,Value03, ... Value29,Value30
Value01,Value02,Value03, ... Value29,Value30
...etc...
==============================================

I want to extract and store Value01 in String01, Value02 in String02, and so on.

I've figured out the mechanics of how to do this statically, i.e. manually create String01, String02, String03, etc. Each line in the CSV has 30 values, but tomorrow it may swell to 40 or 50 or 100 values per line. I'd like to do this "extract every value into a dedicated string" thing a little more intelligently. It would be great to do something like this:

==============================================
void CVS_Parser(int NumOfValues)
{
// Each row in MyFile has NumOfValues data values in it.
for(int i=0; i<NumOfValues; i++)
{
// Create string Value_${i}
}

// open MyFile, a CSV
while(not at CSF.eof)
{
// extract one line from the CSV file
for( int j=0; j<NumOfValues; j++ )
{
// Extract the j_th value from the current line
// Store that value in String_${j}
}
// Do stuff with all these data values
// Clear all strings for the next loop
}
}
==============================================

Here's the actual code I already have: (written to collect only four values, but you get the idea)

==============================================
{
int size=5000;
char * MasterString, * x, * y;
char * String01, * String02, * String03, * String04;
MasterString = (char*) malloc (sizeof(char) * 5000);
String01 = (char*) malloc (sizeof(char) * 30);
String02 = (char*) malloc (sizeof(char) * 30); // Some way to do this smarter???
String03 = (char*) malloc (sizeof(char) * 30);
String04 = (char*) malloc (sizeof(char) * 30);

fstream f("MyFile", ios::in | ios::binary);

while (!f.eof())
{
// Clear all strings...
memset(MasterString, 0, 5000);
memset(String01, 0, 30);
memset(String02, 0, 30); // Again - is there a smarter way???
memset(String03, 0, 30);
memset(String04, 0, 30);

// Extract one line from the CSV file...
f.getline(MasterString, size);

// Reset pointers x and y to the beginning of MasterString:
x = y = MasterString;

// Extract first value
y = strcasestr(MasterString, ",");
String01 = strncpy(String01, MasterString, y-MasterString);
cout<<String01<<" -- ";

// Extract second value
y = y+1; x = y;
y = strcasestr(x, ",");
String02 = strncpy(String02, x, y-x);
cout<<String02<<" -- ";

// Extract third value
y = y+1; x = y;
y = strcasestr(x, ","); // Code for extracting Value02 is exactly the same
String03 = strncpy(String03, x, y-x); // as code for extracting Value03
cout<<String03<<" -- ";

// Extract LAST value
y = y+1; x = y;
y = strrchr( y, '\0' );
String04 = strncpy(String04, x, y-x);
cout<<String04<<"\n";

} // end of "while (!f.eof())" loop...!

cout<<"Done!\n";
}

==============================================

Oh god, my eyes!

I'm just kidding. But honestly, that program is 99% C and 1% C++. Forget everything you know about arrays, malloc, memset etc.

You need to learn about std::string. With this little baby, no need to manage memory, to manage size etc.

For all your input ouput needs, I'm sure this article should give you great insight on how to do it the smart way.

My quick solution:

Code:

---

void readCSV(const string &filename, vector< vector<string> > &filecontents)
{
    ifstream in(filename.c_str());
    string line;
    vector<string> valrow;

    filecontents.clear();

    while (getline(in, line))
    {
        istringstream linestream(line);
        string value;
        valrow.clear();
        while (getline(linestream, value, ','))
            valrow.push_back(value);
        file_contents.push_back(valrow);
    }
}

---

I'll leave it as an exercise to figure out precisely what this is doing and how. I'm also assuming that you don't need to worry about quotation marks or values containing commas; if you do, things get a bit more complicated.