Using ifstream, string stream, and extraction operator >>

[WhatNow46]

Let's say I have a file that is entirely strings but it's binary due to some nulls ('\0') in between the strings. So technically, it's not an ascii file (32 - 128), it's got some 0's in it. So using, cat, it would be cat -A thefile.txt to see that there are nulls in it. cat thefile.txt would just show the pure ascii text.

I can create an ifstream object and read it in line by line using either of the following:

#include (the proper include files, sorry I've omitted them here)

using namespace std;
string buffer;
ifstream if;

getline(if, buffer, '\n'); or getline(if, buffer);

or:

if >> buffer;

The above 3 different methods will ALL read an entire line from the file. I can easily
prove this by cout << "buffer = " << buffer; and it will display the line from the file,
including any nulls or whatever.

But now suppose, I want to use extraction (as in I want to tokenize the lines into parsed strings, right into string variables).

My problem is that stringstream does NOT behave in this manner unless you have a true mixture of binary globs of data. As soon as you stringstream extract into a string variable, it reverts right back to the behavior of the getline and grabs
the entire line.

Is there anyway I can use extraction and have it understand that I want ONE string
at a time? For example, by setting a delimiter?

I know I can set a delimiter in the getline function. It's just that I want to do it with the extraction operator (ie. modify stringstream and/or ifstream's extraction).

Or is there yet another class that I'm missing? I do not want to use boost in this project. And I do not want only Windows-specific classes - I want something generic.
If I really do have to "roll my own", then fine, I have to do that.

In my case, if I DO have to roll a new extraction or overload the behavior, it will need to be done for all types, not just strings - I would guess.

Thanks.

[2kaud]

What compiler are you using?

if >> buffer;

Extracts from the stream ignoring any initial white space chars until a white space char is found. Thus if the file contains

qwe rty

then the above would only extract qwe to buffer.

If you want to change what are the white space chars used by >> then AFAIK this can't be done.

[WhatNow46]

I'm using GNU, currently without c11 extensions. I was hoping that stringstream was different than ifstream as
far as tokenizing. I want a tokenizer type of stream operator.

The text in the file looks more like this:

The cat^@dog^@ which if just outputting with cat would look like this:

The catdog

The ^@ is how text-file dump utilities display nulls ('\0').

So it is a text file with nulls in-between the strings and it can have ALL the spaces it wants in each string.

I would think I can change the behavior that it simply reads in an entire line. I just wondered if it's been done already.
If not, I need to write my own extractor. I might just need to create a class that has that behavior. I will start working on one later today if noone informs me of an otherwise solution.

[2kaud]

Basically you want to extract each element separated by a null? Not using >>. It treats \0 as being part of a string and not a white space. So in the above example if>> buffer would set buffer to cat\0dog\0. If you want to extract each element as delimited by \0 then you'll need to write your own. There may well be one already written somewhere. Have you tried searching the internet?

PS a CSV parser may be what you're after. I've just tried cat\0dog\0 with my basic csv parser with delimiter set to \0 and it extracted each element fine.

[WhatNow46]

Thanks for the info - fixed me right up. I just found a snippet of C++ code that does the job nicely. Now I can just
hand a buffer of '\0' delimited strings, and it fills up a vector with each string in the line. I had wanted to use an operator to do this but I guess I need to somehow start with this for now. Otherwise, I'm too worrisome over such a detail as whether I call extraction or make a function call. At least I have the information read in now.

The below code was found at this URL: http://www.zedwood.com/article/cpp-csv-parser

Declarations:
std::vector<std::string> csv_read_row(std::istream &in, char delimiter);
std::vector<std::string> csv_read_row(std::string &in, char delimiter);

Implementations:
std::vector<std::string> csv_read_row(std::string &line, char delimiter)
{
std::stringstream ss(line);
return csv_read_row(ss, delimiter);
}

std::vector<std::string> csv_read_row(std::istream &in, char delimiter)
{
std::stringstream ss;
bool inquotes = false;
std::vector<std::string> row;//relying on RVO
while(in.good())
{
char c = in.get();
if (!inquotes && c=='"') //beginquotechar
{
inquotes=true;
}
else if (inquotes && c=='"') //quotechar
{
if ( in.peek() == '"')//2 consecutive quotes resolve to 1
{
ss << (char)in.get();
}
else //endquotechar
{
inquotes=false;
}
}
else if (!inquotes && c==delimiter) //end of field
{
row.push_back( ss.str() );
ss.str("");
}
else if (!inquotes && (c=='\r' || c=='\n') )
{
if(in.peek()=='\n') { in.get(); }
row.push_back( ss.str() );
return row;
}
else
{
ss << c;
}
}
}