Read from file optimization

[Skizmo]

It's not a setting or something. It's simply the size of 1 read action.

char *pcBuffer = new char[100000];
CFile clFile;
long lRead;

if (clFile.Open ("mytextfile.txt", CFile::modeRead))
{
  do
  {  
    lRead = clFile.Read (pcBuffer, sizeof(pcBuffer));
    ////
    // at this point you process the buffer to look for the end-of-line '\r\n'
    // so you can process each line
    ////
  }
  while (lRead > 0); // keep reading until there is no more data to process.

  clFile.Close ();
}
delete []pcBuffer;

In this case it reads blocks of 100000 bytes per read and process the data in memory. Doing all the stuff in memory is much faster than reading it line by line, because the hard disk is the slowest element in your process.

[m_power_hax]

How can i send the value stored in the buffer to the GLfloat g_vertices and g_colors? Is the same thing as before : loadData >> g_vertices[m]??

[OReubens]

That's where you loose the convenience of a class like ifstream which does all that for you.
you'll have to create your own parsing at this point.

[m_power_hax]

That's where you loose the convenience of a class like ifstream which does all that for you.
you'll have to create your own parsing at this point.

Well do you have any idea on how to do the parsing in my case?
My text file is something like that : x,y,z,A.
x1,y1,z1,A1
x2,y2,z2,A2
x3,y3,z3,A3
...
All x,y,z in g_vertices array.
All A in g_colors array.
So g_vertices array should be like this : {x1,y1,z1, x2,y2,z2, x3,y3,z3, ...} and g_colors array should be like this : {A1,A1,A1, A2,A2,A2, A3,A3,A3,... }.

[itsmeandnobodyelse]

You can use stringstreams for having the same kind of parsing you had when reading the file line by line in text mode.

#include <sys/stat.h>
#include <fstream>
#include <stream>
#include <string>

...
     struct stat fs = { 0 };
     if (stat(szTextFile, &fs) != 0)
          return 2;  // file not found
     std::string buffer;
     buffer.resize[fs.st_size+2];  // that can fail if the file is too huge
     std::ifstream ifs(szTextFile, std::ios::binary | std::ios::in);
     if (!ifs.read(&buffer[0], fs.st_size))
         return 3;  // cannot read file  
     ifs.close();
     // coming here the whole file is in buffer
     // we add an additional CRLF pair for easier parsing
     buffer[fs.st_size] = '\r';
     buffer[fs.st_size+1] = '\n';

     size_t pos1 = 0;
     size_t pos2 = 0;
     int m = 0; int k = 0;
     while ((pos2 = buffer.find("\r\n", pos1)) != std::string::npos)
     {
           // ignore empty lines
           if (pos2 == pos1)
           {
                pos1 += 2;
                continue;
           } 
           // extract text line from buffer 
          std::string line = buffer.substr(pos1, pos2-pos1);
          // feed stringstream with line
          std::istringstream iss(line);
          iss >> g_vertices[m++];
          iss >> g_vertices[m++];
          iss >> g_vertices[m++];
          iss >> g_colors[k++];
          pos1 = pos2 + 2;
     }

[m_power_hax]

You can use stringstreams for having the same kind of parsing you had when reading the file line by line in text mode.

#include <sys/stat.h>
#include <fstream>
#include <stream>
#include <string>

...
     struct stat fs = { 0 };
     if (stat(szTextFile, &fs) != 0)
          return 2;  // file not found
     std::string buffer;
     buffer.resize[fs.st_size+2];  // that can fail if the file is too huge
     std::ifstream ifs(szTextFile, std::ios::binary | std::ios::in);
     if (!ifs.read(&buffer[0], fs.st_size))
         return 3;  // cannot read file  
     ifs.close();
     // coming here the whole file is in buffer
     // we add an additional CRLF pair for easier parsing
     buffer[fs.st_size] = '\r';
     buffer[fs.st_size+1] = '\n';

     size_t pos1 = 0;
     size_t pos2 = 0;
     int m = 0; int k = 0;
     while ((pos2 = buffer.find("\r\n", pos1)) != std::string::npos)
     {
           // ignore empty lines
           if (pos2 == pos1)
           {
                pos1 += 2;
                continue;
           } 
           // extract text line from buffer 
          std::string line = buffer.substr(pos1, pos2-pos1);
          // feed stringstream with line
          std::istringstream iss(line);
          iss >> g_vertices[m++];
          iss >> g_vertices[m++];
          iss >> g_vertices[m++];
          iss >> g_colors[k++];
          pos1 = pos2 + 2;
     }

Thanks for that example. I'll try it for sure.
One thing, is not the istringstream what slowing the reading? So by doing it like you did, will i see a good improvement over my method?

[m_power_hax]

	struct stat fs = { 0 };
	if (stat("blob3.txt", &fs) != 0)
		return false;  // file not found
	string buffer;
	buffer.resize(fs.st_size+2);  // that can fail if the file is too huge
	std::ifstream ifs("blob3.txt");
	if (!ifs.read(&buffer[0], fs.st_size))
		return false;  // cannot read file

I don't know why, but this line "if (!ifs.read(&buffer[0], fs.st_size))" is always false, even if i see the values from the text file in &buffer in debug mode.

I also changed the line buffer.resize[fs.st_size+2]; to buffer.resize(fs.st_size+2); because with the [] i got errors.

[itsmeandnobodyelse]

Look you opened the file as text file. That way Windows turns CRLF pairs to LF only what makes the input smaller. Hence it always tries to read above file boundary what gives the error.

The istringstream statements are compared to file/io about 100 times faster. If at all you only will get performance wins of milliseconds.

[m_power_hax]

Look you opened the file as text file. That way Windows turns CRLF pairs to LF only what makes the input smaller. Hence it always tries to read above file boundary what gives the error.

The istringstream statements are compared to file/io about 100 times faster. If at all you only will get performance wins of milliseconds.

Ok well, i'm trying to see how to convert the txt file to binary for now. It would probably improve the speed. As for using file/io, i'm still don't know how to write the data from the .txt to the arrays using your code (with the missing line for reading the data).

[itsmeandnobodyelse]

Ok well, i'm trying to see how to convert the txt file to binary for now. It would probably improve the speed. As for using file/io, i'm still don't know how to write the data from the .txt to the arrays using your code (with the missing line for reading the data).

????

Why not simple copy-paste my code?

You opened the file by

std::ifstream ifs("blob3.txt");

and you need to open it

std::ifstream ifs("blob3.txt", std::ios::binary | std::ios::in);

Otherwise your read would encounter the <EOF> what make the ifstream.read return NULL when tested with NOT operator ! .

[m_power_hax]

I thought i had to remove the binary since the file is a .txt.

[itsmeandnobodyelse]

Windows stores text files by turning each \n (linefeed char LF = 0x0A) char into a pair \r\n (carriage-return+linefeed CRLF = 0x0D0A). When you read the file in text mode it makes the reverse conversion CRLF->LF. Therefore the file size - you can retrieve by the stat function - would be the stored size and includes one additional byte for the CR of each text line. So when we read in binary mode we have the advantage that the size we got from stat matches exactly to the contents we get by the read. Moreover, for parsing purposes the CRLF sequence is as good as the single LF if not better.

[m_power_hax]

I see. I'm getting an error when using your code. Probably as something to do with the last line in my text file since it bugs after that last line. It is still slow to read the file using this methog. I just found a demo on google where the code load a 16 mb .raw file almost instantly. It is quite fast. To bad i don't have a raw file! Here his code :

    ifstream volFile("data.raw", ios::binary | ios::in);
    if (!volFile) {
        throw runtime_error("failed to load volume file");
    }

    const int volSize = 128 * 128 * 128;
    vector<GLubyte> volData(volSize);
    volFile.read(reinterpret_cast<char*>(&volData[0]), volSize);

[itsmeandnobodyelse]

How big is the file? Try to find out what the error is by including <errno.h> and printing errno after fail.

Can you post the full code ?

[m_power_hax]

bool Init()
{
	g_vertices = new GLfloat[2097152];
	g_colors = new GLfloat[2097152];
	struct stat fs = { 0 };
	if (stat("blob3.txt", &fs) != 0)
	  return false;  // file not found
	std::string buffer;
	buffer.resize(fs.st_size+2);  // that can fail if the file is too huge
	std::ifstream ifs("blob3.txt", std::ios::binary | std::ios::in);
	if (!ifs.read(&buffer[0], fs.st_size))
	 return false;  // cannot read file  
	ifs.close();
	// coming here the whole file is in buffer
	// we add an additional CRLF pair for easier parsing
	buffer[fs.st_size] = '\r';
	buffer[fs.st_size+1] = '\n';

	size_t pos1 = 0;
	size_t pos2 = 0;
	int m = 0; int k = 0;
	while ((pos2 = buffer.find("\r\n", pos1)) != std::string::npos)
	{
	   // ignore empty lines
	   if (pos2 == pos1)
	   {
			pos1 += 2;
			continue;
	   } 
	   // extract text line from buffer 
	  std::string line = buffer.substr(pos1, pos2-pos1);
	  // feed stringstream with line
	  std::istringstream iss(line);
	  iss >> g_vertices[m++];
	  iss >> g_vertices[m++];
	  iss >> g_vertices[m++];
	  iss >> g_colors[k++];
	  pos1 = pos2 + 2;
	}
}

But like i said, this solution doesn't seem quick enough.