Read from file optimization

[m_power_hax]

The code below is working as intended. It actually takes between 8-10 seconds to read the file (50mb) and write the data in the array (g_vertices and g_colors). After that i do a threshold to remove the values below amplitude 0.80, and put the values in g_verticesT and g_colorsT. I would like to know if anyone know how to optimize my code.
The text file is like this (x,y,z, Amplitude(for the color)) :
1.00 1.00 1.00 0.00
1.00 2.00 1.00 0.00
1.00 3.00 1.00 0.00
1.00 4.00 1.00 0.00
1.00 5.00 1.00 0.00
1.00 6.00 1.00 0.00
1.00 7.00 1.00 0.00
1.00 8.00 1.00 0.00
1.00 9.00 1.00 1.00
1.00 10.00 1.00 0.00
1.00 11.00 1.00 0.00
1.00 12.00 1.00 0.00
1.00 13.00 1.00 1.00
1.00 14.00 1.00 1.00
1.00 15.00 1.00 0.00
1.00 16.00 1.00 0.00

int lineCount=0;
	string t;
	ifstream loadData("blob3.txt");

	if (!loadData) 
	{
	   throw std::runtime_error("Can't open text file.");
    }
    else 
    {
		int i = 0;
        int j = 0;
		int k = 0;
		int m = 0;
		cout.precision(2);
		while(getline(loadData, t, '\n'))
		{
			++lineCount;
		}
		loadData.clear();
		loadData.seekg(0, ios::beg);

		const int arraySize = lineCount*3;
		const int textFileSize = lineCount*4;
		g_vertices = new GLfloat[arraySize];
		g_colors = new GLfloat[arraySize];
		while (j<(textFileSize))
        {	
			for(i=0; i<3;i++)
			{
				loadData >> g_vertices[m];
				m++;
			}
			loadData >> g_colors[k];
			g_colors[k + 1] = g_colors[k];
			g_colors[k + 2] = g_colors[k];
			k=k+3;
			j=j+4;
		}
    }
    loadData.close();

	g_verticesT = new GLfloat[sizeVertices];
	g_colorsT = new GLfloat[sizeVertices];
	int numPointThreshold1 = 0;
	int numPointThreshold2 = 0;
	int jj = 0;
	int ii = 0;
	for (ii=0 ; ii<sizeVertices ; ii=ii+3)
	{
		if (!g_colors[ii])
		{
			numPointThreshold1++;
			continue;
		}
		else if (g_colors[ii]>=0.80)
		{
			numPointThreshold2++;
			g_verticesT[jj]= g_vertices[ii];
			g_verticesT[jj+1]= g_vertices[ii+1];
			g_verticesT[jj+2]= g_vertices[ii+2];
			g_colorsT[jj]= g_colors[ii];
			g_colorsT[jj+1]= g_colors[ii+1];
			g_colorsT[jj+2]= g_colors[ii+2];
			jj=jj+3;
		}
	}
	_TCHAR szBuffer[100];
	_stprintf(szBuffer, _T("%i,%i"), numPointThreshold1,numPointThreshold2);
	MessageBoxA(0, szBuffer, "Verification", MB_OK|MB_ICONINFORMATION);

[OReubens]

it depends how far you want to go to improve speed... In order of what I think will be the biggest impact:

1) You're reading the file twice, only read it once.
Don't use fixed size arrays but use a container class, give it enough of a initial size to make resizing unlikely/impossible. Even a single resize may entirely work against the "read once" rule. You can more or less guess how large the array needs to be by dividing the file size by a certain amount, how much extra memory you can afford will be key here. by what you listed here, the largest array would be size of the file divided by the smallest possible size of a line, a line containing all zero's ("0.00 0.00 0.00 0.00\r\n", or 22 chars per line).
you're removing parts of it, so you could be a lot more memory conservative depending on what percentage you can expect to get removed.
2) Don't store intermediates in a string, especially if you aren't even using the string (getline).
3) don't use ifstream, instead use regular Win API file handles, read 'fairly large' buffers, and hard-parse the data yourself, from the buffer right to the target data. Splitting in substrings tends to be a killer. This'll move away from the C++, be unportable (windows specific). This will be preparation for the next step which will make quite a change.
4) Use overlapped I/O in a double buffering system. That way you can parse one buffer of data, while the other one is being read with more data. This is a BIG thing, depending on your HD speed, you're likely capable of processing the data faster than the HD can read it, effectively making this all about how fast the PC can read the data from disk.
5) don't parse data to store it to then throw it away. Do the throwing away part before so you don't even need to add it to the array in the first place. This is particularly imporant in the case of the 1 pass over the data, since storing less data will help reduce the chance of needing to resize the container.

I have done similar projects. Large text files with similar numeric data that needed to be read and then processed for calculation.
On a PC that's currently 'outdated', it could run through 250Mb of data per second. Which was very close to the limit of what the HD can handle, so it was very close to being entirely I/O bound.
That appears to be 40 to 50 times (yes, times, not percent) faster than what you're getting now. It wasn't the exact same layout of data, but it's fairly close, so yes, depending on how much effort you're prepared to put into this, it could be made quite a bit faster.

The question then is.. how much development time are you prepared to spend to speed this up.

[m_power_hax]

it depends how far you want to go to improve speed... In order of what I think will be the biggest impact:

1) You're reading the file twice, only read it once.
Don't use fixed size arrays but use a container class, give it enough of a initial size to make resizing unlikely/impossible. Even a single resize may entirely work against the "read once" rule. You can more or less guess how large the array needs to be by dividing the file size by a certain amount, how much extra memory you can afford will be key here. by what you listed here, the largest array would be size of the file divided by the smallest possible size of a line, a line containing all zero's ("0.00 0.00 0.00 0.00\r\n", or 22 chars per line).
you're removing parts of it, so you could be a lot more memory conservative depending on what percentage you can expect to get removed.
2) Don't store intermediates in a string, especially if you aren't even using the string (getline).
3) don't use ifstream, instead use regular Win API file handles, read 'fairly large' buffers, and hard-parse the data yourself, from the buffer right to the target data. Splitting in substrings tends to be a killer. This'll move away from the C++, be unportable (windows specific). This will be preparation for the next step which will make quite a change.
4) Use overlapped I/O in a double buffering system. That way you can parse one buffer of data, while the other one is being read with more data. This is a BIG thing, depending on your HD speed, you're likely capable of processing the data faster than the HD can read it, effectively making this all about how fast the PC can read the data from disk.
5) don't parse data to store it to then throw it away. Do the throwing away part before so you don't even need to add it to the array in the first place. This is particularly imporant in the case of the 1 pass over the data, since storing less data will help reduce the chance of needing to resize the container.

I have done similar projects. Large text files with similar numeric data that needed to be read and then processed for calculation.
On a PC that's currently 'outdated', it could run through 250Mb of data per second. Which was very close to the limit of what the HD can handle, so it was very close to being entirely I/O bound.
That appears to be 40 to 50 times (yes, times, not percent) faster than what you're getting now. It wasn't the exact same layout of data, but it's fairly close, so yes, depending on how much effort you're prepared to put into this, it could be made quite a bit faster.

The question then is.. how much development time are you prepared to spend to speed this up.

Thanks for the great tips. Right now the 50mb file is only a testing file. The real file will be much bigger, some might be between 300 to 400mb. Check your pm.

[Skizmo]

The real file will be much bigger, some might be between 300 to 400mb.

Reading the file per 5/10 mb block is always faster than reading it line by line. The only problem you have is that you will have to detect the lines '\r\n' yourself. Compared to the speed you will gain, rewriting that is not really a big job.

[m_power_hax]

My code was made only to work, not for optimization, so here what i need to do :
-Line in text file are as follow (x,y,z,Amplitude)
-I need to read the file as quickly as possible
-User determine the threshold (something like the user only want to see particle with an amplitude over 0.5)
-Read file
-Send position x,y,z to g_vertices when the value of amplitude correspond to the threshold
-Send value of amplitude x3 to g_colors to be able to use the amplitude as a color.
-Send g_vertices and g_colors to a VBO
-Draw the particle

[OReubens]

if you have control over the side that creates the data. You can also save a lot of performance by making fixed size text records, or even better, use binary records.

Reading large buffers is the biggest key issue. But speed gains degrade rappidly on larger sizes. Changing a 1K buffer to a 64k buffer, will be a much much bigger impact than changing a 64K buffer to a 4Mb buffer, even though it both is a x64 increase in size.
My personal experience is that it's rarely worth the extra memory using buffers above 64K. Especially if you intend to use overlapped I/O, it's typically better to use several smaller buffers and chain them rather than trying to work with 1 or 2 large buffers.

[m_power_hax]

if you have control over the side that creates the data. You can also save a lot of performance by making fixed size text records, or even better, use binary records.

Reading large buffers is the biggest key issue. But speed gains degrade rappidly on larger sizes. Changing a 1K buffer to a 64k buffer, will be a much much bigger impact than changing a 64K buffer to a 4Mb buffer, even though it both is a x64 increase in size.
My personal experience is that it's rarely worth the extra memory using buffers above 64K. Especially if you intend to use overlapped I/O, it's typically better to use several smaller buffers and chain them rather than trying to work with 1 or 2 large buffers.

Files will come from matlab so, i could probably use binary instead. So anyone here has a short example on how to use overlapped I/O to read from a text file?

[itsmeandnobodyelse]

it depends how far you want to go to improve speed... In order of what I think will be the biggest impact:

1) You're reading the file twice, only read it once.
Don't use fixed size arrays but use a container class, give it enough of a initial size to make resizing unlikely/impossible. Even a single resize may entirely work against the "read once" rule. You can more or less guess how large the array needs to be by dividing the file size by a certain amount, how much extra memory you can afford will be key here. by what you listed here, the largest array would be size of the file divided by the smallest possible size of a line, a line containing all zero's ("0.00 0.00 0.00 0.00\r\n", or 22 chars per line).
you're removing parts of it, so you could be a lot more memory conservative depending on what percentage you can expect to get removed.
2) Don't store intermediates in a string, especially if you aren't even using the string (getline).
3) don't use ifstream, instead use regular Win API file handles, read 'fairly large' buffers, and hard-parse the data yourself, from the buffer right to the target data. Splitting in substrings tends to be a killer. This'll move away from the C++, be unportable (windows specific). This will be preparation for the next step which will make quite a change.
4) Use overlapped I/O in a double buffering system. That way you can parse one buffer of data, while the other one is being read with more data. This is a BIG thing, depending on your HD speed, you're likely capable of processing the data faster than the HD can read it, effectively making this all about how fast the PC can read the data from disk.
5) don't parse data to store it to then throw it away. Do the throwing away part before so you don't even need to add it to the array in the first place. This is particularly imporant in the case of the 1 pass over the data, since storing less data will help reduce the chance of needing to resize the container.

I have done similar projects. Large text files with similar numeric data that needed to be read and then processed for calculation.
On a PC that's currently 'outdated', it could run through 250Mb of data per second. Which was very close to the limit of what the HD can handle, so it was very close to being entirely I/O bound.
That appears to be 40 to 50 times (yes, times, not percent) faster than what you're getting now. It wasn't the exact same layout of data, but it's fairly close, so yes, depending on how much effort you're prepared to put into this, it could be made quite a bit faster.

The question then is.. how much development time are you prepared to spend to speed this up.

I wouldn't push too much efforts into that and doubt that you can gain much more than 50 percent of your current times. For example, the fastest SATA hard drive I could found is the Western Digital Veloci Raptor. It has a maximum transfer rate of 142.5 MByte/sec and an average transfer rate of 120.9 MByte/sec . That is less than the half what OReubens has told (250 MB) and you only can get that by reading the disk in native mode not using files of a file system where additional overhead couldn't be avoided. Moreover, the disk runs with 10,000 rotations per minute and is very loud because of that. Only SSD drives or RAM disks are silent and would be faster but you can't take such hardware/environments as a rule. Also, when reading text files it is important that those files were stored contiguously on the disk and were not fragmented into thousands of pieces. The latter happens when you were reading the original text file that was written sequentially line per line and therefore the filesystem had no chance to reserve sufficient contiguous space on the disk. But even if you copied the file from somewhere you hardly get them into one piece onto your disk and therefore you will get additional penalties.

To get a realistic upper limit for what is makeable open the text file with a good editor, e. g. notepad++ or Visual Studio (I mean to remember that it has an upper limit of 20 MB but that could have been a restriction of my system). Or if you can't open it with an editor, make a copy of the file (between two disks) at the commandline and divide the time by two.

Note the difference in speed between ReadFile (not using overlapped I/O) and ifstream::read is about 5 percent (I recently made a test program for that) what is less than the spread you reported from your tests. I don't think that it makes much difference whether a run would need 10 or 10.5 seconds, so I would put the efforts to the main points which in my opinion are to not reading the data twice and reading them in chunks as big as possible (if possible with one single read). Overlapped I/O makes sense if you want to read the data line by line as you did now. But I would say that your records are too small to gain benefit from that especially as there is not really anything you could do in the meantime between two inputs.

[m_power_hax]

That would be great if i could actually test something with some code! My previous code is working correctly, but i just don't know how to use overlapped I/O. So i need some example to get me started!

[OReubens]

I wouldn't push too much efforts into that and doubt that you can gain much more than 50 percent of your current times. For example, the fastest SATA hard drive I could found is the Western Digital Veloci Raptor. It has a maximum transfer rate of 142.5 MByte/sec and an average transfer rate of 120.9 MByte/sec . That is less than the half what OReubens has told (250 MB) and you only can get that by reading the disk in native mode not using files of a file system where additional overhead couldn't be avoided. Moreover, the disk runs with 10,000 rotations per minute and is very loud because of that. Only SSD drives or RAM disks are silent and would be faster but you can't take such hardware/environments as a rule. Also, when reading text files it is important that those files were stored contiguously on the disk and were not fragmented into thousands of pieces. The latter happens when you were reading the original text file that was written sequentially line per line and therefore the filesystem had no chance to reserve sufficient contiguous space on the disk. But even if you copied the file from somewhere you hardly get them into one piece onto your disk and therefore you will get additional penalties.

To get a realistic upper limit for what is makeable open the text file with a good editor, e. g. notepad++ or Visual Studio (I mean to remember that it has an upper limit of 20 MB but that could have been a restriction of my system). Or if you can't open it with an editor, make a copy of the file (between two disks) at the commandline and divide the time by two.

Note the difference in speed between ReadFile (not using overlapped I/O) and ifstream::read is about 5 percent (I recently made a test program for that) what is less than the spread you reported from your tests. I don't think that it makes much difference whether a run would need 10 or 10.5 seconds, so I would put the efforts to the main points which in my opinion are to not reading the data twice and reading them in chunks as big as possible (if possible with one single read). Overlapped I/O makes sense if you want to read the data line by line as you did now. But I would say that your records are too small to gain benefit from that especially as there is not really anything you could do in the meantime between two inputs.

The above machine was a machine with a RAID controller. RAID is quite common, even in low end servers and and is even becoming standard in the motherboards of a lot of desktop machines as well. SATA3 is up and arriving, giving even higher speeds.

Overlapped IO, expecially unbuffered pretty much is tied in right into the kernel, and it does give you the highest possible speed. There is as good as no overhead, and you get within a fraction of the speed that raw data reads can handle.

The big issue of using overlapped IO is not that it's a lot faster than reading the file synchronously. It is faster, but that has more to do with the requirement of aligned buffers on overlapped I/O than anything.
The BIG gains are made (if you make the code right for it) because you can process data while the I/O is happening. rather than have a
read block
process block
read block
process block
read block
That's also why it's named "overlapped IO" because the time the IO operation takes, overlaps with whatever you are doing while "waiting" for the data to actually finish reading.

You could more or less achieve the same thing with 2 threads, one for reading the file, and one for processing the data, overlapped IO does exactly that with the benefit of not needing an extra thread, and not needing communication between 2 threads. You DO need to handle the "data ready" event of course.

You can use overlapped IO in combination with multiple threads if IO is fast and processing is slow so you can split the processing over multiple CPU cores. But it works fine on a single thread as well.

[OReubens]

Files will come from matlab so, i could probably use binary instead. So anyone here has a short example on how to use overlapped I/O to read from a text file?

overlapped I/O doesn't do "text files". Every file is considered a binary one and all it can do is read chunks of said files as binary data.

It's up to the programs to "interpret" what the binary data represents. in your case text files, you will need to work through the read buffer, splitting off individual values (spaces apparently) and lines (CR/LF chars) yourself. This is where you will need the parsing I mentioned, you can't use the convenience of the ifstream << operator anymore. Which takes me back to the "how much time do you want to spend on this"

There's a working sample of overlapped IO on MSDN
http://msdn.microsoft.com/en-us/libr...(v=VS.85).aspx

[m_power_hax]

overlapped I/O doesn't do "text files". Every file is considered a binary one and all it can do is read chunks of said files as binary data.

It's up to the programs to "interpret" what the binary data represents. in your case text files, you will need to work through the read buffer, splitting off individual values (spaces apparently) and lines (CR/LF chars) yourself. This is where you will need the parsing I mentioned, you can't use the convenience of the ifstream << operator anymore. Which takes me back to the "how much time do you want to spend on this"

There's a working sample of overlapped IO on MSDN
http://msdn.microsoft.com/en-us/libr...(v=VS.85).aspx

I followed your first advice about approximating the number of line from the size of the file in order to read the file only once. I gained approximatly 1-2 seconds by doing this. An average of 8.5 seconds instead of 10.5 seconds is not bad, but i need it to be quicker! I'm ready to spend some time in order to make it quicker.

[Skizmo]

I'm ready to spend some time in order to make it quicker.

Did you already remove the 'getline' and changed it into a 5/10mb blockread? Because that will increase it a lot.

[m_power_hax]

Did you already remove the 'getline' and changed it into a 5/10mb blockread? Because that will increase it a lot.

No, i don't know how to do that yet.

[m_power_hax]

something like that? http://www.cplusplus.com/reference/i...fstream/rdbuf/

How do i set it to read 5/10mb block?