Getting data in to C++ from excel, any help would be greatly appreciated

Hi there,

I am very very new to C++. A bit of background. I have been writing in excel vba for large number crunching, and the code is now taking quite a while to run. A friend of mine suggested i start writing in C++, so i read up on it. And downloaded Code:Blocks.

My VBA Code is:

Code:

---


Sub arrayss()

Dim NameArray As Variant
Dim datarray As Byte
Dim DirectionArray As Variant
Dim WinArr As Variant
Dim Results As Variant
Dim iColumns As Long
Dim jRows As Long
Dim a As Long
Dim b As Long
Dim c As Long
Dim q As Long
Dim tradevalue As Long
Dim noofrowsWin As Long
Dim noofcolumnsWin As Long
Dim cell1position As Long
Dim cell2position As Long
Dim cell3position As Long
Dim cell4position As Long
Dim cell1value As Long
Dim cell2value As Long
Dim cell3value As Long
Dim cell4value As Long
Dim y As Long
Dim z As Long



'Range("AN:BBB").Delete

Application.Calculation = xlCalculationManual

dataarray = ThisWorkbook.Names("Data1").RefersToRange.Value
NameArray = ThisWorkbook.Names("NameArray").RefersToRange.Value
DirectionArray = ThisWorkbook.Names("DirectionArray").RefersToRange.Value

cell1position = Sheets("Test").Cells(2, 3)
cell2position = Sheets("Test").Cells(3, 3)
cell3position = Sheets("Test").Cells(4, 3)
cell4position = Sheets("Test").Cells(5, 3)
cell1value = Sheets("Test").Cells(7, 3)
cell2value = Sheets("Test").Cells(8, 3)
cell3value = Sheets("Test").Cells(9, 3)
cell4value = Sheets("Test").Cells(10, 3)
tradevalue = Sheets("Test").Cells(12, 3)
jRows = 1
iColumns = 1
a = 3
b = 1
y = 0
z = 0
i = 1
j = 1
Sheets("Final").Cells(3, 1) = "=Counta(2:2)-1"
Sheets("Final").Cells(1, 2) = "=COUNTA(A:A)"
noofcolumnsWin = ((Sheets("Final").Cells(3, 1)) - 1) ^ 2 - (Sheets("Final").Cells(3, 1) - 1)
noofrowsWin = Sheets("Final").Cells(1, 2) + 1

ReDim Results(1 To 500, 1 To 16)
For q = 1 To UBound(dataarray, 2) - 1
   
    For c = 1 To UBound(dataarray, 2) - 1
        If c = q Then
        GoTo labelend3
        End If
        For jRows = 1 To UBound(dataarray, 1) - cell4position
            If IsEmpty(dataarray(jRows, q)) Or DirectionArray(jRows, 1) = 15 Or DirectionArray(jRows, 1) = 45 Then
            ElseIf dataarray(jRows + cell1position, c) = cell1value And dataarray(jRows + cell2position, c) = cell2value And dataarray(jRows + cell3position, c) = cell3value And dataarray(jRows + cell4position, c) = cell4value Then                ' cell check
                If tradevalue = dataarray(jRows, q) Then            ' win or lose
                    y = y + 1
                Else
                    z = z + 1
                End If
            End If
        a = a + 1
labelend:
        Next
       
        If z = 0 Then
            z = 1
        End If
       
        If y = 0 Then
            y = 1
        End If
       
        If y > 20 And y / z > 5 Then
            Results(i, j) = NameArray(1, q)
            Results(i, j + 1) = NameArray(1, c)
            Results(i, j + 2) = y
            Results(i, j + 3) = z
            Results(i, j + 4) = y / z
            Results(i, j + 5) = cell1position
            Results(i, j + 6) = cell2position
            Results(i, j + 7) = cell3position
            Results(i, j + 8) = cell4position
            Results(i, j + 9) = ""
            Results(i, j + 10) = cell1value
            Results(i, j + 11) = cell2value
            Results(i, j + 12) = cell3value
            Results(i, j + 13) = cell4value
            Results(i, j + 14) = ""
            Results(i, j + 15) = tradevalue
            i = i + 1
        End If
       
        y = 0
        z = 0
        a = 3
        b = b + 1
        '''  PUT IN HERE THE TEST FOR COUNTING 1's and zeros or below lol
        ''''
labelend3:
    Next
a = 3
labelend1:
Next


Sheets("Test").Select
Range("an:an").End(xlDown).Offset(1, 0).Select
a = ActiveCell.Row
Range(Range("an" & a & ""), Range("an" & a & "").Offset(500, 16)).Value = Results
 
'Range(Range("m1"), Range("m1").Offset(10000, 16)).Value = Results
'Range(Range("g1"), Range("g1").Offset(noofrowsWin - 1, noofcolumnsWin - 1)).Value = WinArr
'Range(Range("g1"), Range("g1").Offset(noofrowsWin - 1, noofrowsWin - 1)).Value = Application.Transpose(WinArr)
   
Application.Calculation = xlCalculationAutomatic


End Sub

---

Ultimately i would like to recode this to C++, but my first and probably silly question is how do i get the data from Excel to use in C++. I was thinking either to put the data in 3 csv files and convert into three Arrays in C++. Or maybe create a library of the data in C++. Ultimately it is speed i am looking for, so before i start recoding i wanted to start with the best way.

Does that make sense?

The data is like this in excel: (don't know how to create a table)

ABC DCA GFE THE
15 0 1 0 1
30 1 0 1 0
45 1 1 0
00 0 0 1
15 1 1 0
15 1 0 0 1
15 0 1 1 1
30 1 0 0
45 1 1 1

So the headers would be in one array, the 15's, 30's etc would be in another array and the 1's and 0's and Empty ( i need it to record an empty cell) would be in another array

Rgds

Surreall

PS and thankyou for your time

Quote:

---

Originally Posted by Surreall

Hi there,

I am very very new to C++. A bit of background. I have been writing in excel vba for large number crunching, and the code is now taking quite a while to run. A friend of mine suggested i start writing in C++, so i read up on it. And downloaded Code:Blocks.

---

C++ is not a trivial language. You should learn very basics first before even thinking about "speed".

Second, C++ isn't like VBA or some other language, where you have a lot of firewalls and escape hatches when you make a mistake. You make a mistake in VBA such as an out of bounds access, you get a nice error message. You make the same type of mistake in C++, you may not get any error, and your program seems to run, but with that fault in it that you don't know about.

Quote:

---

Ultimately it is speed i am looking for, so before i start recoding i wanted to start with the best way.

---

Honestly, that whole idea of trying to recode something in C++ without knowing C++ is IMO wrought with all sorts of problems. Again, C++ is not a trivial language, it has a lot of nuances that the beginner is not aware of, and requires experience to understand how to write code that is not only efficient, but maintainable.

There are tons of CSV parsers written in C++, but you need to understand C++ to use them.

Regards,

Paul McKenzie

Quote:

---

Originally Posted by Paul McKenzie

C++ is not a trivial language. You should learn very basics first before even thinking about "speed".

Second, C++ isn't like VBA or some other language, where you have a lot of firewalls and escape hatches when you make a mistake. You make a mistake in VBA such as an out of bounds access, you get a nice error message. You make the same type of mistake in C++, you may not get any error, and your program seems to run, but with that fault in it that you don't know about.
Honestly, that whole idea of trying to recode something in C++ without knowing C++ is IMO wrought with all sorts of problems. Again, C++ is not a trivial language, it has a lot of nuances that the beginner is not aware of, and requires experience to understand how to write code that is not only efficient, but maintainable.

There are tons of CSV parsers written in C++, but you need to understand C++ to use them.

Regards,

Paul McKenzie

---

I understand i need to learn C++ first

But not to worry i have been reading up about reading from csv file. And have written code that reads all the values from the csv file, still a long way to go. Was just wondering what would be quick way of doing it for future reference as the data is getting big quick. But as you say i need to learn C++ first. I will go away and learn it. Sorry to bother you with silly questions, my mistake

Rgds

Surreall

Quote:

---

Originally Posted by Paul McKenzie

C++ is not a trivial language. You should learn very basics first before even thinking about "speed".

Second, C++ isn't like VBA or some other language, where you have a lot of firewalls and escape hatches when you make a mistake. You make a mistake in VBA such as an out of bounds access, you get a nice error message. You make the same type of mistake in C++, you may not get any error, and your program seems to run, but with that fault in it that you don't know about.
Honestly, that whole idea of trying to recode something in C++ without knowing C++ is IMO wrought with all sorts of problems. Again, C++ is not a trivial language, it has a lot of nuances that the beginner is not aware of, and requires experience to understand how to write code that is not only efficient, but maintainable.

There are tons of CSV parsers written in C++, but you need to understand C++ to use them.

Regards,

Paul McKenzie

---

I understand i need to learn C++ first

But not to worry i have been reading up about reading from csv file. And have written code that reads all the values from the csv file, still a long way to go. Was just wondering what would be quick way of doing it for future reference as the data is getting big quick. But as you say i need to learn C++ first. I will go away and learn it. Sorry to bother you with silly questions, my mistake

Rgds

Surreall

So do you need to read CSV files or directly from Excel? If it's CSV, mentioning Excel is irrelevant, so first define your problem properly.

If you're using Visual C++, you can use ODBC and the CRecordset class to read directly, but as has been mentioned, you'll have a really steep learning curve.

Quote:

---

Originally Posted by Surreall

Ultimately i would like to recode this to C++, but my first and probably silly question is how do i get the data from Excel to use in C++. I was thinking either to put the data in 3 csv files and convert into three Arrays in C++. Or maybe create a library of the data in C++. Ultimately it is speed i am looking for, so before i start recoding i wanted to start with the best way.

---

There are three possibilities.
1. Loose coupling. You create a program in C++ that reads some files and processes the data in them. You can generate the input files with VB or whatever. This is the easiest option to implement, but it requires data exchange via disk and a fixed format for the data.
2. Run C++ DLL from VBA. You can create a C++ DLL that exports a function that does the data processing. You call this function from VBA and pass it the required data.
3. Run from C++. You can read the data from an Excel file in your C++ program. This requires some interop with Excel, or to use a library which can read Excel files.

Depending on how complex your data processing is, you may be able to achieve the first option with only a limited knowledge of C++. However, you need to master the basics well in order to be able to write highly efficient programs.

Quote:

---

Originally Posted by D_Drmmr

There are three possibilities.
1. Loose coupling. You create a program in C++ that reads some files and processes the data in them. You can generate the input files with VB or whatever. This is the easiest option to implement, but it requires data exchange via disk and a fixed format for the data.
2. Run C++ DLL from VBA. You can create a C++ DLL that exports a function that does the data processing. You call this function from VBA and pass it the required data.
3. Run from C++. You can read the data from an Excel file in your C++ program. This requires some interop with Excel, or to use a library which can read Excel files.

Depending on how complex your data processing is, you may be able to achieve the first option with only a limited knowledge of C++. However, you need to master the basics well in order to be able to write highly efficient programs.

---

Many thanks for your help.

I have written some code now....and the reading in doesnt actually take long less than half a second. So all the time is going to be taken up with anaylising the arrays i have created. So not too worried about it.

Anyways here is where i have got to now, i did make functions initally to do the each array build, but it didnt make any difference in time. So i left them now in the main body.

The part that takes a little while, although it is gone from approx 6 seconds in vba to around 0.1 seconds in C++ (very very very chuffed about c++). I would still like to get this quicker if possible.

So any suggestions for speeding up the following code would be greatly appreciated:

Code:

---

#include <fstream>
#include <iostream>
#include <sstream>
#include <string>

using namespace std;

int main()
{


// this code finds the number of rows in a csv file
ifstream data("Data1.csv");
string line;
int rows = 0;
while (getline(data,line,'\n'))
{
    rows++;
}
//cout << "Counted " << rows << " rows." << endl;
data.close();


// this code finds the number of rows in a csv file
ifstream data1("Data1.csv");
string line1;
int columns = 0;
getline(data1,line1);
stringstream ss(line1);
string field;
while (getline(ss,field,','))
    columns++;
//cout << "Counted " << columns << " columns." << endl;
data.close();



/////This populats the dataarray with data1.csv crap
int dataarray[rows][columns];

ifstream data2("Data1.csv");
string line2;
int rows1= 0;
while (getline(data2,line2,'\n'))
{
    stringstream bb(line2);
    string field1;
    int columns1 = 0;
    while (getline(bb,field1,','))
    {
        istringstream buffer(field1);
        int value = 2;
        buffer >> value;
        dataarray[rows1][columns1] = value;
        columns1++;
    }
    rows1++;
}
//cout << "Datarray element, " << dataarray[3][0] << endl;
data.close();


// do the same for directionarray
int directionarray[rows];
ifstream direction("direction.csv");
string line3;
int rows2 = 0;
while (getline(direction,line3, '\n'))
    {
    istringstream buffer1(line3);
    int value1 = 0;
    buffer1 >> value1;
    directionarray[rows2] = value1;
    rows2++;
    }
//cout << "Directionarray element, " << directionarray[0] << endl;

// do the same for headerarray
//int headerarray[columns];

string headerarray[columns];
ifstream header("Header.csv");
string line4;
int columns2 = 0;
while (getline(header,line4, ','))
{
    headerarray[columns2] = line4;
    columns2++;
}

//cout << "Headerarray element, " << headerarray[36] << endl;

//i have now created arrays, dataarray, directionarray & headerarray with correct elements

int indicator1position = 4;
int indicator2position = 5;
int indicator3position = 6;
int indicator4position = 7;
int indicator1value = 1;
int indicator2value = 1;
int indicator3value = 1;
int indicator4value = 1;

int trade1position = 4;
int trade2position = 5;
int trade3position = 6;
int trade4position = 7;
int trade1value = 1;
int trade2value = 1;
int trade3value = 1;
int trade4value = 1;

int tradevalue = 1;
int numberofcells = 4;

float y = 0;
float z = 0;

//string results[50000][18];


//for (trade1position = 4;trade1position < 12;trade1position++){

float resultsarraynumbers[3000][20];
string resultsarraywords[3000][20];
int resultsrow=0;
int resultscolumn=0;

//for (trade1position = 4;trade1position<13;trade1position++)
//{
for (trade2position = 1;trade2position<9;trade2position++)
{
for (int q = 0;q < columns - 1; q++)
    {
    for (int c = 0;c < columns - 1; c++)
        {
        if (q==c) {} else
            {
            for (int jRows = 0;jRows < rows - 1; jRows++)
                {
                if (dataarray[jRows][q]==2 || directionarray[jRows]==15 || directionarray[jRows]==45){}
                else if (numberofcells == 4)
                    {  if (dataarray[jRows + indicator1position][c]==indicator1value && \
                        dataarray[jRows + indicator2position][c]==indicator2value && \
                        dataarray[jRows + trade1position][q]==trade1value && \
                        dataarray[jRows + trade2position][q]==trade2value)
                        {
                        if (tradevalue == dataarray[jRows][q])
                            {y++;} else {z++;}
                        }
                    }
                }
                if(y==0){y = 1;} else if (z==0){z = 1;}
                if ( y/z > 1.5 && y > 10 )
                {
                resultsarraynumbers[resultsrow][resultscolumn] = y/z;
                resultsarraynumbers[resultsrow][resultscolumn +1] = y;
                resultsarraynumbers[resultsrow][resultscolumn +2] = z;
                resultsarraynumbers[resultsrow][resultscolumn +3] = indicator1position;
                resultsarraynumbers[resultsrow][resultscolumn +4] = indicator2position;
                resultsarraynumbers[resultsrow][resultscolumn +5] = trade1position;
                resultsarraynumbers[resultsrow][resultscolumn +6] = trade2position;
                resultsarraynumbers[resultsrow][resultscolumn +7] = numberofcells;
                resultsarraynumbers[resultsrow][resultscolumn +8] = indicator1value;
                resultsarraynumbers[resultsrow][resultscolumn +9] = indicator2value;
                resultsarraynumbers[resultsrow][resultscolumn +10] = trade1value;
                resultsarraynumbers[resultsrow][resultscolumn +11] = trade2value;
                resultsarraynumbers[resultsrow][resultscolumn +12] = tradevalue;

                resultsarraywords[resultsrow][resultscolumn] = headerarray[q];
                resultsarraywords[resultsrow][resultscolumn + 1] = headerarray[c];
                resultsrow++;
                }
                y = 0;
                z = 0;
            }
        }
    }
}
//}

cout << "Results number, y/z=" << resultsarraynumbers[0][0] << endl;
cout << "Results number, y=" << resultsarraynumbers[0][1] << endl;
cout << "Results number, z=" << resultsarraynumbers[0][2] << endl;
cout << "Results trade, " << resultsarraywords[0][0] << endl;
cout << "Results indicator, " << resultsarraywords[0][1] << endl;
cout << "Results, y= " << y << endl;

return 0;
}

---

As far as i can figure the bottle neck is here:

Code:

---

                if (dataarray[jRows][q]==2 || directionarray[jRows]==15 || directionarray[jRows]==45){}
                else if (numberofcells == 4)
                    {  if (dataarray[jRows + indicator1position][c]==indicator1value && \
                        dataarray[jRows + indicator2position][c]==indicator2value && \
                        dataarray[jRows + trade1position][q]==trade1value && \
                        dataarray[jRows + trade2position][q]==trade2value)
                        {
                        if (tradevalue == dataarray[jRows][q])
                            {y++;} else {z++;}
                        }
                    }

---

By the way i am loving C++ for its quickness, great stuff. And thanks for the help previously. I know my code is butchered but i have had one day of learning it so far so bear with me please

Rgds

Surreall

Quote:

---

Originally Posted by Surreall

I have written some code now....and the reading in doesnt actually take long less than half a second. So all the time is going to be taken up with anaylising the arrays i have created. So not too worried about it.

---

What exactly are you trying to accomplish with the arrays? There are probably better ways to handle the code so that it is maintainable.

Here is the issue I have with your code.

1) You have hard-coded magic numbers. What significance is 2, 15, 45, etc.?

2) This is not legal C++:

Code:

---

int dataarray[rows][columns];
//..
string headerarray[columns];

---

You cannot declare arrays using a variable as the number of rows and columns. What you are probably using is some extension of C++, not ANSI C++. You are more than likely using gcc or some compiler based on gcc. If you set the compiler switches to ANSI for that compiler, you will see that your code will fail to compile. It will definitely not compile with any version of Visual C++. Since it is an extension, you now need to read up on all the quirks and caveats of using such an extension.

The std::vector<T> class is used by standard C++ to declare a dynamic array. This works across all ANSI C++ compilers, is standard, is guaranteed to be on every ANSI C++ compiler, is documented on how it must work, etc.

3)

Code:

---

resultsarraynumbers[resultsrow][resultscolumn] = y/z;
                resultsarraynumbers[resultsrow][resultscolumn +1] = y;
                resultsarraynumbers[resultsrow][resultscolumn +2] = z;
                resultsarraynumbers[resultsrow][resultscolumn +3] = indicator1position;
                resultsarraynumbers[resultsrow][resultscolumn +4] = indicator2position;
                resultsarraynumbers[resultsrow][resultscolumn +5] = trade1position;
                resultsarraynumbers[resultsrow][resultscolumn +6] = trade2position;
                resultsarraynumbers[resultsrow][resultscolumn +7] = numberofcells;
                resultsarraynumbers[resultsrow][resultscolumn +8] = indicator1value;
                resultsarraynumbers[resultsrow][resultscolumn +9] = indicator2value;
                resultsarraynumbers[resultsrow][resultscolumn +10] = trade1value;
                resultsarraynumbers[resultsrow][resultscolumn +11] = trade2value;
                resultsarraynumbers[resultsrow][resultscolumn +12] = tradevalue;

---

If you know the layout of the data, then you define a single struct and create an array of that struct instead of a two dimenisonal array.

Code:

---

#include <vector>
struct MyData
{
  int y;
  int z;
  int indicator1position;
  //...
};

typedef std::vector<MyData> AllMyData;
//...
AllMyData mData( number_of_data_elements );
mData[resultsrow].indicator1position = indicator1.position;
//...etc.

---

This eliminates the +1, +2, +3, etc. magic numbers.

Regards,

Paul McKenzie

Thats a lot to take in, thankyou for your time.

I am using Code::Blocks editor and on the compiler settings menu, it had automatically selected GNU GCC Compiler. Which compiler should i chose, there are over 30 different types. Is gcc bad? I don't mean bad, but i presume i want to be coding in basic c++? (sorry i am a little confused)

for the magic numbers 2, 15, and 45.

What i am acheiving with the arrays, is comparing corresponding 1's and 0's in each column of dataarray.
or matrix of dataarray example:

011
110
101

Eg. if element[1][1] = 1 and element[2][1] = 1 & if element[0][0] = 1, the set (as above) y = 1. Adding in the magic numbers ignore all of this if the corresponding direction array is equal to 15 or 45

In the main dataarray, it is filled with 0's, 1's and 2's. On each iteration i want to ignore it when the dataarray == 2. The same with 15 and 45 in the direction array. I am only interested in the other ones. I cannot delete them as i need the rest of the 0's and 1's that are in the other corresponding rows.

I know i am not explaining too well its hard to get ya head round it

Rgds

Surreall

Okay after lots of searching, got the -ANSI now. And as you said it doesnt like variable length for array

you have 4 nested 4 loops.
From a quick glance, it doesn't look like the innermost code has any obvious shortcuts or way to speed things up (much).

If you want even more speed, then the trick now is to figure out a better method for achieving the same results rather than finding a few tweaks here and there.
Without knowing what your code actually tries to accomplish, there's not much we can do there to help though.

q and c both iterating over columns with an exception if q and c are identical makes me believe you are probably doing things that can be removed.

Quote:

---

Originally Posted by OReubens

you have 4 nested 4 loops.
From a quick glance, it doesn't look like the innermost code has any obvious shortcuts or way to speed things up (much).

If you want even more speed, then the trick now is to figure out a better method for achieving the same results rather than finding a few tweaks here and there.
Without knowing what your code actually tries to accomplish, there's not much we can do there to help though.

q and c both iterating over columns with an exception if q and c are identical makes me believe you are probably doing things that can be removed.

---

Sounds fair enough. With the q and c if you look at my little 3x3 matrix above. I compare column1(ie q) with the second and third column (ie c). I then compare column 2 (q) with the first and third column (ie c) and lastely compare column3 (q) with the first and second column(c).

The actual array has 39 columns, so lots and lots of iterations

Rgds

Surreall

Here is the updated code:

Code:

---

#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>

using namespace std;

int main()
{
/////This populats the dataarray with data1.csv crap
int dataarray[6077][37];

ifstream data2("Data1.csv");
string line2;
int rows1= 0;
while (getline(data2,line2,'\n'))
{
    stringstream bb(line2);
  string field1;
    int columns1 = 0;
    while (getline(bb,field1,','))
    {
        istringstream buffer(field1);
        int value = 2;
        buffer >> value;
        dataarray[rows1][columns1] = value;
        columns1++;
    }
    rows1++;
}

// do the same for directionarray
int directionarray[6077];
ifstream direction("direction.csv");
string line3;
int rows2 = 0;
while (getline(direction,line3, '\n'))
    {
    istringstream buffer1(line3);
    int value1 = 0;
    buffer1 >> value1;
    directionarray[rows2] = value1;
    rows2++;
    }
//cout << "Directionarray element, " << directionarray[0] << endl;

// do the same for headerarray
//int headerarray[columns];

string headerarray[37];
ifstream header("Header.csv");
string line4;
int columns2 = 0;
while (getline(header,line4, ','))
{
    headerarray[columns2] = line4;
    columns2++;
}


//i have now created arrays, dataarray, directionarray & headerarray with correct elements

int indicator1position = 4;
int indicator2position = 5;

int indicator1value = 0;
int indicator2value = 1;

int trade1position = 4;
int trade2position = 5;

int trade1value = 1;
int trade2value = 1;

int tradevalue = 1;
int numberofcells = 4;

int q = 0;
int c = 0;

float y = 0;
float z = 0;

//float resultsarraynumbers[5000][18];
vector<float> resultsarraynumbers;
vector<string> resultsarraywords;

int jRows = 0;
int rows = 6077;
int columns = 37;
int rows3 = 0;


for (indicator1position = 4;indicator1position < 13; indicator1position++)
{
for (indicator2position = 1 + indicator1position;indicator2position < 9+ indicator1position; indicator2position++)
{
for (trade1position = 4;trade1position < 13; trade1position++)
{
for (trade2position = 1 + trade1position;trade2position < 9 + trade1position; trade2position++)
{
for (q = 0;q < columns - 1; q++)
    {
  for (c = 0;c < columns - 1; c++)
        {
        if (q==c) {} else
            {
            for (jRows = 0;jRows < rows - 50; jRows++)
                {
                if (dataarray[jRows][q]==2 || directionarray[jRows]==15 ||  directionarray[jRows]==45){}
                else if (numberofcells == 4)
                    {  if (dataarray[jRows + indicator1position][c]==indicator1value && \
                        dataarray[jRows + indicator2position][c]==indicator2value && \
                        dataarray[jRows + trade1position][q]==trade1value && \
                        dataarray[jRows + trade2position][q]==trade2value)
                        {
                        if (dataarray[jRows][q]==tradevalue)
                            {y++;} else {z++;}
                        }
                    }
                }
                float t = (y/z);
                if(y==0){y = 1;} else if (z==0){z = 1;}
                if ( t > 4 && y > 10 )
                {
                //cout << t << endl;
                resultsarraynumbers.push_back(t);
                resultsarraynumbers.push_back(y);
                resultsarraynumbers.push_back(z);
                //rows3++;
                resultsarraywords.push_back (headerarray[q]);
                resultsarraywords.push_back (headerarray[c]);
                }
              }
                y = 0;
                z = 0;
            }
        }
}
}
}
}

//for (vector<int>::iterator it = v.begin(); it!=v.end(); ++it) {
//printout
//  cout << *it << ", " << endl;
//}

cout << "Results number, t=" << resultsarraynumbers[0] << endl;
cout << "Results number, y=" << resultsarraynumbers[1] << endl;
cout << "Results number, z=" << resultsarraynumbers[2] << endl;
cout << "Results trade, " << resultsarraywords[0] << endl;
cout << "Results indicator, " << resultsarraywords[1] << endl;
cout << "Results, y= " << y << endl;
cout << "Results, z= " << z << endl;

return 0;
}

---

Which runs pretty darn well, albeit takes around 6 minutes to finish. And i still have one more for loop to add :/

But question is i would now like to output the results into a csv file. But stupidly i have coded it so that resultsarraynumbers is a float and resultsarraywords is a string. (only way i could get it to work)

But i need to list the first 3 elements of the float and then the first 2 elements of the string in first line of a csv file.

Then on the next line in the csv i need to put the next 3 of float followed by the next two of elements etc till the end of the vectors.

Is there a way to combine the vectors in this manner? And then export that to csv file?

Rgds

Surreall

Perhaps the easiest way is to define a struct to contain the 3 numbers and the two strings and then declare a vector of this struct. The example below illustrates a possible method

Code:

---

#include <vector>
#include <string>
#include <fstream>
using namespace std;

//define structure to contain what is required for each line in the csv file
struct results {
    double t, y, z;
    string q, c;
};

int main()
{
vector<results> resultsarray;

results oneres;

//Open output file for the csv
ofstream ofs("results.csv");

//Put some data into the result vector for test
        for (int l = 1; l < 20; l++) {
                oneres.t = l + 3.4;
                oneres.y = l + 3.5;
                oneres.z = l + 3.6;
                oneres.q = "s1";
                oneres.c = "s2";
                resultsarray.push_back(oneres);
        }

//Write result vector to output csv file
        for (vector<results>::iterator it = resultsarray.begin(); it != resultsarray.end(); ++it) {
                ofs << it->t << "," << it->y << "," << it->z << "," << it->q << "," << it->c << endl;
        }
        ofs.close();
       
        return 0;
}

---

I note that you are now using fixed size arrays and that there is no error detection in case your data contains more rows/columns than expected. As c++ doesn't as standard provide bounds checking for arrays, may I suggest that when you populate these arrays you check that the bounds aren't exceeded?

Quote:

---

Originally Posted by Surreall

Which runs pretty darn well, albeit takes around 6 minutes to finish.

---

First, are you running an optimized build, not a debug build? If you're running a debug build, it should not be used for timing purposes. Always time a fully optimized build.

6 minutes to do something means you have either:

1) a huge amount of data (millions of records), or
2) the code you wrote is ineffficient, or
3) you're running a debug build, or
4) some combination of the above 3 items.

Regards,

Paul McKenzie

Unless you have conditional breakpoints set, a debug build shouldn't slow you down a whole lot.

Quote:

---

Originally Posted by Surreall

Sounds fair enough. With the q and c if you look at my little 3x3 matrix above. I compare column1(ie q) with the second and third column (ie c). I then compare column 2 (q) with the first and third column (ie c) and lastely compare column3 (q) with the first and second column(c).

---

Forget about C++ for a moment and think about a generic approach. That's why I asked about the magic numbers -- those numbers should be data that drives the generic approach, not hardcoded.

Too many times, persons write the code so that it becomes inflexible, just to get something working. I never take that approach. Right now, you are using 2, 15, 45, etc. But what if you get the same setup, but the columns are now 3, 13, and 27? Are you going to continually change the code to accommodate those new numbers?

For example, you have a matrix with n columns, and for each iteration up to n, you want to compare columns that are not n.

Code:

---

int currentColumn = someValue;
for ( int i = 0; i < numTotalColumnsInArray; ++i )
{
    if ( i == currentColumn )
      continue;
    if ( CompareMatrix(currentColumn, i) == whatever )
    {
        // do whatever
    }
}

---

I whittled down your explanation to those few lines.

Regards,

Paul McKenzie

Quote:

---

Originally Posted by GCDEF

Unless you have conditional breakpoints set, a debug build shouldn't slow you down a whole lot.

---

I don't know about gcc, but any usage of STL in Visual C++ is a huge bottleneck in debug builds, due to the iterator checking.

Regards,

Paul McKenzie

I have very very few debugging options, most are greyed out. Those that are on in settings are:

- Autobuild project if it is not up to date
- When stopping, autoswitch to first frame with valid source info
- enable watch scripts
- catch c++ expressions

None others are ticked

Code:

---

int currentColumn = someValue;
for ( int i = 0; i < numTotalColumnsInArray; ++i )
{
    if ( i == currentColumn )
      continue;
    if ( CompareMatrix(currentColumn, i) == whatever )
    {
        // do whatever
    }
}

---

Yeah thats pretty much it, should i change the two for loops (q and c) for something similar to this?

As for the 15's and 30's etc, i cannot forsee needing to change them. Same with the 1's or 0's or 2's or headers. At least not for the next 6 months.

Quote:

---

Originally Posted by Surreall

I have very very few debugging options, most are greyed out. Those that are on in settings are:

- Autobuild project if it is not up to date
- When stopping, autoswitch to first frame with valid source info
- enable watch scripts
- catch c++ expressions

None others are ticked

---

The options you want are ones that control optimizations, not debugging options. You want to turn off debugging and turn on optimizations if you want to know exactly how your program performs.

The gcc compiler itself has numerous optimization settings -- maybe there is an entry in the IDE you're using to enter these commands if those options do not show up in the IDE.

Remember that CodeBlocks is not the compiler, gcc is the compiler If you want to know the gcc options, go to the command line and type (I believe)

Quote:

---

gcc --help

---

Or something similar to that. The gcc.exe program has to be in your system path (or you CD to the directory that contains the executable).

Regards,

Paul McKenzie

Quote:

---

Originally Posted by Surreall

Code:

---

int currentColumn = someValue;
for ( int i = 0; i < numTotalColumnsInArray; ++i )
{
    if ( i == currentColumn )
      continue;
    if ( CompareMatrix(currentColumn, i) == whatever )
    {
        // do whatever
    }
}

---

Yeah thats pretty much it, should i change the two for loops (q and c) for something similar to this?

---

If that fits the description, then you can use the same pattern above.

Quote:

---

As for the 15's and 30's etc, i cannot forsee needing to change them.

---

Always think of future changes now, not when it comes time when things do change.

What inevitably happens is that you now build a mountain of code driven by hard-coded values, and then it becomes very difficult, if not near impossible to salvage anything when requirements do change. Or worse, you do the old "I'll change this 2 to a 10, and this 4 to a 7, etc." and now

1) The old program is gone, unless you saved a backup.
2) You've risked breaking the code by changing the source, hoping that you got every constant that needs changing.
3) You have to rebuild each and every time a requirement changes, instead of just changing the data that drives the program.

Regards,

Paul McKenzie

In the global compiler settings, these are the ones that seem to do with optimising. If that is what you mean?

-optimize generated code (for speed) [-O]
-optimize more (for speed) [-O1]
-optimize even more (for speed) [-O2]
-Optimize fully (for speed) [-O3]
-Optimize generate code (for size) [-Os]
-Expensive optimizations [-fexpensive-optimizations]
-strip all symbols from binary (minimizes size) [-s]


There are also a shed load of warning options, which i presume i don't need, and some "have g++ follow language standards"

Good point, for example if i could get the currently 6 mins down to say under 30 seconds i might be tempted to use more data that is available, and those 15's which are minutes, could become singles upto 60. Although that would mean completely changing the code anyway. But i get what you mean, it would be painfully to debug afterwards if there was a change. It is very annoying when code doesnt work, and you have no idea why, just to find in a couple of hours later a little number you forgot about

Quote:

---

Originally Posted by 2kaud

Code to help with exporting

---

Thankyou 2kaud i will give that ago tomorrow, and get back to you with the results

Another thought i had while out running, with this part of the code:

Code:

---

if (dataarray[jRows + indicator1position][c]==indicator1value && \
dataarray[jRows + indicator2position][c]==indicator2value && \
dataarray[jRows + trade1position][q]==trade1value && \
dataarray[jRows + trade2position][q]==trade2value)
  {

---

I am probably wrong, but as this part is iterated almost every time. What about splitting it up. So it only does the first check, and if false it doesnt do the other ones, and if that is true then do the second check and so on to the last check?

In c++ with a multi condition statement, the conditions are only evaluated left to right as needed to correctly obtain the result. So in the case of &&, the left test if evaluted first. If it is false then no more evaluations are done as the result must be false. Only if the test is true is the next condition evaluated. This repeats until either a false is found with && (or true for ||) or the whole expression is evaluated. Splitting it up will make no difference to execution speed. However the order as stated from left to right might. If one condition is more likely to be false than others then this condition should be the first one so that if it is false then the others aren't evaluted.

Excellent thankyou for the info, i will go through and see which is false more often. That should help a bit at least

I also note in your code that you are not pre-reserving space for the vectors. If a vector does not have enough memory to store an extra element then it will re-allocate memory and copy existing elements into the new space when the extra element is added. If a vector has many elements added then this will take time. To avoid this reallocation initial space can be reserved. From my previous example, if it is expected that the result vector has, say, 5000 elements then memory for these can be pre-allocated

Code:

---

vector<results> resultsarray;
resultsarray.reserve(5000);

---

This means that memory re-allocation will now only occur if more than 5000 elements are stored in the vector.

Quote:

---

Originally Posted by 2kaud

I also note in your code that you are not pre-reserving space for the vectors. If a vector does not have enough memory to store an extra element then it will re-allocate memory and copy existing elements into the new space when the extra element is added. If a vector has many elements added then this will take time. To avoid this reallocation initial space can be reserved. From my previous example, if it is expected that the result vector has, say, 5000 elements then memory for these can be pre-allocated

Code:

---

vector<results> resultsarray;
resultsarray.reserve(5000);

---

This means that memory re-allocation will now only occur if more than 5000 elements are stored in the vector.

---

Your previous code for export to csv file worked a treat, ty very much.

I am getting a few bizarre results though:

With this part of the code:

Code:

---

                float t = ( y / z );
                if (y==0){y = 1;}
                if (z==0){z = 1;}
                if ( t > 3 && y > 20 )

---

When i set t > 2 and y > 10, it works fine. And i see i get results where t is also > 3 and y > 20.

So i reset it to t > 3 & y > 20, to reduce the number of results. As i know there are results with these. But the programme closes without finishing. I presume cause results is empty. So it fails.

Any suggestions?

Rgds

Surreall

Quote:

---

Originally Posted by Surreall

I am getting a few bizarre results though:

With this part of the code:

Code:

---

                float t = ( y / z );
                if (y==0){y = 1;}
                if (z==0){z = 1;}
                if ( t > 3 && y > 20 )

---

---

I wonder why you check the value of z to be zero after if has been used in

Code:

---

                float t = ( y / z );

---

Why not before it? :confused:

Quote:

---

Originally Posted by Surreall

... But the programme closes without finishing. I presume cause results is empty. So it fails.

Any suggestions?

---

Debug it and you will see!

Quote:

---

Originally Posted by VictorN

I wonder why you check the value of z to be zero after if has been used in

Code:

---

                float t = ( y / z );

---

Why not before it? :confused:

Debug it and you will see!

---

You are absolutely right, very silly of me :) (and a little embarressed lol)

What is the best way to debug in code::blocks?

Quote:

---

Originally Posted by Surreall

What is the best way to debug in code::blocks?

---

Perhaps, you should read about it in the "code::blocks" documentation?
Or in Google!

Quote:

---

Originally Posted by VictorN

Perhaps, you should read about it in the "code::blocks" documentation?
Or in Google!

---

Yep got it now..i had to click on the project.

You mention that it is taking about 6 minutes to run. Roughly, do you know how much of this time is spent reading the csv file and building the arrays?

Quote:

---

Originally Posted by 2kaud

You mention that it is taking about 6 minutes to run. Roughly, do you know how much of this time is spent reading the csv file and building the arrays?

---

Approx it takes, 0.45 seconds to read and load the intial arrays. Ie dataarray, directionarray & headerarray.

I have ran the debugger, it is something to do with the stack. I get this error:

#0 76F315DE ntdll!LdrQueryProcessModuleInformation() (C:\Windows\system32\ntdll.dll:??)
#1 76F315DE ntdll!LdrQueryProcessModuleInformation() (C:\Windows\system32\ntdll.dll:??)
#2 76F2014E ntdll!LdrFindResource_U() (C:\Windows\system32\ntdll.dll:??)
#3 001AD80C ?? () (??:??)
#4 004010FD __mingw_CRTStartup () (??:??)
#5 ?? ?? () (??:??)


On this part of the code:

Code:

---

for (int l = 0; l < 200000; l++)
    {
                oneres.q = resultsarraystring[m];
                oneres.c = resultsarraystring[m+1];
                oneres.y = resultsarrayfloat[l];
                oneres.z = resultsarrayfloat[l+1];
                oneres.t = resultsarrayfloat[l+2];
                oneres.indicator1position = resultsarrayint[n];
                oneres.indicator2position = resultsarrayint[n+1];
                oneres.trade1position = resultsarrayint[n+2];
                oneres.trade2position = resultsarrayint[n+3];
                oneres.numberofcells = resultsarrayint[n+4];
                oneres.indicator1value = resultsarrayint[n+5];
                oneres.indicator2value = resultsarrayint[n+6];
                oneres.trade1value = resultsarrayint[n+7];
                oneres.trade2value = resultsarrayint[n+8];
                oneres.tradevalue = resultsarrayint[n+9];
                resultsarray.push_back(oneres);
                m = m + 2;
                l = l + 2;
                n = n + 10;
        }

---

I have noticed when i add watches to q & c, in the watch window they come up int, and description in the watch window says "<optimized out>" when they are in the code above.


Current full code;

Code:

---

#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>

using namespace std;

// defines structure to contain waht is required for each line in the csv file
struct results {
    string q, c;
    double y, z, t;
    int indicator1value, indicator2value, trade1value, trade2value;
    int numberofcells;
    int indicator1position, indicator2position, trade1position, trade2position;
    int tradevalue;
};

int main()
{
/////This populats the dataarray with data1.csv crap
int dataarray[6077][37];

ifstream data2("Data1.csv");
string line2;
int rows1= 0;
while (getline(data2,line2,'\n'))
{
    stringstream bb(line2);
  string field1;
    int columns1 = 0;
    while (getline(bb,field1,','))
    {
        istringstream buffer(field1);
        int value = 2;
        buffer >> value;
        dataarray[rows1][columns1] = value;
        columns1++;
    }
    rows1++;
}

// do the same for directionarray
int directionarray[6077];
ifstream direction("direction.csv");
string line3;
int rows2 = 0;
while (getline(direction,line3, '\n'))
    {
    istringstream buffer1(line3);
    int value1 = 0;
    buffer1 >> value1;
    directionarray[rows2] = value1;
    rows2++;
    }
//cout << "Directionarray element, " << directionarray[0] << endl;

// do the same for headerarray
//int headerarray[columns];

string headerarray[37];
ifstream header("Header.csv");
string line4;
int columns2 = 0;
while (getline(header,line4, ','))
{
    headerarray[columns2] = line4;
    columns2++;
}

int binaryarray[32][5];

ifstream binary("Binary.csv");
string line5;
int rows4= 0;
while (getline(binary,line5,'\n'))
{
    stringstream cc(line5);
  string field2;
    int columns4 = 0;
    while (getline(cc,field2,','))
    {
        istringstream buffer2(field2);
        int value2 = 2;
        buffer2 >> value2;
        binaryarray[rows4][columns4] = value2;
        columns4++;
    }
    rows4++;
}

cout << "Binary element 0,0 =" << binaryarray[0][0] << endl;
cout << "Binary element 0,1 =" << binaryarray[0][1] << endl;
cout << "Binary element 0,2 =" << binaryarray[0][2] << endl;
cout << "Binary element 0,3 =" << binaryarray[0][3] << endl;
cout << "Binary element 0,4 =" << binaryarray[0][4] << endl;
cout << "Binary element 1,0 =" << binaryarray[1][0] << endl;
cout << "Binary element 1,1 =" << binaryarray[1][1] << endl;
cout << "Binary element 1,2 =" << binaryarray[1][2] << endl;
cout << "Binary element 1,3 =" << binaryarray[1][3] << endl;
cout << "Binary element 1,4 =" << binaryarray[1][4] << endl;


//i have now created arrays, dataarray, directionarray & headerarray with correct elements

int indicator1value = 1;
int indicator2value = 0;
int trade1value = 0;
int trade2value = 1;



int indicator1position = 4;
int indicator2position = 5;
int trade1position = 4;
int trade2position = 5;
int numberofcells = 4;

int tradevalue = 1;


int q = 0;
int c = 0;
float y = 0;
float z = 0;
float t = 0;
int jRows = 0;
int rows = 6077;
int columns = 37;
//int binary1 = 0;

//float resultsarraynumbers[5000][18];
vector<float> resultsarrayfloat;
resultsarrayfloat.reserve(5000);
vector<int> resultsarrayint;
resultsarrayint.reserve(5000);
vector<string> resultsarraystring;
resultsarraystring.reserve(5000);

//for (binary1 = 0;binary1 < 32; binary1++)
//{
//int indicator1value = binaryarray[binary1][0];
//int indicator2value = binaryarray[binary1][1];
//int trade1value = binaryarray[binary1][2];
//int trade2value = binaryarray[binary1][3];
//int tradevalue = binaryarray[binary1][4];

for (indicator1position = 4;indicator1position < 13; indicator1position++)
{
for (indicator2position = 1 + indicator1position;indicator2position < 9+ indicator1position; indicator2position++)
{
for (trade1position = 4;trade1position < 13; trade1position++)
{
for (trade2position = 1 + trade1position;trade2position < 9 + trade1position; trade2position++)
{
for (q = 0;q < columns - 1; q++)
    {
  for (c = 0;c < columns - 1; c++)
        {
        if (q==c) {} else
            {
            for (jRows = 0;jRows < rows - indicator2position - trade2position; jRows++)
                {
                if (dataarray[jRows][q]==2 || directionarray[jRows]==15 || directionarray[jRows]==45){}
                else if (numberofcells == 4)
                    {
                        if (dataarray[jRows + indicator1position][c]==indicator1value && \
                        dataarray[jRows + indicator2position][c]==indicator2value && \
                        dataarray[jRows + trade1position][q]==trade1value && \
                        dataarray[jRows + trade2position][q]==trade2value)
                        {
                        if (dataarray[jRows][q]==tradevalue)
                            {y++;} else {z++;}
                        }
                    }
                }
                //t = ( y / z );
                if (y==0){y = 1;}
                if (z==0){z = 1;}
                t = y / z;
                if ( t > 3 && y > 9 )
                {
                //cout << t << endl;
                resultsarrayfloat.push_back(t);
                resultsarrayfloat.push_back(y);
                resultsarrayfloat.push_back(z);

                resultsarrayint.push_back(indicator1position);
                resultsarrayint.push_back(indicator2position);
                resultsarrayint.push_back(trade1position);
                resultsarrayint.push_back(trade2position);
                resultsarrayint.push_back(numberofcells);
                resultsarrayint.push_back(indicator1value);
                resultsarrayint.push_back(indicator2value);
                resultsarrayint.push_back(trade1value);
                resultsarrayint.push_back(trade2value);
                resultsarrayint.push_back(tradevalue);

                resultsarraystring.push_back (headerarray[q]);
                resultsarraystring.push_back (headerarray[c]);
                }
              }
                y = 0;
                z = 0;
            }
        }
}
}
}
}
//}

//for (vector<int>::iterator it = v.begin(); it!=v.end(); ++it) {
//printout
//  cout << *it << ", " << endl;
//}

cout << "Results number, t=" << resultsarrayfloat[0] << endl;
cout << "Results number, y=" << resultsarrayfloat[1] << endl;
cout << "Results number, z=" << resultsarrayfloat[2] << endl;
cout << "Results trade, " << resultsarraystring[0] << endl;
cout << "Results indicator, " << resultsarraystring[1] << endl;
cout << "Results, y= " << y << endl;
cout << "Results, z= " << z << endl;


vector<results> resultsarray;

results oneres;

//Open output file for the csv
ofstream ofs("results.csv");

int m = 0;
int n = 0;

//Put some data into the result vector for test
        for (int l = 0; l < 200000; l++)
    {
                oneres.q = resultsarraystring[m];
                oneres.c = resultsarraystring[m+1];
                oneres.y = resultsarrayfloat[l];
                oneres.z = resultsarrayfloat[l+1];
                oneres.t = resultsarrayfloat[l+2];
                oneres.indicator1position = resultsarrayint[n];
                oneres.indicator2position = resultsarrayint[n+1];
                oneres.trade1position = resultsarrayint[n+2];
                oneres.trade2position = resultsarrayint[n+3];
                oneres.numberofcells = resultsarrayint[n+4];
                oneres.indicator1value = resultsarrayint[n+5];
                oneres.indicator2value = resultsarrayint[n+6];
                oneres.trade1value = resultsarrayint[n+7];
                oneres.trade2value = resultsarrayint[n+8];
                oneres.tradevalue = resultsarrayint[n+9];
                resultsarray.push_back(oneres);
                m = m + 2;
                l = l + 2;
                n = n + 10;
        }

//Write result vector to output csv file
        for (vector<results>::iterator it = resultsarray.begin(); it != resultsarray.end(); ++it) {
                ofs << it->q << "," << it->c << "," << it->y << "," << it->z << "," << it->t << "," << "," << \
                      it->indicator1position << "," << it->indicator2position << "," << it->trade1position << "," << it->trade2position << "," << \
                      "," << it->numberofcells << "," << "," << \
                      it->indicator1value << "," << it->indicator2value << "," << it->trade1value << "," << it->trade2value << "," << "," << \
                      it->tradevalue << endl;
        }
        ofs.close();

cout << "DONE!" << endl;

return 0;
}

---

Just as a point of interest. All those for loops iterate 36889042944 times.

Rather than having three different results vectors - one each for float, int and string - just have one vector and put the data direct into this rather than into the three different and later on copying to yet another vector.

Replace these

Code:

---

vector<float> resultsarrayfloat;
resultsarrayfloat.reserve(5000);
vector<int> resultsarrayint;
resultsarrayint.reserve(5000);
vector<string> resultsarraystring;
resultsarraystring.reserve(5000);

---

with

Code:

---

vector<results> resultsarray;
resultsarray.reserve(5000);
results oneres;

ofstream ofs("results.csv");

---

Replace

Code:

---


                resultsarrayfloat.push_back(t);
                resultsarrayfloat.push_back(y);
                resultsarrayfloat.push_back(z);

                resultsarrayint.push_back(indicator1position);
                resultsarrayint.push_back(indicator2position);
                resultsarrayint.push_back(trade1position);
                resultsarrayint.push_back(trade2position);
                resultsarrayint.push_back(numberofcells);
                resultsarrayint.push_back(indicator1value);
                resultsarrayint.push_back(indicator2value);
                resultsarrayint.push_back(trade1value);
                resultsarrayint.push_back(trade2value);
                resultsarrayint.push_back(tradevalue);

                resultsarraystring.push_back (headerarray[q]);
                resultsarraystring.push_back (headerarray[c]);

---

with

Code:

---

    oneres.t = t;
    oneres.y = y;
    oneres.z = z;
    oneres.indicator1position = indicator1position;
    .... etc

    resultsarray.push_back(oneres);

---

Then remove

Code:

---

vector<results> resultsarray;

results oneres;

//Open output file for the csv
ofstream ofs("results.csv");

int m = 0;
int n = 0;

//Put some data into the result vector for test
        for (int l = 0; l < 200000; l++)
    {
                oneres.q = resultsarraystring[m];
                oneres.c = resultsarraystring[m+1];
                oneres.y = resultsarrayfloat[l];
                oneres.z = resultsarrayfloat[l+1];
                oneres.t = resultsarrayfloat[l+2];
                oneres.indicator1position = resultsarrayint[n];
                oneres.indicator2position = resultsarrayint[n+1];
                oneres.trade1position = resultsarrayint[n+2];
                oneres.trade2position = resultsarrayint[n+3];
                oneres.numberofcells = resultsarrayint[n+4];
                oneres.indicator1value = resultsarrayint[n+5];
                oneres.indicator2value = resultsarrayint[n+6];
                oneres.trade1value = resultsarrayint[n+7];
                oneres.trade2value = resultsarrayint[n+8];
                oneres.tradevalue = resultsarrayint[n+9];
                resultsarray.push_back(oneres);
                m = m + 2;
                l = l + 2;
                n = n + 10;
        }

---

Would these two lines be right?

Previously:

Code:

---

                resultsarraystring.push_back (headerarray[q]);
                resultsarraystring.push_back (headerarray[c]);

---

New?

Code:

---

                oneres.headerarray[q] = q;
                oneres.headerarray[c] = c;

---

ignore that, i was being thick

oneres.q = q;

You would have

oneres.q = headerarray[q];
oneres.c = headerarray[c];

Yep, struts definitely confuse me.

Working wonderfully thank you very much for your time and effort

I am having one slight problem, with the headerarray[q] & c
Getting this warning,
C:\Users\Mike\Desktop\C++\Final\main.cpp||In function 'int main()':|
C:\Users\Mike\Desktop\C++\Final\main.cpp|10|warning: inlining failed in call to 'results::~results()': call is unlikely and code size would grow [-Winline]|
C:\Users\Mike\Desktop\C++\Final\main.cpp|238|warning: called from here [-Winline]|
C:\Users\Mike\Desktop\C++\Final\main.cpp|10|warning: inlining failed in call to 'results::~results()': call is unlikely and code size would grow [-Winline]|
C:\Users\Mike\Desktop\C++\Final\main.cpp|238|warning: called from here [-Winline]|
||=== Build finished: 0 errors, 4 warnings (0 minutes, 3 seconds) ===|


And after exporting, the q's and c's are empty

Can you post your code again please after all the changes. I'll crank it through my VC compiler and see what comes out.

I think it was me (user error), maybe i hadnt compiled before one of changes. But getting all info that was requested on the export now: Still getting those warnings though

Code:

---

#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>

using namespace std;

// defines structure to contain waht is required for each line in the csv file
struct results {
    string q, c;
    double y, z, t;
    int indicator1value, indicator2value, trade1value, trade2value;
    int numberofcells;
    int indicator1position, indicator2position, trade1position, trade2position;
    int tradevalue;
};

int main()
{
/////This populats the dataarray with data1.csv crap
int dataarray[6077][37];

ifstream data2("Data1.csv");
string line2;
int rows1= 0;
while (getline(data2,line2,'\n'))
{
    stringstream bb(line2);
  string field1;
    int columns1 = 0;
    while (getline(bb,field1,','))
    {
        istringstream buffer(field1);
        int value = 2;
        buffer >> value;
        dataarray[rows1][columns1] = value;
        columns1++;
    }
    rows1++;
}

// do the same for directionarray
int directionarray[6077];
ifstream direction("direction.csv");
string line3;
int rows2 = 0;
while (getline(direction,line3, '\n'))
    {
    istringstream buffer1(line3);
    int value1 = 0;
    buffer1 >> value1;
    directionarray[rows2] = value1;
    rows2++;
    }
//cout << "Directionarray element, " << directionarray[0] << endl;

// do the same for headerarray
//int headerarray[columns];

string headerarray[37];
ifstream header("Header.csv");
string line4;
int columns2 = 0;
while (getline(header,line4, ','))
{
    headerarray[columns2] = line4;
    columns2++;
}

int binaryarray[32][5];

ifstream binary("Binary.csv");
string line5;
int rows4= 0;
while (getline(binary,line5,'\n'))
{
    stringstream cc(line5);
  string field2;
    int columns4 = 0;
    while (getline(cc,field2,','))
    {
        istringstream buffer2(field2);
        int value2 = 2;
        buffer2 >> value2;
        binaryarray[rows4][columns4] = value2;
        columns4++;
    }
    rows4++;
}

cout << "Binary element 0,0 =" << binaryarray[0][0] << endl;
cout << "Binary element 0,1 =" << binaryarray[0][1] << endl;
cout << "Binary element 0,2 =" << binaryarray[0][2] << endl;
cout << "Binary element 0,3 =" << binaryarray[0][3] << endl;
cout << "Binary element 0,4 =" << binaryarray[0][4] << endl;
cout << "Binary element 1,0 =" << binaryarray[1][0] << endl;
cout << "Binary element 1,1 =" << binaryarray[1][1] << endl;
cout << "Binary element 1,2 =" << binaryarray[1][2] << endl;
cout << "Binary element 1,3 =" << binaryarray[1][3] << endl;
cout << "Binary element 1,4 =" << binaryarray[1][4] << endl;
cout << "Binary element 1,4 =" << headerarray[0] << endl;
cout << "Binary element 1,4 =" << headerarray[1] << endl;


//i have now created arrays, dataarray, directionarray & headerarray with correct elements

int indicator1value = 1;
int indicator2value = 0;
int trade1value = 0;
int trade2value = 1;



int indicator1position = 4;
int indicator2position = 5;
int trade1position = 4;
int trade2position = 5;
int numberofcells = 4;

int tradevalue = 1;


int q = 0;
int c = 0;
float y = 0;
float z = 0;
float t = 0;
int jRows = 0;
int rows = 6077;
int columns = 37;
//int binary1 = 0;

//float resultsarraynumbers[5000][18];
vector<results> resultsarray;
resultsarray.reserve(10000);
results oneres;

ofstream ofs("results.csv");

//for (binary1 = 0;binary1 < 32; binary1++)
//{
//int indicator1value = binaryarray[binary1][0];
//int indicator2value = binaryarray[binary1][1];
//int trade1value = binaryarray[binary1][2];
//int trade2value = binaryarray[binary1][3];
//int tradevalue = binaryarray[binary1][4];

for (indicator1position = 4;indicator1position < 13; indicator1position++)
{
for (indicator2position = 1 + indicator1position;indicator2position < 9+ indicator1position; indicator2position++)
{
for (trade1position = 4;trade1position < 13; trade1position++)
{
for (trade2position = 1 + trade1position;trade2position < 9 + trade1position; trade2position++)
{
for (q = 0;q < columns - 1; q++)
    {
  for (c = 0;c < columns - 1; c++)
        {
        if (q==c) {} else
            {
            for (jRows = 0;jRows < rows - indicator2position - trade2position; jRows++)
                {
                if (dataarray[jRows][q]==2 || directionarray[jRows]==15 || directionarray[jRows]==45){}
                else if (numberofcells == 4)
                    {
                        if (dataarray[jRows + indicator1position][c]==indicator1value && \
                        dataarray[jRows + indicator2position][c]==indicator2value && \
                        dataarray[jRows + trade1position][q]==trade1value && \
                        dataarray[jRows + trade2position][q]==trade2value)
                        {
                        if (dataarray[jRows][q]==tradevalue)
                            {y++;} else {z++;}
                        }
                    }
                }
                //t = ( y / z );
                if (y==0){y = 1;}
                if (z==0){z = 1;}
                t = y / z;
                if ( t > 3 && y > 9 )
                {
                oneres.t = t;
                oneres.y = y;
                oneres.z = z;
                oneres.indicator1position = indicator1position;
                oneres.indicator2position = indicator2position;
                oneres.trade1position = trade1position;
                oneres.trade2position = trade2position;
                oneres.numberofcells = numberofcells;
                oneres.indicator1value = indicator1value;
                oneres.indicator2value = indicator2value;
                oneres.trade1value = trade1value;
                oneres.trade2value = trade2value;
                oneres.tradevalue = tradevalue;
                oneres.q = headerarray[q];
                oneres.c = headerarray[c];
                resultsarray.push_back(oneres);
                }
              }
                y = 0;
                z = 0;
            }
        }
}
}
}
}
//}

//for (vector<int>::iterator it = v.begin(); it!=v.end(); ++it) {
//printout
//  cout << *it << ", " << endl;
//}

//cout << "Results number, t=" << results[0] << endl;
//cout << "Results number, y=" << results[1] << endl;
//cout << "Results number, z=" << results[2] << endl;
//cout << "Results trade, " << results[3] << endl;
//cout << "Results indicator, " << results[4] << endl;
//cout << "Results, y= " << y << endl;
//cout << "Results, z= " << z << endl;


//Write result vector to output csv file
        for (vector<results>::iterator it = resultsarray.begin(); it != resultsarray.end(); ++it) {
                ofs << it->q << "," << it->c << "," << it->y << "," << it->z << "," << it->t << "," << "," << \
                      it->indicator1position << "," << it->indicator2position << "," << it->trade1position << "," << it->trade2position << "," << \
                      "," << it->numberofcells << "," << "," << \
                      it->indicator1value << "," << it->indicator2value << "," << it->trade1value << "," << it->trade2value << "," << "," << \
                      it->tradevalue << endl;
        }
        ofs.close();

cout << "DONE!" << endl;

return 0;
}

---

It compiles without any warnings with my VS. Is the info generated correct and what you expected? How long is the program now taking to run?

Getting all info now ( i stupidly had the csv open when running the programme), and is still running at just under 6 mins. Although now i have unchecked the 1st for loop, it will take around 180mins to completely run

Pleased to be of help. Hope it all goes OK for you now. Welcome to the world of c++!

Quote:

---

Originally Posted by Surreall

I have been writing in excel vba for large number crunching, and the code is now taking quite a while to run. A friend of mine suggested i start writing in C++,

---

Switching to C++ may not have the great impact on efficiency you expect. Proper choise of algorithms and data structures is often more important than language. I suggest you spend some time reviewing your current program from that perspective. It may very well be that after the considerable effort of learning C++ and making the port you'll have to do it anyway. Although C++ is a very efficent language it's not a silver bullet for all efficiency problems.

Quote:

---

Originally Posted by nuzzle

Switching to C++ may not have the great impact on efficiency you expect. Proper choise of algorithms and data structures is often more important than language. I suggest you spend some time reviewing your current program from that perspective. It may very well be that after the considerable effort of learning C++ and making the port you'll have to do it anyway. Although C++ is a very efficent language it's not a silver bullet for all efficiency problems.

---

Fair point, although i am very glad i made the change. As the vba code takes over 2 weeks to run, and C++ takes 3 hours. I would say that is a massive improvement

Quote:

---

Originally Posted by Surreall

Getting all info now ( i stupidly had the csv open when running the programme), and is still running at just under 6 mins. Although now i have unchecked the 1st for loop, it will take around 180mins to completely run

---

If you can provide some sample data, others may be able to test and suggest performance enhancements. Are you sure that you have all optimizations enabled when building?

With these kinds of data intensive computations, cache locality is very important for performance. You need to lay out your data and your nested loops to minimize cache misses. E.g. if you look at

Code:

---

            for (jRows = 0;jRows < rows - indicator2position - trade2position; jRows++)
                {
                if (dataarray[jRows][q]==2 || directionarray[jRows]==15 || directionarray[jRows]==45){}
                else if (numberofcells == 4)
                    {
                        if (dataarray[jRows + indicator1position][c]==indicator1value && \
                        dataarray[jRows + indicator2position][c]==indicator2value && \
                        dataarray[jRows + trade1position][q]==trade1value && \
                        dataarray[jRows + trade2position][q]==trade2value)
// ...

---

You are accessing four memory locations that are relatively distance from each other. Making the 'dataarray' column-major may drastically improve cache performance. You could also consider storing these comparisons in a std::vector<bool>, which is space optimized to use only one bit per element. Finally, you can try rearranging your nested for-loops to get better performance.

Of course, multi-threading is also an option to make use of multiple processor(core)s.