C++ memory issues

I am trying to write a function which attempts to allocate memory 3d vectors and returns false if it is unable to allocate the memory. It is not working. I have pasted the code below.

Code:

---

# include <iostream>
# include <fstream>
# include <string>
# include <cstring>
# include <sstream>
# include <time.h>
# include <iomanip>
# include <vector>

using namespace std;

        template <class T>
bool SafeAlloc(vector<T> &v, int Size)
{
        bool MemoryAllocated=true;
        try
        {
                v.resize(Size, T(0));
        }
        catch(bad_alloc&)
        {
                MemoryAllocated=false;
        }
        return MemoryAllocated;
}

        template <class T>
bool Create3DArray(vector< vector< vector<T> > > &array, int size1, int size2, int size3)
{
        bool MemoryAllocated=true;
        int i;
        vector<T> v;
        vector< vector<T> > matrix;

        array.clear();
        try
        {
                v.resize(size3, T(0));
                matrix.resize(size2, v);
                array.resize(size1, matrix);
        }
        catch (bad_alloc&)
        {
                MemoryAllocated=false;
        }
        return MemoryAllocated;
}

int main(int argc, char * argv[])
{
        int big, small, big1d;
        vector<double> big_array, small_array;
        vector< vector< vector<double> > > big_array3d, small_array3d;

        big1d=100000000;
        big=500;
        small=10;

        if (!SafeAlloc(small_array, small))
        {
                cout <<"Unable to allocate memory for small array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for small array"<<endl;
        }

        if (!SafeAlloc(big_array, big1d))
        {
                cout <<"Unable to allocate memory for big array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big array"<<endl;
        }
       
        if (!Create3DArray(small_array3d, small, small, small))
        {
                cout <<"Unable to allocate memory for small 3d array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for small 3d array"<<endl;
        }
       
        if (!Create3DArray(big_array3d, big1d, big1d, big1d))
        {
                cout <<"Unable to allocate memory for big 3d array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big 3d array"<<endl;
        }
       
        if (!Create3DArray(big_array3d, big, big, big))
        {
                cout <<"Unable to allocate memory for big 3d array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big 3d array"<<endl;
        }
       
        return EXIT_SUCCESS;
}

---

The output I get is:

Allocated memory for small array
Unable to allocate memory for big array
Allocated memory for small 3d array
Unable to allocate memory for big 3d array
Killed

The program crashes when it tries to allocate a large 3d vector, but works when it tries to allocate memory for an extremely large 3d vector or a small 3d vector. My function for doing the same thing with 1d vectors always works. It seems that the bad alloc is only caught when it is the first memory allocation in the try block. I am very confused. Thanks in advance for any help.

Your posted program does not compile.

I took the time to correct it enough to run. I saw nothing that could lead to the behavior you are speaking of, nor have I experienced it.

You can post a minimal but complete example that duplicates this behavior, and we'll see what we can do from there.

Note:
Small arrays will immediatly allocate.
Huge arrays will immediatly fail
Medium Arrays (2 to 6 GB) will not fail, but make massive use of your swap. This will make it look like your program has "hung" or "crashed", but this is not the case. It takes about 1 minute on my computer to allocate an 1000xCube 3D vector.

Is this what you are experiencing?

Quote:

---

Originally Posted by BrainInVat

I am trying to write a function which attempts to allocate memory 3d vectors and returns false if it is unable to allocate the memory. It is not working. I have pasted the code below.

---

You didn't paste one of the most important parts of your program -- what is "Real" in your program?

Regards,

Paul McKenzie

My apologies for posting code that does not work. The code I posted was in two source files, and I thought it would be more convenient to post them as one file, but I copied and pasted the wrong version of SafeAlloc. Also I forgot to post the contents of TypeDef.h

which is

Code:

---

#ifndef _TypeDef_included_
#define _TypeDef_included_

typedef double Real;

#endif

---

Here is a version of the code which compiles.

Code:

---


# include <iostream>
# include <fstream>
# include <string>
# include <cstring>
# include <sstream>
# include <time.h>
# include <iomanip>
# include <vector>

using namespace std;

        template <class T>
bool SafeAlloc(vector<T> &v, int Size)
{
        bool MemoryAllocated=true;
        try
        {
                v.resize(Size, T(0));
        }
        catch(bad_alloc&)
        {
                MemoryAllocated=false;
        }
        return MemoryAllocated;
}

        template <class T>
bool Create3DArray(vector< vector< vector<T> > > &array, int size1, int size2, int size3)
{
        bool MemoryAllocated=true;
        int i;
        vector<T> v;
        vector< vector<T> > matrix;

        array.clear();
        try
        {
                v.resize(size3, T(0));
                matrix.resize(size2, v);
                array.resize(size1, matrix);
        }
        catch (bad_alloc&)
        {
                MemoryAllocated=false;
        }
        return MemoryAllocated;
}

int main(int argc, char * argv[])
{
        int big, small, big1d;
        vector<double> big_array, small_array;
        vector< vector< vector<double> > > big_array3d, small_array3d;

        big1d=100000000;
        big=500;
        small=10;

        if (!SafeAlloc(small_array, small))
        {
                cout <<"Unable to allocate memory for small array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for small array"<<endl;
        }

        if (!SafeAlloc(big_array, big1d))
        {
                cout <<"Unable to allocate memory for big array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big array"<<endl;
        }
       
        if (!Create3DArray(small_array3d, small, small, small))
        {
                cout <<"Unable to allocate memory for small 3d array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for small 3d array"<<endl;
        }
       
        if (!Create3DArray(big_array3d, big1d, big1d, big1d))
        {
                cout <<"Unable to allocate memory for big 3d array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big 3d array"<<endl;
        }
       
        if (!Create3DArray(big_array3d, big, big, big))
        {
                cout <<"Unable to allocate memory for big 3d array"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big 3d array"<<endl;
        }
       
        return EXIT_SUCCESS;
}

---

The program does indeed crash. The output is:

Allocated memory for small array
Unable to allocate memory for big array
Allocated memory for small 3d array
Unable to allocate memory for big 3d array
Killed

The output I expect is:

Allocated memory for small array
Unable to allocate memory for big array
Allocated memory for small 3d array
Unable to allocate memory for big 3d array
Unable to allocate memory for big 3d array

It does take it about a minute to crash after printing Unable to allocate memory for big 3d array. The behavior of this program is dependent on the computer memory. I am running this on a 32-bit machine with 1 GB of RAM. If this program does not reproduce the output I posted here maybe you can try adjusting the sizes of the vectors. Thanks again.

Quote:

---

Originally Posted by BrainInVat

My apologies for posting code that does not work.

---

Please mention the compiler and version of the compiler that you used.

Regards,

Paul McKenzie

I use g++ version 4.1.

Again, does your program create an actual illegal operation and crash? Or does it just quit responding, and the OS gives you the option to kill it?

As long as your program does not actually provoke illegal operations, than it is still running, even if it is not responding.

There is nothing wrong with your program (as far as can see), but there are some sizes of memory allocation which are under the threshold, but reside mostly in swap. On my machine, this allocation takes about 2 minutes. Are you sure this is not what you are experiencing?

I have told you the program crashes. I am not the one killing the program. I am running this on a 32-bit linux machine with 1 GB of RAM. I am running this on a computer cluster and when I submit it to the queue or run it on the head node the final print statement is not printed. If it works on your computer you probably have more than 1 GB of RAM, try increasing the value of big to 2000 or 20000 and see what you get.

Quote:

---

Originally Posted by BrainInVat

I have told you the program crashes. I am not the one killing the program. I am running this on a 32-bit linux machine with 1 GB of RAM. I am running this on a computer cluster and when I submit it to the queue or run it on the head node the final print statement is not printed. If it works on your computer you probably have more than 1 GB of RAM, try increasing the value of big to 2000 or 20000 and see what you get.

---

I see none of these problems running ViC++ 2008.

Do a simpler test. You don't need vectors to see what the problem could be.

Calculate the approximate amount of memory where you are crashing, and write a simple program that just calls new[] using that amount of memory. What results do you get? If that gives you the same problem, then call malloc() with equivalent amount of memory. If malloc() does work, then maybe the free-store is broken in that compiler and not the memory allocation proper.

If neither works, then you have a 'C' library issue with memory allocation, not a C++ problem.

Regards,

Paul McKenzie

This is strange. If I run under my default user account settings, I get the result
that you expected.

Quote:

---

> limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse 3443180 kbytes
vmemoryuse 4917760 kbytes
descriptors 1024
memorylocked 64 kbytes
maxproc 31586

> ./a.out
Allocated memory for small array
Allocated memory for big array
Allocated memory for small 3d array
Unable to allocate memory for big 3d array
Unable to allocate memory for big 3d array

---

If I increase my limits, I also get the "killed" :

Quote:

---

> limit vmemoryuse unlimited
> limit memoryuse unlimited
> limit stacksize unlimited
> ./a.out
Allocated memory for small array
Allocated memory for big array
Allocated memory for small 3d array
Unable to allocate memory for big 3d array
Killed

---

SUSE Linux , g++ version 4.4.1 (64 bit)

Any chance you are actually throwing an out_of_range exception (but not bad_alloc)? Vectors throw out of range if you ask for a size that is bigger than max_size().

Try catching the more general run_time_exception, and print out e.what().

Thanks for your responses. I have done some tests.

I believe Paul suggested I run the following code.

Code:

---


# include <iostream>
# include <fstream>
# include <string>
# include <cstring>
# include <sstream>
# include <time.h>
# include <iomanip>
# include <vector>

using namespace std;

int main(int argc, char * argv[])
{
        int big, small, big1d;
        double *array;

        big=500;
        big1d=big*big*big;
        small=10;

        array=new double[big1d];
       
        cout <<"DONE"<<endl;
        return EXIT_SUCCESS;
}

---

The output I get is:

terminate called after throwing an instance of 'std::bad_alloc'
what(): stdbad_alloc
Aborted

I also tried the following code

Code:

---

# include <iostream>
# include <fstream>
# include <string>
# include <cstring>
# include <sstream>
# include <time.h>
# include <iomanip>
# include <vector>

using namespace std;

int main(int argc, char * argv[])
{
        int big, small, big1d;
        double *array;

        big=500;
        big1d=big*big*big;
        small=10;

        array=(double *)malloc(big1d*sizeof(double));
       
        cout <<"DONE"<<endl;
        return EXIT_SUCCESS;
}

---

The output I get is:

DONE

I don't really know what the differences between new and malloc are, and how this result is significant.

I also tried monarch_dodra's suggestion. I changed bad_alloc& to const exception& e, printed e.what(), and made a couple of other minor changes. The code is below:

Code:

---


# include <iostream>
# include <fstream>
# include <string>
# include <cstring>
# include <sstream>
# include <time.h>
# include <iomanip>
# include <vector>

using namespace std;

        template <class T>
bool SafeAlloc(vector<T> &v, int Size)
{
        bool MemoryAllocated=true;
        try
        {
                v.resize(Size, T(0));
        }
        catch(const exception& e)
        {
                cout <<e.what()<<endl;
                MemoryAllocated=false;
        }
        return MemoryAllocated;
}

        template <class T>
bool Create3DArray(vector< vector< vector<T> > > &array, int size1, int size2, int size3)
{
        bool MemoryAllocated=true;
        int i;
        vector<T> v;
        vector< vector<T> > matrix;

        array.clear();
        try
        {
                v.resize(size3, T(0));
                matrix.resize(size2, v);
                array.resize(size1, matrix);
        }
        catch (const exception& e)
        {
                cout <<e.what()<<endl;
                MemoryAllocated=false;
        }
        return MemoryAllocated;
}

int main(int argc, char * argv[])
{
        int big, small, big1d;
        vector<double> big_vector, small_vector;
        vector< vector< vector<double> > > big_vector3d, small_vector3d;
        double ***big_array3d;

        if (argc<2)
        {
                cout <<"Usage: "<<argv[0]<<" size"<<endl;
                exit(EXIT_FAILURE);
        }

        big=atoi(argv[1]);
        big1d=big*big*big;
        small=10;

        if (!SafeAlloc(small_vector, small))
        {
                cout <<"Unable to allocate memory for small vector"<<endl;
        }
        else
        {
                cout <<"Allocated memory for small vector"<<endl;
        }
        small_vector.clear();
        if (!SafeAlloc(big_vector, big1d))
        {
                cout <<"Unable to allocate memory for big vector"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big vector"<<endl;
        }
        big_vector.clear();
        if (!Create3DArray(small_vector3d, small, small, small))
        {
                cout <<"Unable to allocate memory for small 3d vector"<<endl;
        }
        else
        {
                cout <<"Allocated memory for small 3d vector"<<endl;
        }
        small_vector3d.clear();
        if (!Create3DArray(big_vector3d, big1d, big1d, big1d))
        {
                cout <<"Unable to allocate memory for very big 3d vector"<<endl;
        }
        else
        {
                cout <<"Allocated memory for very big 3d vector"<<endl;
        }
        big_vector3d.clear();
        if (!Create3DArray(big_vector3d, big, big, big))
        {
                cout <<"Unable to allocate memory for big 3d vector"<<endl;
        }
        else
        {
                cout <<"Allocated memory for big 3d vector"<<endl;
        }
        big_vector3d.clear();
        cout <<"DONE"<<endl;
        return EXIT_SUCCESS;
}

---

I tested this on a couple of computer clusters. One cluster is 32-bit and has 1 GB of RAM the other is 64-bit and has 4 GB of RAM. For the 32-bit cluster I submitted the program to the queue and ran it on the head node. I only submitted it to the queue on the 64-bit cluster. On both clusters I tested the code for various values of big. The results are summarized bellow:

./MemTest 100 on 32-bit head node:
Allocated memory for small vector
Allocated memory for big vector
Allocated memory for small 3d vector
Killed

./MemTest 100 on 32-bit qsub:
Allocated memory for small vector
Allocated memory for big vector
Allocated memory for small 3d vector
St9bad_alloc
Unable to allocate memory for very big 3d vector
Allocated memory for big 3d vector
DONE

./MemTest 100 on 64-bit:
Allocated memory for small vector
Allocated memory for big vector
Allocated memory for small 3d vector

./MemTest 500 on 32-bit head node:
Allocated memory for small vector
St9bad_alloc
Unable to allocate memory for big vector
Allocated memory for small 3d vector
St9bad_alloc
Unable to allocate memory for very big 3d vector
Killed

./MemTest 500 on 32-bit qsub:
Allocated memory for small vector
Allocated memory for big vector
Allocated memory for small 3d vector
St9bad_alloc
Unable to allocate memory for very big 3d vector
Allocated memory for big 3d vector
DONE


./MemTest 500 on 64-bit:
Allocated memory for small vector
Allocated memory for big vector
Allocated memory for small 3d vector
St9bad_alloc
Unable to allocate memory for very big 3d vector
Allocated memory for big 3d vector
DONE


./MemTest 1000 on 32-bit head node:
Allocated memory for small vector
vector::_M_fill_insert
Unable to allocate memory for big vector
Allocated memory for small 3d vector
vector::_M_fill_insert
Unable to allocate memory for very big 3d vector
Killed

./MemTest 1000 on 32-bit qsub:
Allocated memory for small vector
vector::_M_fill_insert
Unable to allocate memory for big vector
Allocated memory for small 3d vector
vector::_M_fill_insert
Unable to allocate memory for very big 3d vector
St9bad_alloc
Unable to allocate memory for big 3d vector
DONE

./MemTest 1000 on 64-bit:
Allocated memory for small vector
Allocated memory for big vector
Allocated memory for small 3d vector
St9bad_alloc
Unable to allocate memory for very big 3d vector
Allocated memory for big 3d vector
DONE

Changing bad_alloc to exception did help, but did not entirely solve the problem. I am still confused as to why the program works in some instances, but not in others. Thanks again.

One of the differences between new and malloc (amongst others) is that new throws bad_alloc, whereas malloc just returns 0. If you don't check the return of malloc for 0, then the test isn't meaningful.

You code is correct, as you use vectors correctly, and the only two types of exceptions they can throw, you catch, so your program should not be the problem.

You say you "ran it on the head node", what exactly does that mean? Is there any chance there is some kind of supervision program that kills your process for excessive memory usage? It sounds stupid, but I can't really find any other explanation for it.

EDIT: Can you debug the program? This should tell you at least why it is crashing: Illegal read, segfault, or maybe sigterm.

Quote:

---

Originally Posted by BrainInVat

Thanks for your responses. I have done some tests.

I believe Paul suggested I run the following code.

---

Throw in the sizeof(vector<double>, sizeof(vector<vector<double> >), etc.) in your calculation for the memory amount. You forgot to do that.

Quote:

---

I don't really know what the differences between new and malloc are, and how this result is significant.

---

There are major differences between the two.

1)new constructs objects, malloc doesn't.

2) new allocates memory from the free store, while malloc() gets its memory from the heap. The free-store and the heap need not be the same.

3) new can be overloaded, malloc can't.

Regards,

Paul McKenzie

On Windows I've had crashes when I try to allocate a huge amount of memory even if nothing was wrong with my code.