CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 15 of 15
  1. #1
    Join Date
    Jan 2007
    Posts
    9

    C++ memory issues

    I am trying to write a function which attempts to allocate memory 3d vectors and returns false if it is unable to allocate the memory. It is not working. I have pasted the code below.



    Code:
    # include <iostream>
    # include <fstream>
    # include <string>
    # include <cstring>
    # include <sstream>
    # include <time.h>
    # include <iomanip>
    # include <vector>
    
    using namespace std;
    
            template <class T>
    bool SafeAlloc(vector<T> &v, int Size)
    {
            bool MemoryAllocated=true;
            try
            {
                    v.resize(Size, T(0));
            }
            catch(bad_alloc&)
            {
                    MemoryAllocated=false;
            }
            return MemoryAllocated;
    }
    
            template <class T>
    bool Create3DArray(vector< vector< vector<T> > > &array, int size1, int size2, int size3)
    {
            bool MemoryAllocated=true;
            int i;
            vector<T> v;
            vector< vector<T> > matrix;
    
            array.clear();
            try
            {
                    v.resize(size3, T(0));
                    matrix.resize(size2, v);
                    array.resize(size1, matrix);
            }
            catch (bad_alloc&)
            {
                    MemoryAllocated=false;
            }
            return MemoryAllocated;
    }
    
    int main(int argc, char * argv[])
    {
    	int big, small, big1d;
            vector<double> big_array, small_array;
    	vector< vector< vector<double> > > big_array3d, small_array3d;
    
            big1d=100000000;
    	big=500;
    	small=10;
    
    	if (!SafeAlloc(small_array, small))
    	{
    		cout <<"Unable to allocate memory for small array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for small array"<<endl;
    	}
    
    	if (!SafeAlloc(big_array, big1d))
    	{
    		cout <<"Unable to allocate memory for big array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big array"<<endl;
    	}
    	
            if (!Create3DArray(small_array3d, small, small, small))
    	{
    		cout <<"Unable to allocate memory for small 3d array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for small 3d array"<<endl;
    	}
    	
            if (!Create3DArray(big_array3d, big1d, big1d, big1d))
    	{
    		cout <<"Unable to allocate memory for big 3d array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big 3d array"<<endl;
    	}
    	
            if (!Create3DArray(big_array3d, big, big, big))
    	{
    		cout <<"Unable to allocate memory for big 3d array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big 3d array"<<endl;
    	}
    	
            return EXIT_SUCCESS;
    }
    The output I get is:

    Allocated memory for small array
    Unable to allocate memory for big array
    Allocated memory for small 3d array
    Unable to allocate memory for big 3d array
    Killed

    The program crashes when it tries to allocate a large 3d vector, but works when it tries to allocate memory for an extremely large 3d vector or a small 3d vector. My function for doing the same thing with 1d vectors always works. It seems that the bad alloc is only caught when it is the first memory allocation in the try block. I am very confused. Thanks in advance for any help.
    Last edited by BrainInVat; August 31st, 2010 at 01:04 PM.

  2. #2
    Join Date
    Jun 2009
    Location
    France
    Posts
    2,513

    Re: C++ memory issues

    Your posted program does not compile.

    I took the time to correct it enough to run. I saw nothing that could lead to the behavior you are speaking of, nor have I experienced it.

    You can post a minimal but complete example that duplicates this behavior, and we'll see what we can do from there.

    Note:
    Small arrays will immediatly allocate.
    Huge arrays will immediatly fail
    Medium Arrays (2 to 6 GB) will not fail, but make massive use of your swap. This will make it look like your program has "hung" or "crashed", but this is not the case. It takes about 1 minute on my computer to allocate an 1000xCube 3D vector.

    Is this what you are experiencing?
    Is your question related to IO?
    Read this C++ FAQ article at parashift by Marshall Cline. In particular points 1-6.
    It will explain how to correctly deal with IO, how to validate input, and why you shouldn't count on "while(!in.eof())". And it always makes for excellent reading.

  3. #3
    Join Date
    Apr 1999
    Posts
    27,449

    Re: C++ memory issues

    Quote Originally Posted by BrainInVat View Post
    I am trying to write a function which attempts to allocate memory 3d vectors and returns false if it is unable to allocate the memory. It is not working. I have pasted the code below.
    You didn't paste one of the most important parts of your program -- what is "Real" in your program?

    Regards,

    Paul McKenzie

  4. #4
    Join Date
    Jan 2007
    Posts
    9

    Re: C++ memory issues

    My apologies for posting code that does not work. The code I posted was in two source files, and I thought it would be more convenient to post them as one file, but I copied and pasted the wrong version of SafeAlloc. Also I forgot to post the contents of TypeDef.h

    which is

    Code:
    #ifndef _TypeDef_included_
    #define _TypeDef_included_ 
    
    typedef double Real;
    
    #endif
    Here is a version of the code which compiles.

    Code:
    # include <iostream>
    # include <fstream>
    # include <string>
    # include <cstring>
    # include <sstream>
    # include <time.h>
    # include <iomanip>
    # include <vector>
    
    using namespace std;
    
            template <class T>
    bool SafeAlloc(vector<T> &v, int Size)
    {
            bool MemoryAllocated=true;
            try
            {
                    v.resize(Size, T(0));
            }
            catch(bad_alloc&)
            {
                    MemoryAllocated=false;
            }
            return MemoryAllocated;
    }
    
            template <class T>
    bool Create3DArray(vector< vector< vector<T> > > &array, int size1, int size2, int size3)
    {
            bool MemoryAllocated=true;
            int i;
            vector<T> v;
            vector< vector<T> > matrix;
    
            array.clear();
            try
            {
                    v.resize(size3, T(0));
                    matrix.resize(size2, v);
                    array.resize(size1, matrix);
            }
            catch (bad_alloc&)
            {
                    MemoryAllocated=false;
            }
            return MemoryAllocated;
    }
    
    int main(int argc, char * argv[])
    {
    	int big, small, big1d;
            vector<double> big_array, small_array;
    	vector< vector< vector<double> > > big_array3d, small_array3d;
    
            big1d=100000000;
    	big=500;
    	small=10;
    
    	if (!SafeAlloc(small_array, small))
    	{
    		cout <<"Unable to allocate memory for small array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for small array"<<endl;
    	}
    
    	if (!SafeAlloc(big_array, big1d))
    	{
    		cout <<"Unable to allocate memory for big array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big array"<<endl;
    	}
    	
            if (!Create3DArray(small_array3d, small, small, small))
    	{
    		cout <<"Unable to allocate memory for small 3d array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for small 3d array"<<endl;
    	}
    	
            if (!Create3DArray(big_array3d, big1d, big1d, big1d))
    	{
    		cout <<"Unable to allocate memory for big 3d array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big 3d array"<<endl;
    	}
    	
            if (!Create3DArray(big_array3d, big, big, big))
    	{
    		cout <<"Unable to allocate memory for big 3d array"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big 3d array"<<endl;
    	}
    	
            return EXIT_SUCCESS;
    }
    The program does indeed crash. The output is:

    Allocated memory for small array
    Unable to allocate memory for big array
    Allocated memory for small 3d array
    Unable to allocate memory for big 3d array
    Killed

    The output I expect is:

    Allocated memory for small array
    Unable to allocate memory for big array
    Allocated memory for small 3d array
    Unable to allocate memory for big 3d array
    Unable to allocate memory for big 3d array

    It does take it about a minute to crash after printing Unable to allocate memory for big 3d array. The behavior of this program is dependent on the computer memory. I am running this on a 32-bit machine with 1 GB of RAM. If this program does not reproduce the output I posted here maybe you can try adjusting the sizes of the vectors. Thanks again.

  5. #5
    Join Date
    Apr 1999
    Posts
    27,449

    Re: C++ memory issues

    Quote Originally Posted by BrainInVat View Post
    My apologies for posting code that does not work.
    Please mention the compiler and version of the compiler that you used.

    Regards,

    Paul McKenzie
    Last edited by Paul McKenzie; August 31st, 2010 at 04:06 PM.

  6. #6
    Join Date
    Jan 2007
    Posts
    9

    Re: C++ memory issues

    I use g++ version 4.1.

  7. #7
    Join Date
    Jun 2009
    Location
    France
    Posts
    2,513

    Re: C++ memory issues

    Again, does your program create an actual illegal operation and crash? Or does it just quit responding, and the OS gives you the option to kill it?

    As long as your program does not actually provoke illegal operations, than it is still running, even if it is not responding.

    There is nothing wrong with your program (as far as can see), but there are some sizes of memory allocation which are under the threshold, but reside mostly in swap. On my machine, this allocation takes about 2 minutes. Are you sure this is not what you are experiencing?
    Is your question related to IO?
    Read this C++ FAQ article at parashift by Marshall Cline. In particular points 1-6.
    It will explain how to correctly deal with IO, how to validate input, and why you shouldn't count on "while(!in.eof())". And it always makes for excellent reading.

  8. #8
    Join Date
    Jan 2007
    Posts
    9

    Re: C++ memory issues

    I have told you the program crashes. I am not the one killing the program. I am running this on a 32-bit linux machine with 1 GB of RAM. I am running this on a computer cluster and when I submit it to the queue or run it on the head node the final print statement is not printed. If it works on your computer you probably have more than 1 GB of RAM, try increasing the value of big to 2000 or 20000 and see what you get.

  9. #9
    Join Date
    Apr 1999
    Posts
    27,449

    Re: C++ memory issues

    Quote Originally Posted by BrainInVat View Post
    I have told you the program crashes. I am not the one killing the program. I am running this on a 32-bit linux machine with 1 GB of RAM. I am running this on a computer cluster and when I submit it to the queue or run it on the head node the final print statement is not printed. If it works on your computer you probably have more than 1 GB of RAM, try increasing the value of big to 2000 or 20000 and see what you get.
    I see none of these problems running ViC++ 2008.

    Do a simpler test. You don't need vectors to see what the problem could be.

    Calculate the approximate amount of memory where you are crashing, and write a simple program that just calls new[] using that amount of memory. What results do you get? If that gives you the same problem, then call malloc() with equivalent amount of memory. If malloc() does work, then maybe the free-store is broken in that compiler and not the memory allocation proper.

    If neither works, then you have a 'C' library issue with memory allocation, not a C++ problem.

    Regards,

    Paul McKenzie
    Last edited by Paul McKenzie; September 1st, 2010 at 11:23 AM.

  10. #10
    Join Date
    Aug 2000
    Location
    West Virginia
    Posts
    7,721

    Re: C++ memory issues

    This is strange. If I run under my default user account settings, I get the result
    that you expected.

    > limit
    cputime unlimited
    filesize unlimited
    datasize unlimited
    stacksize 8192 kbytes
    coredumpsize 0 kbytes
    memoryuse 3443180 kbytes
    vmemoryuse 4917760 kbytes
    descriptors 1024
    memorylocked 64 kbytes
    maxproc 31586

    > ./a.out
    Allocated memory for small array
    Allocated memory for big array
    Allocated memory for small 3d array
    Unable to allocate memory for big 3d array
    Unable to allocate memory for big 3d array
    If I increase my limits, I also get the "killed" :

    > limit vmemoryuse unlimited
    > limit memoryuse unlimited
    > limit stacksize unlimited
    > ./a.out
    Allocated memory for small array
    Allocated memory for big array
    Allocated memory for small 3d array
    Unable to allocate memory for big 3d array
    Killed
    SUSE Linux , g++ version 4.4.1 (64 bit)

  11. #11
    Join Date
    Jun 2009
    Location
    France
    Posts
    2,513

    Re: C++ memory issues

    Any chance you are actually throwing an out_of_range exception (but not bad_alloc)? Vectors throw out of range if you ask for a size that is bigger than max_size().

    Try catching the more general run_time_exception, and print out e.what().
    Is your question related to IO?
    Read this C++ FAQ article at parashift by Marshall Cline. In particular points 1-6.
    It will explain how to correctly deal with IO, how to validate input, and why you shouldn't count on "while(!in.eof())". And it always makes for excellent reading.

  12. #12
    Join Date
    Jan 2007
    Posts
    9

    Re: C++ memory issues

    Thanks for your responses. I have done some tests.

    I believe Paul suggested I run the following code.

    Code:
    # include <iostream>
    # include <fstream>
    # include <string>
    # include <cstring>
    # include <sstream>
    # include <time.h>
    # include <iomanip>
    # include <vector>
    
    using namespace std;
    
    int main(int argc, char * argv[])
    {
    	int big, small, big1d;
            double *array;
    
    	big=500;
            big1d=big*big*big;
    	small=10;
    
            array=new double[big1d];
            
            cout <<"DONE"<<endl;
            return EXIT_SUCCESS;
    }
    The output I get is:

    terminate called after throwing an instance of 'std::bad_alloc'
    what(): stdbad_alloc
    Aborted

    I also tried the following code

    Code:
    # include <iostream>
    # include <fstream>
    # include <string>
    # include <cstring>
    # include <sstream>
    # include <time.h>
    # include <iomanip>
    # include <vector>
    
    using namespace std;
    
    int main(int argc, char * argv[])
    {
    	int big, small, big1d;
            double *array;
    
    	big=500;
            big1d=big*big*big;
    	small=10;
    
            array=(double *)malloc(big1d*sizeof(double));
            
            cout <<"DONE"<<endl;
            return EXIT_SUCCESS;
    }
    The output I get is:

    DONE

    I don't really know what the differences between new and malloc are, and how this result is significant.

    I also tried monarch_dodra's suggestion. I changed bad_alloc& to const exception& e, printed e.what(), and made a couple of other minor changes. The code is below:

    Code:
    # include <iostream>
    # include <fstream>
    # include <string>
    # include <cstring>
    # include <sstream>
    # include <time.h>
    # include <iomanip>
    # include <vector>
    
    using namespace std;
    
            template <class T>
    bool SafeAlloc(vector<T> &v, int Size)
    {
            bool MemoryAllocated=true;
            try
            {
                    v.resize(Size, T(0));
            }
            catch(const exception& e)
            {
    		cout <<e.what()<<endl;
                    MemoryAllocated=false;
            }
            return MemoryAllocated;
    }
    
            template <class T>
    bool Create3DArray(vector< vector< vector<T> > > &array, int size1, int size2, int size3)
    {
            bool MemoryAllocated=true;
            int i;
            vector<T> v;
            vector< vector<T> > matrix;
    
            array.clear();
            try
            {
                    v.resize(size3, T(0));
                    matrix.resize(size2, v);
                    array.resize(size1, matrix);
            }
            catch (const exception& e)
            {
    		cout <<e.what()<<endl;
                    MemoryAllocated=false;
            }
            return MemoryAllocated;
    }
    
    int main(int argc, char * argv[])
    {
    	int big, small, big1d;
            vector<double> big_vector, small_vector;
    	vector< vector< vector<double> > > big_vector3d, small_vector3d;
            double ***big_array3d;
    
    	if (argc<2)
    	{
    		cout <<"Usage: "<<argv[0]<<" size"<<endl;
    		exit(EXIT_FAILURE);
    	}
    
    	big=atoi(argv[1]);
            big1d=big*big*big;
    	small=10;
    
    	if (!SafeAlloc(small_vector, small))
    	{
    		cout <<"Unable to allocate memory for small vector"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for small vector"<<endl;
    	}
            small_vector.clear();
    	if (!SafeAlloc(big_vector, big1d))
    	{
    		cout <<"Unable to allocate memory for big vector"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big vector"<<endl;
    	}
    	big_vector.clear();
            if (!Create3DArray(small_vector3d, small, small, small))
    	{
    		cout <<"Unable to allocate memory for small 3d vector"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for small 3d vector"<<endl;
    	}
    	small_vector3d.clear();
            if (!Create3DArray(big_vector3d, big1d, big1d, big1d))
    	{
    		cout <<"Unable to allocate memory for very big 3d vector"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for very big 3d vector"<<endl;
    	}
    	big_vector3d.clear();
            if (!Create3DArray(big_vector3d, big, big, big))
    	{
    		cout <<"Unable to allocate memory for big 3d vector"<<endl;
    	}
    	else 
    	{
    		cout <<"Allocated memory for big 3d vector"<<endl;
    	}
    	big_vector3d.clear();
    	cout <<"DONE"<<endl;
            return EXIT_SUCCESS;
    }
    I tested this on a couple of computer clusters. One cluster is 32-bit and has 1 GB of RAM the other is 64-bit and has 4 GB of RAM. For the 32-bit cluster I submitted the program to the queue and ran it on the head node. I only submitted it to the queue on the 64-bit cluster. On both clusters I tested the code for various values of big. The results are summarized bellow:

    ./MemTest 100 on 32-bit head node:
    Allocated memory for small vector
    Allocated memory for big vector
    Allocated memory for small 3d vector
    Killed

    ./MemTest 100 on 32-bit qsub:
    Allocated memory for small vector
    Allocated memory for big vector
    Allocated memory for small 3d vector
    St9bad_alloc
    Unable to allocate memory for very big 3d vector
    Allocated memory for big 3d vector
    DONE

    ./MemTest 100 on 64-bit:
    Allocated memory for small vector
    Allocated memory for big vector
    Allocated memory for small 3d vector

    ./MemTest 500 on 32-bit head node:
    Allocated memory for small vector
    St9bad_alloc
    Unable to allocate memory for big vector
    Allocated memory for small 3d vector
    St9bad_alloc
    Unable to allocate memory for very big 3d vector
    Killed

    ./MemTest 500 on 32-bit qsub:
    Allocated memory for small vector
    Allocated memory for big vector
    Allocated memory for small 3d vector
    St9bad_alloc
    Unable to allocate memory for very big 3d vector
    Allocated memory for big 3d vector
    DONE


    ./MemTest 500 on 64-bit:
    Allocated memory for small vector
    Allocated memory for big vector
    Allocated memory for small 3d vector
    St9bad_alloc
    Unable to allocate memory for very big 3d vector
    Allocated memory for big 3d vector
    DONE


    ./MemTest 1000 on 32-bit head node:
    Allocated memory for small vector
    vector::_M_fill_insert
    Unable to allocate memory for big vector
    Allocated memory for small 3d vector
    vector::_M_fill_insert
    Unable to allocate memory for very big 3d vector
    Killed

    ./MemTest 1000 on 32-bit qsub:
    Allocated memory for small vector
    vector::_M_fill_insert
    Unable to allocate memory for big vector
    Allocated memory for small 3d vector
    vector::_M_fill_insert
    Unable to allocate memory for very big 3d vector
    St9bad_alloc
    Unable to allocate memory for big 3d vector
    DONE

    ./MemTest 1000 on 64-bit:
    Allocated memory for small vector
    Allocated memory for big vector
    Allocated memory for small 3d vector
    St9bad_alloc
    Unable to allocate memory for very big 3d vector
    Allocated memory for big 3d vector
    DONE

    Changing bad_alloc to exception did help, but did not entirely solve the problem. I am still confused as to why the program works in some instances, but not in others. Thanks again.

  13. #13
    Join Date
    Jun 2009
    Location
    France
    Posts
    2,513

    Re: C++ memory issues

    One of the differences between new and malloc (amongst others) is that new throws bad_alloc, whereas malloc just returns 0. If you don't check the return of malloc for 0, then the test isn't meaningful.

    You code is correct, as you use vectors correctly, and the only two types of exceptions they can throw, you catch, so your program should not be the problem.

    You say you "ran it on the head node", what exactly does that mean? Is there any chance there is some kind of supervision program that kills your process for excessive memory usage? It sounds stupid, but I can't really find any other explanation for it.

    EDIT: Can you debug the program? This should tell you at least why it is crashing: Illegal read, segfault, or maybe sigterm.
    Last edited by monarch_dodra; September 2nd, 2010 at 01:11 AM.
    Is your question related to IO?
    Read this C++ FAQ article at parashift by Marshall Cline. In particular points 1-6.
    It will explain how to correctly deal with IO, how to validate input, and why you shouldn't count on "while(!in.eof())". And it always makes for excellent reading.

  14. #14
    Join Date
    Apr 1999
    Posts
    27,449

    Re: C++ memory issues

    Quote Originally Posted by BrainInVat View Post
    Thanks for your responses. I have done some tests.

    I believe Paul suggested I run the following code.
    Throw in the sizeof(vector<double>, sizeof(vector<vector<double> >), etc.) in your calculation for the memory amount. You forgot to do that.
    I don't really know what the differences between new and malloc are, and how this result is significant.
    There are major differences between the two.

    1)new constructs objects, malloc doesn't.

    2) new allocates memory from the free store, while malloc() gets its memory from the heap. The free-store and the heap need not be the same.

    3) new can be overloaded, malloc can't.

    Regards,

    Paul McKenzie

  15. #15
    Join Date
    Apr 2004
    Location
    England, Europe
    Posts
    2,492

    Re: C++ memory issues

    On Windows I've had crashes when I try to allocate a huge amount of memory even if nothing was wrong with my code.
    My hobby projects:
    www.rclsoftware.org.uk

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  





Click Here to Expand Forum to Full Width

Featured