std::bad_alloc
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Page 1 of 2 12 LastLast
Results 1 to 15 of 20

Thread: std::bad_alloc

  1. #1
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    std::bad_alloc

    I have a program that throws a std::bad_alloc exception, even though there is clearly plenty of free memory left on my computer (see attached picture of page file usage history in Task Manager). Prior to program execution, memory usage was about 600 MB, during program execution it went up to ~1.3 GB, at which point a bad_alloc was thrown. I'm wondering, why would my program run out of memory so soon?

    p.s. my OS in WinXP SP2
    Attached Images Attached Images  
    Last edited by HighCommander4; August 7th, 2008 at 09:45 PM.
    Old Unix programmers never die, they just mv to /dev/null

  2. #2
    Join Date
    Aug 2005
    Location
    LI, NY
    Posts
    576

    Re: std::bad_alloc

    How big is the allocation? If a sufficiently large allocation is attempted on a sufficiently fragmented heap, an allocation can fail despite there appearing to be enough free memory. It's not a matter of how much memory is free, but of whether or not there is a contiguous region of free memory large enough for the allocation.
    - Alon

  3. #3
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    Re: std::bad_alloc

    Quote Originally Posted by Hermit
    How big is the allocation? If a sufficiently large allocation is attempted on a sufficiently fragmented heap, an allocation can fail despite there appearing to be enough free memory. It's not a matter of how much memory is free, but of whether or not there is a contiguous region of free memory large enough for the allocation.
    My situation is quite the opposite. I have a large number of small memory allocations. The objects that I am allocating are only 30-40 bytes in size each, but I have ~13 million of them. So, I don't think that the fragmentation is the problem.
    Old Unix programmers never die, they just mv to /dev/null

  4. #4
    Join Date
    Oct 2002
    Location
    Singapore
    Posts
    3,128

    Re: std::bad_alloc

    It does sound like a memory fragmentation problem to me. It occurs because of objects of different size are being created and destroyed. You can write a customize allocator that allocate and deallocate from a specific memory pool to minimize fragmentation.

    <edit>
    Sorry, I mean it sound like memory fragmentation problem. :P
    Last edited by Kheun; August 8th, 2008 at 03:09 AM.
    quoted from C++ Coding Standards:

    KISS (Keep It Simple Software):
    Correct is better than fast. Simple is better than complex. Clear is better than cute. Safe is better than insecure.

    Avoid magic number:
    Programming isn't magic, so don't incant it.

  5. #5
    Join Date
    Nov 2002
    Location
    Los Angeles, California
    Posts
    3,863

    Re: std::bad_alloc

    you should be aware that the default allocator adds extra bytes of memory for each allocation with new you make (I don't know the exact number, something between 4 and 32 bytes) so, instead of 35 bytes being your average allocation, it is in reality 60 bytes. then 13 million of these is about 7 MB which is consistent with your memory usage of 1.3 GB.

    You should consider using a small object allocator. The loki library has one and it is free. You can't use it with
    VC++ 6.0, but it works with 2005 etc.

    You may want to look at this as well http://www.ddj.com/cpp/184402039

    As to your problem. You have probably corrupted memory somewhere. Are you sure you don't have any first chance access violations before the bad_alloc is thrown?
    Wakeup in the morning and kick the day in the teeth!! Or something like that.

    "i don't want to write leak free code or most efficient code, like others traditional (so called expert) coders do."

  6. #6
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    Re: std::bad_alloc

    Quote Originally Posted by souldog
    you should be aware that the default allocator adds extra bytes of memory for each allocation with new you make (I don't know the exact number, something between 4 and 32 bytes) so, instead of 35 bytes being your average allocation, it is in reality 60 bytes. then 13 million of these is about 7 MB which is consistent with your memory usage of 1.3 GB.

    You should consider using a small object allocator. The loki library has one and it is free. You can't use it with
    VC++ 6.0, but it works with 2005 etc.

    You may want to look at this as well http://www.ddj.com/cpp/184402039
    Thanks, I'll look into Loki's allocator.

    As to your problem. You have probably corrupted memory somewhere. Are you sure you don't have any first chance access violations before the bad_alloc is thrown?
    I'm not sure what you mean by 'first chance access violation'. How do I know if one has occurred before the bad_alloc is thrown?
    Old Unix programmers never die, they just mv to /dev/null

  7. #7
    Join Date
    Nov 2002
    Location
    Los Angeles, California
    Posts
    3,863

    Re: std::bad_alloc

    sorry, I was not clear. You may get a bad_alloc, not because you have run out of memory, but because you have corrupted the heap. If you are lucky this may show up in the debug window as an access violation somewhere before the bad_alloc is thrown. If you are not lucky , then ...
    Wakeup in the morning and kick the day in the teeth!! Or something like that.

    "i don't want to write leak free code or most efficient code, like others traditional (so called expert) coders do."

  8. #8
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    Re: std::bad_alloc

    Quote Originally Posted by souldog
    sorry, I was not clear. You may get a bad_alloc, not because you have run out of memory, but because you have corrupted the heap. If you are lucky this may show up in the debug window as an access violation somewhere before the bad_alloc is thrown. If you are not lucky , then ...
    I'm still not entirely clear on what you mean... do you mean that, given that I still had memory available, the bad_alloc was thrown necessarily because the heap was corrupted? Or could there be some other reason for the bad_alloc being thrown?
    Old Unix programmers never die, they just mv to /dev/null

  9. #9
    Join Date
    Aug 2005
    Location
    LI, NY
    Posts
    576

    Re: std::bad_alloc

    Quote Originally Posted by HighCommander4
    I'm still not entirely clear on what you mean... do you mean that, given that I still had memory available, the bad_alloc was thrown necessarily because the heap was corrupted? Or could there be some other reason for the bad_alloc being thrown?
    I believe he means that it's one of several possibilities. Although, if a bad_alloc were to be thrown as a result of a corrupted heap, it would only be by chance, and not because that's the defined behavior if a corruption in the heap is somehow detected. More likely is that you'll get an assertion failure in the debug CRT, if you're lucky enough to not just get random behavior and crashing.
    - Alon

  10. #10
    Join Date
    Nov 2002
    Location
    Los Angeles, California
    Posts
    3,863

    Re: std::bad_alloc

    All I was saying is that bad_alloc is thrown when an allocation attempt fails. This could be due to running out of memory, or memory fragmentation, or hardware failure, or a bug in your program that corrupts the free store, or .... Could even be that something other than new is throwing the bad_alloc (like some objects constructor)
    I only mentioned this, because you should look over your program and make sure you do not have a bug, instead of just focusing on the size/fragmentation issue
    Wakeup in the morning and kick the day in the teeth!! Or something like that.

    "i don't want to write leak free code or most efficient code, like others traditional (so called expert) coders do."

  11. #11
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    Re: std::bad_alloc

    Quote Originally Posted by souldog
    you should be aware that the default allocator adds extra bytes of memory for each allocation with new you make (I don't know the exact number, something between 4 and 32 bytes) so, instead of 35 bytes being your average allocation, it is in reality 60 bytes. then 13 million of these is about 7 MB which is consistent with your memory usage of 1.3 GB.

    You should consider using a small object allocator. The loki library has one and it is free. You can't use it with
    VC++ 6.0, but it works with 2005 etc.

    You may want to look at this as well http://www.ddj.com/cpp/184402039

    As to your problem. You have probably corrupted memory somewhere. Are you sure you don't have any first chance access violations before the bad_alloc is thrown?
    I took a look at Loki's small object allocator, but I can't quite figure out how to use it. I was expecting an allocator class like std::allocator that I could use instead of std::allocator in instantiating the vectors and other containers that contain my small objects. However, I see nothing of the sort. The only documentation that I could find is rather vague and technical.

    Could someone point me in the right direction? A piece of example code using the small object allocator would be really helpful.
    Last edited by HighCommander4; August 19th, 2008 at 09:43 PM.
    Old Unix programmers never die, they just mv to /dev/null

  12. #12
    Join Date
    Nov 2002
    Location
    Los Angeles, California
    Posts
    3,863

    Re: std::bad_alloc

    The book has a chapter on it, which just happens to be online for preview

    You may also want to look here
    Wakeup in the morning and kick the day in the teeth!! Or something like that.

    "i don't want to write leak free code or most efficient code, like others traditional (so called expert) coders do."

  13. #13
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    Re: std::bad_alloc

    Quote Originally Posted by souldog
    The book has a chapter on it, which just happens to be online for preview

    You may also want to look here
    Thanks, souldog, those links were very helpful.

    The book preview had some key pages missing, but I think I got the gist of how the small object allocator works.

    From what I've read, it seems to me that the small object allocator is designed to allocate small objects one-by-one, i.e. the way you would if you used operator new.

    My problem is that my small objects are not allocated one-by-one, but are rather stored in small vectors (the vectors store a small number of small objects, ranging from 1 to 15 small objects).

    I noticed that the class SmallObject, while overriding operators new and delete, does not override operators new[] and delete[]. Does this mean that if I make my small object class derive from Loki::SmallObject and use these in a vector (which uses new[] for allocation), then the default operator new[] will be invoked?

    If that's the case, should I instead use the Loki SmallObjAllocator class directly, as in vector<MyType, SmallObjAllocator>? I read that the SmallObjAllocaotr will use a different memory pool for each object size, so this method may end up using 15 different memory pools, one for single objects, one for arrays of two objects, and so on... does that sound reasonable?

    Or is there, perhaps, a different allocator class (perhaps something from boost) that would suit my needs (millions of vectors, each with 1 to 15 small objects, each 3 bytes in size) better?
    Old Unix programmers never die, they just mv to /dev/null

  14. #14
    Join Date
    Apr 2004
    Location
    Canada
    Posts
    1,342

    Re: std::bad_alloc

    Quote Originally Posted by HighCommander4
    I noticed that the class SmallObject, while overriding operators new and delete, does not override operators new[] and delete[]. Does this mean that if I make my small object class derive from Loki::SmallObject and use these in a vector (which uses new[] for allocation), then the default operator new[] will be invoked?
    OK, it turns out I was wrong about that. vector does not, after all, use operator new[] for allocation. It uses (by default) std::allocator, which always calls the non-array version of operator new (this was a little counterintuitive for me - seeing as vector manages an array of elements, I thought for sure I'd use the array version, but apparently not). So there is no problem with using a SmallObject-derived class in vector (other than the fact that, in my case, the SmallObjAllocator is likely to be using as many as 15 FixedAllocator-s behind the scenes).

    Anyways, so I tried using the SmallObjAllocator, and it was certainly an improvement: now my program gets to ~1.6 GB memory usage (of which about 1.1 GB is my program's) before getting an out of memory error. This still seems a little low for me (I have 2 GB RAM + 1.6 GB pagefile, and each process under Windows should be able to address 2 GB of virtual memory), I guess the problem is still memory fragmentation, which is difficult to avoid when you have allocations in the hundreds of megabytes. (I should mention that using the SmallObjAllocator also resulted in a slight (maybe 10-20%) improvement in speed, in spite of using 15 different FixedAllocators behind the scenes).

    In any case, I can estimate now that my program will require ~4 GB of memory to run to completion, which is simply not possible under 32-bit Windows (again, 2 GB virtual memory limit), so I'll need more than a good allocator to get it to work (I'll probably have to manually "page" some of my data to the hard drive, or something similar).

    In any case, I thank you for all your help, and I will be back if I encounter any other problems.

    Old Unix programmers never die, they just mv to /dev/null

  15. #15
    Join Date
    May 2007
    Location
    Scotland
    Posts
    1,145

    Re: std::bad_alloc

    Just wondering, but what are you trying to do? Why do you need to simultaneously create 13 million entries?

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center