my c++ application cannot allocate more than 400MB with new[]

[ybenaabud]

hi,
i am developping an mfc application than need to handle a large amount of data.
my PC have 3Gb of physical RAM (i am using only 700MB of it).
I can not understand wy i can´t allocate more than 400 continus Mega bytes with new[] or malloc.
like this: short *buffer = new short[200000000]; or short *buffer = (short*) malloc(400000000);
I do not know if there is something to do with heaps size ??

Thank you for your help.

[Jehjoa]

To be honest I'm not sure why it doesn't let you do it, but why would you even want to allocate such a large amount of memory in one go? If you're trying to load a large file into memory, have a look at file mapping on MSDN.

[ybenaabud]

thank you for your responses but the thing is that i am working with medical data from CT images (3D ray casting) and the data must be in memory so the rotation of the 3D object can be fast.
I can do more that one allocation (100 of 4 MB) but i would prefere only one of 400 or 600 or more even when I have 3 GB of RAM.

[ovidiucucu]

You can use HeapXXX functions as shown in the example below:

#define HANDLE_ERROR(x) if(!x) {HandleError(); return;}

void CHeapTestDlg::OnButtonHeapTest() 
{
   const SIZE_T bytes = 0x40000000; // 1GB

   // create a private "growable" heap which has the maximum size
   // limited only by the available memory (see HeapCreate in MSDN for more details).
   HANDLE hHeap = ::HeapCreate(0, 0, 0);
   HANDLE_ERROR(hHeap);

   // alloc 1GB
   short* buffer = (short*)::HeapAlloc(hHeap, 0, bytes);
   HANDLE_ERROR(buffer);
   // ...enjoy of 1GB buffer.
   // ...
   //... free the buffer when is no more needed.
   BOOL bRet = ::HeapFree(hHeap, 0, buffer);
   HANDLE_ERROR(bRet);

   //... destroy the heap when is no more needed.
   bRet = ::HeapDestroy(hHeap);
   HANDLE_ERROR(bRet);
   // ...
}

void CHeapTestDlg::HandleError()
{
   DWORD dwError = ::GetLastError();
   // ... handle error (see this FAQ).
}

Note that available memory is limited by the operating system: less than 2 or 3 GB for 32-bit Windows (for sure is much more higher on 64-bit systems such as Windows Vista x64).
And... if directly using of HeapXXX looks ugly, you can define a "Buffer" class which encapsulates them by overloading operators new and delete.

[Jehjoa]

Okay so I got interested and wanted to test it out, so I launched Visual Studio 2008 and started hacking away.

Here is my code:

common.h

#ifndef _COMMON_H_
#define _COMMON_H_

#define WIN32_LEAN_AND_MEAN
#define OEMRESOURCE

#define _WIN32_WINNT		0x0501	// Windows XP
#define WINVER			0x0501	// Windows XP
#define _WIN32_IE		0x0600	// Internet Explorer 6.0

#include <typeinfo> // for typeid, also includes <exception>

#include <windows.h>
#include <wchar.h>

#endif // _COMMON_H_

memeater.cpp

#include "common.h"

#define OMFG_THIS_NUMBER_IS_HUGE_LOLOL		1024*1024*1024/2 // 200000000

typedef struct __MEGABYTE
{
	unsigned char bytes[1024*1024];
} MEGABYTE;

int wmain(int argc, WCHAR ** argv)
{
	short * ThisPointsToMyShorts = 0;
	MEGABYTE * pMB = 0;
	unsigned long ulCounter = 0;

	try
	{
		ThisPointsToMyShorts = new short[OMFG_THIS_NUMBER_IS_HUGE_LOLOL];
	}
	catch (std::exception & e)
	{
		wprintf_s(L"Failed to allocate %d bytes!  An exception of type %S has occured.\n", OMFG_THIS_NUMBER_IS_HUGE_LOLOL * sizeof(short), typeid(e).name());
		return -1;
	}

	wprintf_s(L"Successfully allocated %d bytes!\n\nLet's see how far we get if we go in steps of 1 MB...\n", OMFG_THIS_NUMBER_IS_HUGE_LOLOL * sizeof(short));

	delete[] ThisPointsToMyShorts;
	ThisPointsToMyShorts = 0;

	try
	{
		while(1) // this WILL cause an exception sooner or later  :)
		{
			ulCounter++;
			wprintf_s(L"%d MB... ", ulCounter);
			pMB = new MEGABYTE[ulCounter];
			wprintf_s(L"Worked!\n");
			delete[] pMB;
			pMB = 0;
		}
	}
	catch (std::exception & e)
	{
		wprintf_s(L"\n\nAllocation failed at %d megabytes!  An exception of type %S has occured.\n", ulCounter, typeid(e).name());
		return -1;
	}

	wprintf_s(L"Umm... Okay... How did we manage to end up here!?");

	return 0;
}

As you can see by the #define in memeater.cpp, I first tried what you tried; Allocating 200000000 shorts. Worked flawlessly. Then I tried some other numbers which worked too. Then I decided to let it try every MB until new throws an exception.

Here's the output of the program:

Successfully allocated 1073741824 bytes!

Let's see how far we get if we go in steps of 1 MB...
1 MB... Worked!
2 MB... Worked!
3 MB... Worked!
4 MB... Worked!
5 MB... Worked!
...
1728 MB... Worked!
1729 MB... Worked!
1730 MB... Worked!
1731 MB... Worked!
1732 MB... Worked!
1733 MB...

Allocation failed at 1733 megabytes!  An exception of type class std::bad_alloc
has occured.

It took about 10 minutes before it finally failed! This is on a machine with Windows XP SP3, and 2GB of physical memory...

So I don't know what to tell you dude... I'm thinking the problem lies with the compiler you use, or maybe your Windows SDK is outdated. Worst case scenario would be your RAM is getting faulty, but I'd wait with that conclusion until someone more knowledgeable about compilers comes along.

[ovidiucucu]

So I don't know what to tell you dude... I'm thinking the problem lies with the compiler you use, or maybe your Windows SDK is outdated. Worst case scenario would be your RAM is getting faulty, but I'd wait with that conclusion until someone more knowledgeable about compilers comes along.

That is not a matter of compiler, outdated Windows SDK, or RAM getting faulty.
The "maximum-maximorum" allocable memory depends on what amount of virtual memory a process can access in user-mode: ~2GB or ~3GB for x86 (32-bit) Windows systems, as I already said.

[Jehjoa]

That is not a matter of compiler, outdated Windows SDK, or RAM getting faulty.
It depends on what amount of virtual memory a process can access in user-mode: ~2GB or ~3GB for x86 (32-bit) Windows systems, as I already said.

Yes, we know. But, as I understand it, ybenaabud's problem is that he (or she ) can't even allocate 400 MB on a 3 GB system while I can allocate 1732 MB on a 2 GB system with 1 call to operator new.

[ovidiucucu]

Ah, forgot to add "and... of course on the amount of memory already used.".
And... on others....

[ later edit ]
In Windows, the amount of memory that a process can access, DOES NOT depend on the amount of RAM.

[ybenaabud]

I think that there is the problem of memory fragmentation too (for allocation continus 400MB or more)

The "maximum-maximorum" allocable memory depends on what amount of virtual memory a process can access in user-mode: ~2GB or ~3GB for x86 (32-bit) Windows systems, as I already said.

I tried with different new[] 1 MB each time and I can reach 1,9GB. so my process can only access 2GB theorical memory. how can turn it to 3GB in my XP Pro PC ?

ah... ybenaabud is he

think you

[MrViggy]

There's a boot setting (I think in the 'boot.ini' file on the system), and then you have to enable access to the extra GB with a linker switch, when you compile your app.

Viggy

[ybenaabud]

yes /3GB in boot.ini. but when a do it in my windows xp pro pc with 3GB of physical ram I lose the network connexions !!

[Ajay Vijay]

For larger allocations (>1MB), new is sluggish on Windows. Always use HeapAlloc for larger memory. Try your test code with new and HeapAlloc, you'd find the speed difference. With Heap functions you can definitely allocate larger chunks of memory which aren't possible with new.

To prove this assertion, just single step (each and every internal function call), the new call and HeapAlloc call. For better memory management you can have separate heap (i.e. not the process' default heap retrieved by GetProcessHeap).

On Linux, new is as faster as HeapAlloc in Windows. But what it does is: what HA does on Windows. That is - it (OS) just marks memory "to be allocated" for this program. And when memory read/write is required, it actually allocates memory. No, it doesn't degrade performance (don't know about OSs complicated logic behind it).

For more larger memory requirements, you can use PAE, or can go for x64.