malloc/HeapAlloc returns NULL but there is plenty of memory

[martinf31]

Hi,

I have a situation in which malloc() returns NULL and sets errno to ENOMEM. But the CRT heap (which is growable) has plenty of memory to work with. At the time of malloc, my process memory is about 900 MB.

The malloc() I'm doing is 80 megabytes, and fails. If I do a 60 MB allocation, it succeeds. After that, a 50 MB allocation, followed by another one, and another one *also succeed*: clearly, I still have a lot of memory left but the 80 MB malloc seems too "big" to digest for the OS.

I'm using Windows 7 x64 SP1 with 4 GB RAM. My process is a 32-bit process. I'm using the Low Fragmentation Heap, which is the default on Win 7 - I have also verified with HeapQueryInformation.

What the heck can cause malloc() to fail if this is not a lack of memory ? Is my memory too fragmented ?? This is really weird, I have never seen this behavior before.

Thanks,
Martin

[ovidiucucu]

Countless are the ways of the Lord in a heap.
For large amount of memory, use VirtualAlloc instead of malloc/HeapAlloc.

[martinf31]

Thanks for your answer but this does not seem to be the correct answer

The CRT creates the Heap this way:

HeapCreate(0, BYTES_PER_PAGE, 0)

in other words, it sets the heap maximum size to 0. In this case, according to the documentation of HeapCreate, for blocks larger than 512KB HeapAlloc calls VirtualAlloc automatically.

HeapCreate
http://msdn.microsoft.com/en-us/libr...=VS.85%29.aspx

So that doesn't explain why malloc fails with ENOMEM.

Thanks,

Martin

[Paul McKenzie]

So that doesn't explain why malloc fails with ENOMEM.

The malloc() I'm doing is 80 megabytes, and fails. If I do a 60 MB allocation, it succeeds. After that, a 50 MB allocation, followed by another one, and another one *also succeed*: clearly, I still have a lot of memory left but the 80 MB malloc seems too "big" to digest for the OS.

The call to malloc() requires contiguous bytes of memory to be free. You left out that very important aspect.

Just because you have all of this memory doesn't guarantee you can get a contiguous chunk of memory of that size. Maybe the heap became fragmented and there is not enough free, contiguous space. Heap managers use different strategies, and maybe fragmentation and the heap manager's strategy (best fit, first fit, etc.) is causing the issue.

Secondly, test your theory by writing a very simple program:

#include <stdlib.h>

int main()
{
    char *p = (char *)malloc(60000000);    
    char *p2 = (char *)malloc(50000000);    
    char *p3 = (char *)malloc(50000000);    
    free(p);
    free(p2);
    free(p3);
    char *p4 = (char *)malloc(80000000);
    free(p4);
}

Regards,

Paul McKenzie

[Eri523]

Good point, Paul, however, I don't think it's likely your sample program will actually cause heap fragmentation with a flawlessly working heap manager: The heap manager should be able to easily merge adjacent free blocks. Freeing only p2 before allocating the memory block for p4 is IMO much more likely to provoke the "desired" result.

[martinf31]

I understand what you guys mean but consider the following:

The thing I am extremely surprised with, is that my process memory totalizes 900 MB, so it means I still have about 1.1 GB of unused process memory (max 2 GB per process on Win32, by default).

My process Peak memory usage *also* is about 910 MB, which means the remaining 1.1 GB have been left completely untouched.

In this situation, how come the OS is unable to honour a 80 MB allocation, considering that the upper 1.1 GB of process memory is still in virgin condition ?

Does that mean that the OS allocated the 900 MB of used memory over the whole 2 GB address space and, as a matter of fact, there is not a single continuous block of 80 MB in size over this very 2 GB address space ? (isn't that a bit surprising ?)

Martin

[Eri523]

Another idea, Martin: If you really want to know what's going on with your CRT heap, you may want to try and find out for sure with your own actual code. You may want to have a look at the _heapwalk() CRT function and perhaps add some diagnostic code using it. Though the MSDN documentation is as of VC6, the function is still there and operational in VC++ 2010. I can't guarantee that it works with the Low Fragmentation Heap, though, still I think it's worth a try.

[ovidiucucu]

The thing I am extremely surprised with, is that my process memory totalizes 900 MB, so it means I still have about 1.1 GB of unused process memory (max 2 GB per process on Win32, by default).
...

C'mon, Martin! Take a look at "virtual" in a dictionary.

[martinf31]

You may want to have a look at the _heapwalk() CRT function and perhaps add some diagnostic code using it.

I used it to calculate the amount of used CRT heap memory, which is about ~250 to 300 MB at the time of the malloc failure. Are you suggesting something else I could do with this function ?
The size of my Java heap, obtained with System.gc.maxMemory(), is 330 MB.

[ovidiucucu]

...
The size of my Java heap, obtained with System.gc.maxMemory(), is 330 MB.

We are dealing with Java stuff, then complain about (process) memory fragmentaton. Really cool!...

[Eri523]

I used it to calculate the amount of used CRT heap memory, which is about ~250 to 300 MB at the time of the malloc failure. Are you suggesting something else I could do with this function ?

You can use it to enumerate the heap blocks, so it allows you to determine, among other things, the size of the largest available block and/or largest contigous sequence of available blocks. (Note that, as I recall the way the heap manager works, the effectively available memory may be more than the sum of free blocks because the heap manager may be able to expand the heap on demand. However, too much available memory doesn't seem to be your problem here anyway... )

We are dealing with Java stuff, then complain about (process) memory fragmentaton. Really cool!...

[Igor Vartanov]

Windows 7 64-bit, 32-bit exe

#include <windows.h>
#include <stdio.h>

int main()
{
	int size = 500 * 1024 * 1024;
	BYTE* pbuff = (BYTE*)malloc(size);
	if (!pbuff)
	{
		printf("Failure\n");
		return 1;
	}
	memset(pbuff, size, 0);
	printf ("Success\n");
	return 0;
}

D:\Temp\17>17.exe

Success

[Paul McKenzie]

Are you suggesting something else I could do with this function ?

One thing no one has mentioned so far -- maybe your program has corrupted the heap manager.

The call to malloc() is not the same as a call to the OS heap functions. The malloc() call goes through the 'C' runtime heap manager first, and then that heap manager decides to either go to the OS for memory, or move a few pointers around and return to you a "free block". If you need proof, you can debug into the call to malloc(), if you have the runtime source code.

In other words, with malloc, you're dealing with a 'C' runtime structure that has the possibility of being corrupted by a wayward progrram.

Regards,

Paul McKenzie

[martinf31]

One thing no one has mentioned so far -- maybe your program has corrupted the heap manager.

Paul, thanks for your reply. I've got some more information:

A) I tried to run my application under Windows 7 x64 under Oracle's JRockit JVM, instead of Sun's HotSpot JVM.

Sun's javaw.exe executable hasn't got the Large Address Aware header flag set, while Oracle's JVM does. Guess what: my application indeed runs fine with Oracle's JVM. The reason ? Running a 32-bit process with the LAA flag under a 64-bit OS gives an address space of 4 GB, not 2 GB.

B) At the time malloc fails, Process Explorer displays the following information for my process (javaw.exe):

- Private Bytes: 900 MB
- Peak Private Bytes: 910 MB
- Virtual Size : 1.5 GB (eg. reserved memory, either committed or not)

My interpretation of the situation, at the time malloc fails, is the following: as you said, there is no block of 80 MB of contiguous memory in the Virtual Address Space of the process.

One question remains, though: how come can my process reach such a huge amount of Virtual Size, that is, memory that has been reserved but NOT committed (Virtual Size = reserved size but not necessarily committed)?

In a C/C++ program, I usually reserve and commit memory at the same time, with malloc, and then free this memory. The memory never goes through the "reserved but not yet committed" state. So it seems having the Virtual Size grow is not because of C/C++ activity here.

Did the JVM cause such a difference between committed (900 MB) and reserved memory (1.5 GB) you think ? Is this an acceptable conclusion ?

Best regards and thanks again for your reply,

Martin

[martinf31]

We are dealing with Java stuff, then complain about (process) memory fragmentaton. Really cool!...

Wrong again. The JVM allocates one large, fixed, reserved memory block and commits portions of it as Java memory usage climbs. You can verify this easily with PrcView or even VMMap. You'll find the reserved memory corresponding to the Java runtime's maxMemory value.

A regular C/C++ application only using malloc/new has the same problems:

http://stackoverflow.com/questions/4...-in-my-process