Out of virtual memory?

[wlof]

Hello everyone,

I have a problem with an application. It establishes a bunch (between 5 and 100) of client connections, plus 2 server connections, to other applications and sends, receives and processes commands using these connections. Each connection has its own thread. There are also a few other threads dedicated to processing, and the application frequently creates temporary threads that serve a simple timer purpose (all they do is Sleep(), send a signal, and terminate).

For a stress test, I repeatedly closed and reestablished one of the connections (doesn't matter which one). After a while (a few hundred times), my application started expecting strange behavior. Further investigation showed that the application was unable to connect sockets (calls to connect() systematically fail with error code WSAENOBUFS) and create new threads (calls to CreateThread() systematically fail with error ERR_NOT_ENOUGH_MEMORY).

I added a snipet of code to call GlobalMemoryStatus() when CreateThread() fails, and here is the output:

The MemoryStatus structure is 32 bytes long. It should be 32.
26 percent of memory is in use.
There are 2138554368 total bytes of physical memory.
There are 1577644032 free bytes of physical memory.
There are 4125798400 total bytes of paging file.
There are 3577077760 free bytes of paging file.
There are 2147352576 total bytes of virtual memory.
There are 11612160 free bytes of virtual memory.

So it seems my app is running out of virtual memory. What could cause this? Common memory leaks? Or something more subtle?

Sorry if I'm not being very clear, English is not my first language and the exact problem is hard to describe without explaining every inner working of the app. Basically I'm looking for any tips as to what causes my app to run out of virtual memory.

Thanks in advance!

[Lindley]

You have some type of resource leak. Are you remembering to close all HANDLE objects? Simple memory management leaks can mostly be eliminated by preferring STL containers and smart pointers over manual new/delete handling, so that's worth considering.

Also, I recommend against frequent creation/teardown of threads. Consider using Windows' built-in timer functionality instead, or maintain a dedicated timer thread which never terminates (but may spend a lot of time doing nothing). A thread pool is also a reasonable idea. Frequent thread creation won't leak resources if done right, but it's inefficient.

[wlof]

Thanks for your reply.

You have some type of resource leak. Are you remembering to close all HANDLE objects?

I think so, but it's hard to be sure. Is there a function like GlobalMemoryStatus(), or some external utility, that could help me track the number of opened HANDLE objects?

EDIT: It seems Process Explorer can do it. I'll try and see what I can find.

Simple memory management leaks can mostly be eliminated by preferring STL containers and smart pointers over manual new/delete handling, so that's worth considering.

Also, I recommend against frequent creation/teardown of threads. Consider using Windows' built-in timer functionality instead, or maintain a dedicated timer thread which never terminates (but may spend a lot of time doing nothing). A thread pool is also a reasonable idea. Frequent thread creation won't leak resources if done right, but it's inefficient.

Thank you for these advices. I was already aware of most of them, unfortunately this is a legacy application with some very weird (i.e. bad) design choices that I'm not at liberty to modify. Though maybe they'll let me if I can pinpoint the exact problem and prove that a design change is necessary.

[wlof]

I figured out what was wrong.

Many of the temporary threads I mentioned in the first post need to be destroyed prematurely (i.e. while they're still blocked in the Sleep() function and before they can send their signal). This destruction was done using TerminateThread(), which was wrong: TerminateThread() does not deallocate the virtual memory used by the thread's stack (by default, 1 Mb). So after roughly 2 000 such threads were destroyed, the application ran out of virtual memory and started showing the strange behavior I explained.

The solution was to modify the temporary threads so that instead of Sleep()ing for a long time, they Sleep() by small increments of time and regularly checks a flag. If the flag is set, the thread ends itself properly, deallocating the memory used by its stack and correcting the problem.

[MrViggy]

I figured out what was wrong.

Many of the temporary threads I mentioned in the first post need to be destroyed prematurely (i.e. while they're still blocked in the Sleep() function and before they can send their signal). This destruction was done using TerminateThread(), which was wrong: TerminateThread() does not deallocate the virtual memory used by the thread's stack (by default, 1 Mb). So after roughly 2 000 such threads were destroyed, the application ran out of virtual memory and started showing the strange behavior I explained.

The solution was to modify the temporary threads so that instead of Sleep()ing for a long time, they Sleep() by small increments of time and regularly checks a flag. If the flag is set, the thread ends itself properly, deallocating the memory used by its stack and correcting the problem.

A better option would be to signal the threads when they need to be woken up. Look at WaitForSingleObject

Viggy

[wlof]

A better option would be to signal the threads when they need to be woken up. Look at WaitForSingleObject

I just tried it, it seems to work fine and it makes the code simpler and more elegant. Thanks for the suggestion!

[MrViggy]

No problem! The other nice thing about this method is you don't waste CPU cycles by polling.

Viggy