Releasing heap memory???

On a linux system at work, there is this one particular process that many of us have observed on occasion that will continue to grow in memory during specific processing periods, and never shrink. We've observed this through some command line tools like "top". Once this process reaches some critical point, the system simply locks up. No other processes seem to produce any significant work. They've tried to mitigate some of this by altering queue sizes and timing constraints for this and other processes. Some of us, me included, have mentioned this problem to one of the programmers of this particular system, and have indicated that there must be some sort of memory leak in this particular process. The programmer indicated that once a process allocates memory on the heap, that even though you later release that memory, the process still maintains usage of that memory, and no other processes will have access to, or be able to use it. Is this true??? This is completely contrary to everything I've believed for the last umpteen years. He claims that this is simply the way computers are architected (irrespective of the OS), and there is nothing you can do about it. I'm flabbergasted!!!

I can't seem to find anything on the internet with regard to how releasing memory from one process will or won't affect other processes. All I've run across is how you should always release heap memory you've previously allocated as the programmer is completely responsible for this type of memory manipulation in order to prevent memory leaks. Nothing about how a process will still continue to own that memory even though you've release it. If this is true, I guess I need to go back to Computer Basics 101!!!

Can someone please confirm or deny this accusation?

The programmer is right, a process may not give back memory to the OS but this isn't an excuse for not fixing the problem. If the memory increase is unrelated to an increase in workload it usually is a memory leak.

Usually the memory footprint of a process will never shrink, only expand. It depends on the runtime system of the programming language technology you're using. When a programmer releases heap memory it's to the runtime system. It's then up to the runtime system whether to pass memory on to the OS. The usual strategy is to keep it. If the program required this much memory before it most likely will require it again. Sometimes you can specify an upper limit to the amount of heap space the runtime system is allowed to allocate. That's the case with the Java VM for example. If a program needs more it gets terminated with an out-of-memory exception.

But leaks aren't always the source of these problems. It can also be due to heap fragmentation for example. Say a program on occasion allocates a very large chunk of contiguous memory. If the heap has become fragmented the requested memory may not fit anywhere and the heap ends up being expanded even when there's plenty of space available. This cannot happen with Java because the heap is constantly reorganized but a C++ runtime system cannot do that. It cannot move around live heap objects because the language requires them to stay put.

Quote:

---

Originally Posted by nuzzle

The programmer is right, a process may not give back memory to the OS ....

---

Sorry to disagree.. But if the Program is written correctly, and has a proper Memory management, it should return unused memory to the OS. There are some older memory managers that may not return all Freed up memory to the OS, but that was quite some time ago..

The Proggy obviously does have a memory leak.. Even if he's using older Memory management, the proggy should be reusing this pre-allocated memory irrespective..

There are some programming techniques that can cause a problem with the Heap if memory is not passed back to the OS.. IE:

Code:

---

//Pseudo code
define Outlist as Array of Dataobject
Datareader = SQLQuery("Select * from table order by ID asc")
For Each DRItem in Datareader
    Redim Outlist (Outlist.lastindex + 1)
    Outist (Outlist.lastindex).id = DRItem.Selectcolumn("ID")
    Outist (Outlist.lastindex).Name = DRItem.Selectcolumn("Name")
    Outist (Outlist.lastindex).Address = DRItem.Selectcolumn("Address")
    ......... //etc.
Next
Return Outlist

---

While this code looks well refined and will only use the QTY of memory needed for all the Data read from SQL, it has one Glaring Problem.
Every Redimming of the array (Size adjustment) requires more memory that the previous one.. This will lead to heap fragmentation like nuzzle describes.

The programmer is trying to blow smoke up someone's as, if he thinks that he can blame incorrect memory management to the OS...

have a look at this for more info about correct memory management (Note this piece is from back in 2004)

Quote:

---

Originally Posted by nuzzle

The programmer is right, a process may not give back memory to the OS but this isn't an excuse for not fixing the problem. If the memory increase is unrelated to an increase in workload it usually is a memory leak.

---

If this is true, then there may not actually be a memory leak. There is definitely high levels of workload on occasaions. Data comes in in spurts... there'll be nothing for periods of time, then suddenly large amounts of data are received. It's during those lull periods that we do not see a release of memory from one particular process. So, again, if you're correct, in that a process may possibly not return memory to the OS, then that would explain why we see no decrease in memory. However, we do see decreases in other processes. As I mentioned, one of the tools used to monitor memory usage is "top". Is this not a valid way to assess this possible problem?

Quote:

---

Originally Posted by nuzzle

Usually the memory footprint of a process will never shrink, only expand. It depends on the runtime system of the programming language technology you're using. When a programmer releases heap memory it's to the runtime system. It's then up to the runtime system whether to pass memory on to the OS.

---

The process in question is written in C++ using a GNU compiler (version 4.1.2). It runs on a Redhat linux OS (version 5.4). I have no idea how this runtime system works, or even how to find out. However, if memory is not being released back to the OS, then why would we see memory reductions (again, using "top") with other processes?