Memory mapping & process size

[sszd]

Can memory mapping be used to solve a problem with the size of a running process being too large?

I've been told that Solaris has a process limit of 4GB. There's a process on our system that keeps coring, and it never cores in the same place. Examining the core files shows that usually an address for some argument being passed to some random function has been stomped on. I've also been informed this problem will go away when certain files on the system are cleaned up. These files are read in by the process in question upon startup. Some of these files are rather large, and others, which are quite small are quite numerous, in the vicinity of hundreds of thousands (no kidding!!!) One of the original programmers told me that this process does use memory mapping, but when there are too many files, or they have gotten too large the process will exceed the 4GB limit and crash.

I won't have access to the code until sometime next week, so I won't be able to look through it until then to try to see what the heck it's doing. I'm just trying to get some initial info for the moment before I get around to it.

If the code truly is using memory mapping then it must be doing something rather unusual, or I don't quite understand the concept of memory mapping with regard to the ultimate size of a running process. Surely a memory mapped file isn't included as part of the size of a process is it?

I've personally never had to use memory mapped files, so any info regarding this would be helpful. Thanks.

[originalfamousjohnsmith]

He needs to not run out of memory, that's all there is to it. It's kind of insane that it just mindlessly reads files and grabs everything it can like that. It's more like a virus than a program.

Memory mapped files are probably not relavent to you at all. It's what the program does that's a problem, not so much how it does it, fixing that depends on the exact details of what it does but it's ultimately just a very poorly written program.

[sszd]

It's kind of insane that it just mindlessly reads files and grabs everything it can like that. It's more like a virus than a program.

Without going into a lot of detail, since I don't know much yet, I don't believe the program mindlessly reads every file it can get its hands on. The data read in is from flat files that were retrieved by a completely unrelated process from an extremely large database that is not directly accessible by the process in question. The contents of these files are used by the process in question to do a kind of pattern matching against some live data being run through the system. Hence, it needs to be extremely fast. I don't know exactly how the files correlate with one another, so I don't know if they can be read in at different times. From what I understand, all the data needs to be available at a given moment. I'm not saying this couldn't have been designed differently, but I'm told memory mapping was used to allow all the data to be available at one time.

Memory mapped files are probably not relavent to you at all. It's what the program does that's a problem, not so much how it does it, fixing that depends on the exact details of what it does but it's ultimately just a very poorly written program.

You're probably right regarding it being poorly written. I'm told, in the beginning, when the process was first put into operations, the database wasn't nearly as large as it is today, and the files weren't nearly as numerous. Over time, this has obviously changed. It seems to me this program is merely a time bomb that will eventually reach it's limits. For the time being it seems that the file cleanup is what they're currently doing to solve the problem, which is being done more and more frequently. Regardless, I would still like some basic answers to my first post. Thanks.