Debugging suspected memory overwrite

[nixius]

Hello,

My team and I have been creating some software from a large engine that we have complete access to but are not hugely familiar with.

The software crashes intermittently, when the crash does crop up with some amount of reliance, any changes made to the software (to log function calls etc) stop it from occurring.

This has lead us to believe that we have some form of memory overwrite. I have personally been through every single memory allocation and array in the software to look for any obvious mis-use or lack of bounds checking to no avail. Fortunately, we mostly use stl containers so it wasn't too rough for the software, however, we don't have the luxury of time to do the same with the engine.

I was hoping that any readers of this thread would have some suggestions to help track down memory overwrites. All suggestions would be greatly appreciated.

One method I have seen a few times whilst searching, is to overload new and delete, then add a buffer to the beginning and end. Upon deletion of the object if the buffer has any data in it, there has been an overwrite.

I am not very technical so this solution seems somewhat daunting, if anyone has used this technique or has any further/related reading on it, I would be equally appreciative of you sharing such information.

Thank you for reading,
nixius

Edit: using C++ and VS2010, Windows7.

[Amleto]

what other possible causes have you considered, and why have you rejected them? e.g. threading issue.

I'm sure googling will throw up plenty of information. You may need to read up on 'placement new'.

[VictorN]

...
The software crashes intermittently, when the crash does crop up with some amount of reliance, any changes made to the software (to log function calls etc) stop it from occurring.

Didn't you try to debug your app to find out what where and why goes wrong?

[nixius]

Hi.

When the program exits, the log outputs:

CAUGHT[signal] : SIGSEGV : Segment violation (11)

CAUGHT[UnhandledException] : Access violation @ 0X7C9118CA: Bad read on 0x00000024 The thread tried to read from or write to a virtual address for which it does not have the appropriate access.

We don't do any multi-threading.

To be honest, a memory overwrite is the only thing I could think the issue to be.

Of course we have all tried debugging, I did omit to mention that the issue has only occurred on the 'live PC' (This machine goes out to clients with the software on). None of our dev machines in debug/release have been able to reproduce the bug.

I suppose it could be the RAM on the live machine.

I have been googling a lot, but mostly I find people talking about code for memory leaks, which is not very helpful for us since 99% of our memory is assigned at the start and released at the end of the software running cycle, or commercial software which is not an option for us.

Thanks for the responses.

[VictorN]

Could you try to use map file to locate the place of Access violation in your code?
Finding crash information using the MAP file

[nixius]

This has been mentioned, unfortunately on the latest occasion of the crash, the .Map file was not copied across to the live machine =(

We have a rough idea already from the game logging (i.e it happens between where it logged last and didn't get to the next log message) where the crash happened last however, it runs through this same piece of code absolutely fine several times and does not always crash which is odd in itself.

The crash has been around the same point in the software since it first started happening, but after code changes (bug fixes, adding logging etc) the crash doesn't occur (or the frequency drops) or it happens slightly later/earlier on.

When looking around that area for suspect code, all arrays are containers like std::vector/map/list (so no accessing outside of bounds), the only alarm bell is there are 4 'new' objects created around that time but even then the log either reports it crashes at roughly the time of the the first two being allocated, later on creating the second 2, or it does not crash at all and runs through.

My point with all this information is, that it is not really the location of the crash that is the problem, more over it is the fact it happens randomly, in different locations; this leads me to believe it is a memory overwrite. I think.

Mulling it over, if it is always around a 'new' call, perhaps it may be a problem allocating memory under certain circumstances... hmmm...

This one has had me frustrated for several days. I have told my colleagues to put .map files on for future builds, thanks for the reminder.

[VictorN]

This has been mentioned, unfortunately on the latest occasion of the crash, the .Map file was not copied across to the live machine =(

Why do you think you need to copy " .Map file ... across to the live machine"?
Map file is created while compiling + linking the exe on your dev machine. Then you can use it to find out the crash location...

[nixius]

Because the project has been been modified since, therefore the map file does not match the executable that crashed.

[VictorN]

Because the project has been been modified since, therefore the map file does not match the executable that crashed.

Ah.., well. Understand.

[VladimirF]

Because the project has been been modified since, therefore the map file does not match the executable that crashed.

Well, if you don’t preserve auxiliary files for each released version (you really should), you sure can pull that version from your source code version control system and rebuild it, right?

[nixius]

It was a local version, not checked into any version control =(. Even so, it happened with a lot of infrequency, and the .map file will only point to a function if I remember correctly?

Edit: I should mention we have a 'live' system for us to test on, this is not a final release procedure or anything, this is part of our testing process.

[Eri523]

When looking around that area for suspect code, all arrays are containers like std::vector/map/list (so no accessing outside of bounds), the only alarm bell is there are 4 'new' objects created around that time but even then the log either reports it crashes at roughly the time of the the first two being allocated, later on creating the second 2, or it does not crash at all and runs through.

Since I think we can reasonably assume that the VC++ CRT's new is bug-free, I think there's some chance that the crash is caused by the costructors of the classes in question or something that is called by them, directly or indirectly. Perhaps it's worth to focus the search for the bug on that code. (This doesn't mean though, that you only should look for the bug there.)

The address your app is trying to read from which causes the access violation indicates that the cause is the use of an invalid pointer. This may or may not be due to memory corruption. It may also be a logic bug, like, for instance, indexing a null pointer.

[nixius]

Thank you for your response Eri.

I will take another look into that class.

I am still quite confident that it is a memory overwrite, so going back to the first post, any tips on how to debug those would still be appreciated.

Thanks.

[Lindley]

Does this software contain Windows-specific code? The best tool for debugging these sorts of issues is valgrind, but it's only available on Linux (and I think OSX).

There are similar tools available for Windows, but they aren't free (so far as I know). What you want is a dynamic code analysis tool.

By the way, usage of STL containers does not guarantee lack of out-of-bounds access. It's true that Microsoft's vector implementation does bounds-checking by default on both [] and at(), but this can be disabled (and often is for speed reasons), and checking of [] is actually not mandated by the standard so if you aren't using the default Dinkumware implementation, you may not even get this.

[Paul McKenzie]

Of course we have all tried debugging, I did omit to mention that the issue has only occurred on the 'live PC' (This machine goes out to clients with the software on). None of our dev machines in debug/release have been able to reproduce the bug.

Your application should have produced a crash dump. If not, then consider putting one in your application and have it enabled. Then when the application crashes, you get the dump from your clients and you can then debug the issue on your machine by loading the crash dump in Visual Studio.

Read this thread:
www.codeguru.com/forum/showthread.php?t=517072

Also, you have something called "remote debugging" if you have TCP/IP access to one of these machines.

Also, mentioning that it crashes only on live machines is not out of the ordinary. I bet that every one that has responded to you has had applications that "work" in the shop, but crash elsewhere.

I suppose it could be the RAM on the live machine.

If you're a C++ programmer and said anything like that, in many companies, it could get you fired, or at the very least, have your stature as a C++ program greatly diminished.

A C++ (or C) programmer never blames hardware or anything else for their program crashing -- the blame always goes to the program having bugs. Unless you have verifiable proof that the problem is elsewhere, never mention anything having to cause the problem except for the program itself.

Another thing you don't do -- don't start moving code around until you've debugged the crash and know exactly why it happened. What I see some coders doing is rearrange some code here and there, and voila, the crash is gone without any explanation. For example, they introduce a couple more variables, or they remove a function, etc. Unfortunately, all that has happened is that the bug was moved to another part of the application, and who knows when it will appear again.

Regards,

Paul McKenzie