How to debug without debugging?

[Access_Denied]

So I'm in a bit of a pickle. Recently, I released an update for the program I work on. This update was a bit unique because it was the first time I did a branch and merge in Git with the code base. But the merge went fine. There were only 10 or so conflicts and I fixed them all. So I continue on with the release and put it up. My boss and I tested extensively. No issues. But then we had customers download and all hell broke loose.

Basically, we had 2 customers who couldn't use the program. They would attempt to open their database and the program wouldn't start. I'm pretty sure I've traced the issue based on the description, and what I think happened is that one of the main objects in our program is silently failing during construction. Here's the real problem: I CANNOT reproduce this issue. The two customers were on Windows 7 and XP. I've tried 2 different XP machines, 2 7 machines and even a Windows 8 machine. I've tired with 10 different databases. No luck.

The good news is that this particular customer is fairly tech savvy and agreed to test our program for us. That's great and all, but where I need help is how best to do that. I don't want to send her 100 builds to test. What's the best way to have her get information to me? I've thought about logging, but I'm not sure if that would be viable. What happens if the program crashes during logging? Won't that corrupt the log file? I have pretty much 1 or 2 chances for this lady to help me, and I really need to make the most out of them. How should I go about this?

[Paul McKenzie]

Basically, we had 2 customers who couldn't use the program. They would attempt to open their database and the program wouldn't start. I'm pretty sure I've traced the issue based on the description, and what I think happened is that one of the main objects in our program is silently failing during construction.

Look at the code and make sure you're checking all (and I mean all) return values from API and third-party functions you're calling. If it's a Windows API call or the function provides more information if it fails, are you calling GetLastError() or similar function?

If you're calling functions, and you're assuming they just work without checking return value, then that is poor coding, especially if that application will be run on systems you have no idea about. I have experience in having to fix code that just called API functions without checking to see if they failed, and having those calls fixed solved the problem (since now it was reported what function was failing). If you are not checking for failures, what happens is that your program takes the path of assuming everything is OK when it isn't. You then get the hangs, crashes, and weird behaviour you may be experiencing now on the customer's machine.

Maybe you won't fix the problem with the build you give them with these checks, but at the very least, you know pretty much what fails, and then you can do your own research from there without bothering the customer too much (usually, once you do the research and make a tentative fix, that fixes the problem at the customer end).

In addition, what information do you have on the system that's running? That is also information that is important. Do you log that information? Since you say something is up with the database, do you have smaller utility programs to check the viability of the database?

Regards,

Paul McKenzie

[Paul McKenzie]

I've thought about logging, but I'm not sure if that would be viable. What happens if the program crashes during logging? Won't that corrupt the log file?

Not if you

1) Open and close the log for each write

or

2) Write a crash dump (see DBGHELP.DLL).

Regards,

Paul McKenzie

[Igor Vartanov]

I've tried 2 different XP machines, 2 7 machines and even a Windows 8 machine. I've tired with 10 different databases. No luck.

The good news is that this particular customer is fairly tech savvy and agreed to test our program for us.

Not to discourage you, but this sort of things happens. Many dozens of installations all over the world, extensive logging for every critical component, automatic minidump generation, in-lab test and load runs and users' action log playback, endless code reviews, etc. One installation shows rare but quite regular crashes even after a number of version upgrades and hardware changes without any clue. Customer is very competent and cooperative, and endures the situation for several years. No luck.

So do your best with the all listed above and never lose your hope to find the root cause.

[Paul McKenzie]

Here's the real problem: I CANNOT reproduce this issue. The two customers were on Windows 7 and XP. I've tried 2 different XP machines, 2 7 machines and even a Windows 8 machine. I've tired with 10 different databases. No luck.

I think that might be the mistake you're making. You're trying to duplicate the issue on machines you know have been set up correctly or set up in a way where the problem can never be duplicated.

This is when you have to do what I stated and what Igor has stated. The issue is not that you must try your hardest to duplicate the problem (even though it is helpful), you have to do the code review, code analysis, making sure you're checking for all possible scenarios (i.e. return values), create a mini crash dump if the app crashes, get the database or whatever other third-party component information from the customer that your program relies on, etc.

One trick you can try is when you debug your program, by force you make a failure occur (database connect fails, return a NULL from a function, etc.) and see what happens. If the program hangs or does weird things, then you've discovered a hole in the program -- maybe not the same hole as the customer, but a hole nonetheless.

Regards,

Paul McKenzie

[GCDEF]

I don't see why a crash would corrupt a log file. At least you'd get some idea where it was and how far it got when it crashed.

[Access_Denied]

I think that might be the mistake you're making. You're trying to duplicate the issue on machines you know have been set up correctly or set up in a way where the problem can never be duplicated.

This is when you have to do what I stated and what Igor has stated. The issue is not that you must try your hardest to duplicate the problem (even though it is helpful), you have to do the code review, code analysis, making sure you're checking for all possible scenarios (i.e. return values), create a mini crash dump if the app crashes, get the database or whatever other third-party component information from the customer that your program relies on, etc.

One trick you can try is when you debug your program, by force you make a failure occur (database connect fails, return a NULL from a function, etc.) and see what happens. If the program hangs or does weird things, then you've discovered a hole in the program -- maybe not the same hole as the customer, but a hole nonetheless.

Regards,

Paul McKenzie

That's exactly what I've done. I used the debugger to set the path to the database to just an empty path before initializing the database object. I came up with the same situation as the customer. That's what led me to believe that the database object (or another critical object) is failing during initialization. However, that doesn't help me too much, as half of the object in the program are declared during the initialization of the program, so it really doesn't narrow it too much.

And finding the root of the problem will be difficult. I'm working with a very large code base, and to put it nicely, the code sucks. Return values aren't checked, exceptions are caught and suppressed, methods are 200+ lines long, etc. It's really just downright impossible to try to track it down without some sort of hint. That's why I'm hoping to get a log file from our customer. Narrowing it down to at least the object that crashed will help me tremendously.

I think I'm going to try to get some logging into the build. Hopefully I can find a nice library to help. It's also very possible that these customers have corrupted databases, which is why I'm waiting for them to send them to me. I'm not counting on that though, as history tells me that it would be too easy.

Thanks for the suggestions guys. Hopefully I can figure this out. :/

[Paul McKenzie]

so it really doesn't narrow it too much.

Why doesn't it help you much? You have duplicated a scenario, so then you write code to either state the error in a graceful way (no crash), or actually fix the problem. At this point, you should be working on coding so that the program doesn't crash or hang -- figuring out why the database doesn't connect is another issue.

And finding the root of the problem will be difficult. I'm working with a very large code base, and to put it nicely, the code sucks. Return values aren't checked, exceptions are caught and suppressed, methods are 200+ lines long, etc. It's really just downright impossible to try to track it down without some sort of hint.

Well that's what using a debugger is for, regardless of how messy or large the code base is. You are not trying to rewrite or add features to a program (in that case, a badly written, large set of code would be an issue) -- you're trying to fix a bug that you now have focus on.

Seriously, a good programmer can fix a bug in any program, even programs they've never seen at all, no matter how large the program is. As long as they can duplicate the bug and have the code, they can fix it, and you can chalk a lot of this up to using the debugger. Even if it takes you 10 or 20 debug sessions to get a handle of the track that the code takes when the DB cannot connect, then so be it.

It's also very possible that these customers have corrupted databases, which is why I'm waiting for them to send them to me. I'm not counting on that though, as history tells me that it would be too easy.

Well again, assume they don't send it to you. This is where the program must be able to handle whether the database is corrupt missing, cannot connect, whatever. I'm surprised this is lacking in the program to begin with.

Regards,

Paul McKenzie

[Access_Denied]

But I haven't really been able to reproduce the bug. I've been able to reproduce something that has the same effect as the bug, but there's about 1,000 things that produce that same effect. I may not be a programming guru, but I consider myself pretty good with the VC++ debugger. Unfortunately, without narrowing it down, there's just too many things it could be. There's about 5,000 lines of code that could be causing this issue (I think), so to try to fix it without narrowing it down would be to go through 5,000 lines and make sure every exception is handled and every API function is checked. It's quite the task.

[GCDEF]

But I haven't really been able to reproduce the bug. I've been able to reproduce something that has the same effect as the bug, but there's about 1,000 things that produce that same effect. I may not be a programming guru, but I consider myself pretty good with the VC++ debugger. Unfortunately, without narrowing it down, there's just too many things it could be. There's about 5,000 lines of code that could be causing this issue (I think), so to try to fix it without narrowing it down would be to go through 5,000 lines and make sure every exception is handled and every API function is checked. It's quite the task.

http://www.persistenceunlimited.com/...tivity-secret/

[S_M_A]

There are a lot of log support libraries out there. I've used ACE http://www.cs.wustl.edu/~schmidt/ACE.html but that might be an overkill here. At least as a quick solution now when you have a customer standing by to help you.

Just a thought, even if there are a lot of code would it be to much work to add OutputDebugString http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx calls in strategic places (or even in every function/method) and ask the customer to run DebugView http://technet.microsoft.com/en-us/s.../bb896647.aspx while running your app? That way you don't have to risk that the log file is thrashed.

[Arjay]

If you want to get a crappy app working so that your customers have a good user experience, you are just going to have to go through the app and start checking the return codes and add logging support.

Start out by doing this to the most problem areas that you can think of (i.e the areas in the app that are most likely to fail).

Then expand these areas into other areas until you get operations for the whole app logged.

If this seems expensive to do (and takes a lot of your time), explain to your bosses that is probably would take more time to try to triage the customer complaints and repro their problems, rather than just have a customer send you a log file.

If you take the check return value/logging approach, you'll probably improve the app to the point, where customer failures practically never happen (and then you are at the point where you can congratulate yourself on writing quality software).

Lastly, if you are worried about logfile corruption, simply open and close the file each time you write to it. Not a problem if you do it this way. Or another approach is to simply use an existing logging library - there are many free ones to choose from.

[John E]

And finding the root of the problem will be difficult. I'm working with a very large code base, and to put it nicely, the code sucks. Return values aren't checked, exceptions are caught and suppressed, methods are 200+ lines long, etc. It's really just downright impossible to try to track it down without some sort of hint.

Hello Access_Denied. Are you aware that Visual C++ supports remote debugging? You need to generate a Release build with debugging information and it's a bit tricky to set up but I believe it does work. I've used it myself on many occasions albeit only across a local network. Only a few weeks ago I used it to track down a bug that was only happening when our app was run on Windows XP. Since my development system runs Win7 I wasn't able to reproduce the bug. So we set up an XP box on our network and that's how I debugged it. You get all the usual debugging features such as the ability to set break points and single-step through the code etc. At the absolute worst, maybe your customer could send you the machine that crashes and you could hook it up to your local network?

Hopefully your boss has learned a salutary lesson though.... As far as possible, EVERY program should check for mission critical errors and report them somehow. It's always a mistake to assume that c'tors will always succeed or that memory allocations will never fail etc.

And there's rarely any good excuse for catching an exception then failing to report it.!