Case study of a really hard to find bug

[Zaccheus]

One type of bug I really hate is the one which only happens when a debug build is run in non-debug-mode.

I've just now solved one of these, and boy what a nightmare it was.

As already mentioned, it only happened when I launched the debug build using "Start Without Debugging". When the application started, I would get the message box shown in the attachment - not much of a clue.

So I started using my logging class to find out how far my application gets before it hits the assertion. It never wrote a single entry! Eventually I ended up at WinMain and still the assertion happened.

Cutting a long story short, here is a simplified version of what WinMain looked like now:

int WINAPI WinMain
	(
	HINSTANCE     inInstance,
	HINSTANCE,
	LPSTR,
	int           inCmdShow
	)
{
	Rcl::Logger::report("Reached!", __FILE__,__LINE__);

	RCL_DEBUGOUT_INIT;

        //...

	return 0;
}

Then I thought "hmm, fwrite and logging" and when I removed everything from WinMain the assertion disappeard! It turned out that the logging attempt was now actually causing the assertions!!!

So I started using message boxes and managed to track down the problem.

The original culprit seems to have been RCL_DEBUGOUT_INIT. It is part of a bunch of RCL_DEBUGOUT macros which use classes which call OutputDebugString as shown in the following code:

#ifdef RCL_USEDEBUG

// ...

if(::IsDebuggerPresent())
{
    ::OutputDebugString(txt.c_str());
}
else
{
    // This code was causing an assertion when an application 
    // was run as a debug build but not in debug mode.
    
    std::vector<uint8_t> buffer(Rcl::StringUtils::sizeUTF8(txt));
    RCL_VERIFY( Rcl::StringUtils::toUTF8(txt, &buffer[0], buffer.size()) );

    FILE* file = std::fopen("C:\\RclsoftwareOrgUk\\Logging\\Debugout.txt", "a+b");
    std::fwrite(&buffer[0], sizeof(uint8_t), buffer.size(), file);
    std::fclose(file);
}

// ...

#endif

As you can see, if I have a debug build but there is no debugger present, then I try to create a file in a folder which may not (and did not) exist. The bug here is that I do not check whether 'file' is NULL.

My logging class also streams all its output to the RCL_DEBUGOUT macros, itself causing the assertion!

[Paul McKenzie]

As you can see, if I have a debug build but there is no debugger present, then I try to create a file in a folder which may not (and did not) exist. The bug here is that I do not check whether 'file' is NULL.

In general, you must check return values from all functions that return potential errors. You cannot take any function for granted, no matter how sure you think the function will succeed.

I had to debug a Windows program that used the GDI, and very few places checked to see if functions such as StretchBlt, BitBlt, etc. actually were successful. What ended up happening is that certain printer drivers would produce totally black pages while others didn't and produced good pages. The reason is that the GDI functions were returning failure for the black pages, but they were never checked and therefore assumed to be working.

That's why I tell posters to make sure they are checking all functions for failure if they have a bug, and not assume that any function will work. This is especially the case if you are creating software that will be used by the general public. In that case, you have no choice but to check for errors in every function or at least set up a scenario where if there is an error, you handle it correctly.

Regards,

Paul McKenzie

[Zaccheus]

In general, you must check return values from all functions that return potential errors.

Yes, of course you have to. I always do.

I use macros like this all the time:

#define RCL_COM_OK(F_) {HRESULT r_=(F_);if(FAILED(r_)){RCL_THROW_1(RclsoftwareOrgUk::XCom,r_);}}

RCL_COM_OK( mbDirect3DDevice->SetRenderState(D3DRS_LIGHTING, TRUE) );

The bug shown in the first post is ancient code which was made a decade ago. I'm really quite shocked that I ever wrote that, including the hard coded pathname, even if it was only included in debug builds. The whole 'else' block has now been removed.

[D_Drmmr]

As you can see, if I have a debug build but there is no debugger present, then I try to create a file in a folder which may not (and did not) exist. The bug here is that I do not check whether 'file' is NULL.

There is a small utility program to display the debug output of all running applications here.
If you use this, you can remove the if/else from your code entirely.