How do you test your c++ code?

[Russco]

I am about to embark on a fair size project. Up until now I write classes/functions with testing functions as part of the same code controlled by #defines and ifdef/ifndef/endif blocks. This works ok for everything I've done so far but all projects have been fairly small and I'm not overly happy with the obfuscation to code readability that this method leads to.
So for you programmers who work mainly on larger projects, what is your testing regime. How do you test your classes and functions and what are the pros and cons of each method?
Do you make a separate project in the same solution?
Do you intergrate your tests within the main project?
Do you use some sort of test framework?
Do your tests cause obfuscation to code readability?

What is basically the best habit to get into for writing tested c++ code?

PS. Any suggested 3rd party frameworks etc. would have to be freely available for me to be interested in them.

[TheCPUWizard]

I divide testing into (al least) two distinct arenas. BlackBox testing which only has access to the exposed functionallity defined by the specification, and WhiteBox testing which handles implementation specific details.

In an ideal world, if a program provides correct observable behaviour and state for all possible inputs, then it is in and of itself correct. (See signature for theory/practice). Unfortunately it is usually not possible/practical to provide all of the necessary environments to fully test in this manner.

Unfortunately [IMPO] too many people immediately jump to WhiteBox and even the addition of "testing methods". In many cases this can be avoided by a better class design. Consider the following code:

class Foo:
{
    private int SomeWork(int j, int k) {...}
}

Depending onhow thid method is called from within Foo, it can be impossible to check behaviour for all (significant) combinations of j,k.

Now look at the following:

class Worker
{
  public static int SomeWork(int j, int k) {...}
}

class Foo:
{
    private int SomeWork(int j, int k) {return Worker.SomeWork(j,k);}
}

The computational ligic is now 100% testable, and all that needs to be verified is the proper passing of the three values.

In most projects I have worked on, this can be used to increase the explicit test coverage to about the 97% level. The remaining 3% many well be suited for "examination" type testing for a variety of uses, but there will be situations where 100% testing is really required.

At this point, it will become necessary to introduce test methods into the objects themselves. This can be done with a single #include to minimize the polution:

// Foo_Tests.inh
public:
  int TestSomething();

// Foo.h
class Foo:
{
    private int SomeWork(int j, int k) {return Worker.SomeWork(j,k);}
    #include "Foo_Tests.inh"
}

I do use a number of commercial frameworks (especially for client projects), but actually prefer a proprietary framework I developed starting back in the 1990's.

[GCDEF]

I run everything I write through the debugger and watch every line execute to be sure it's doing what I think it's doing. After that, we have a test group that does black box testing.

As to all the #ifdef stuff, I would bend over backwards to avoid it and prohibit anybody that works for me from using it. Code should execute the same way each time. Having behavior depend on switches evaluated at compile time is asking for trouble as it's way too easy to have one set in the wrong state and it makes the code harder to read.

[TheCPUWizard]

As to all the #ifdef stuff, I would bend over backwards to avoid it and prohibit anybody that works for me from using it. Code should execute the same way each time. Having behavior depend on switches evaluated at compile time is asking for trouble as it's way too easy to have one set in the wrong state and it makes the code harder to read.

(Although I stop one step shy of prohibition, and simply make it a process requiring documentation and approval...)

ANYTHING that changes the code, in any way, invalidates at least a portion of the tests. Consider:

Your code has a buffer overrun of 4 bytes. The buffer is normally the last member in the memory layout. During testing you add contidionally add a member that is specific for testing. The overrun will corrupt the "Test" member, and probably be undetected.

However as soon ars you re-compile with that member conditionalized out, the overrun goes outside the bounds of your object, posbbily leading to crashes, or undefined behaviour of another object that happens to be next to it....

[Russco]

I run everything I write through the debugger and watch every line execute to be sure it's doing what I think it's doing. After that, we have a test group that does black box testing.

As to all the #ifdef stuff, I would bend over backwards to avoid it and prohibit anybody that works for me from using it. Code should execute the same way each time. Having behavior depend on switches evaluated at compile time is asking for trouble as it's way too easy to have one set in the wrong state and it makes the code harder to read.

I tend to watch my variables closely too while single stepping code. I think we all do, but is that enough?

And I totally agree on the conditional compilation. It was the major impetus for this thread.

Generally my virtuals are always private. I use a non virtual invarient in the base class in most cases which calls the private virtual, so this would give me a good hook-point for inserting test code for virtual functions.
Should I maybe do the same with the invarients or value types which almost never have virtuals? Have a public function call a private non virtual that does the real work. Does this really buy me anything?

What do you guys think of the unit testing frameworks that are freely available?

My understanding is these can help greatly but I've never used one. Are they easy to use and understand? I dont like to use code I cant understand myself. Which would you recommend?

[TheCPUWizard]

Generally my virtuals are always private.

Private virtuals make absolutely NO sense.

Using non-virtual re-directors is also a distinct ANTI-pattern, and is usually frowned upon (or even explicitly prohibited.

[GCDEF]

I've never worked with third party testing apps, but I'm skeptical. I'm responsible for a very large suite of apps. No idea how many lines of code we have other than I'm sure it's measured in millions. The type of bugs we run into aren't usually failures in individual functions. I catch those when I'm testing in the debugger.

The biggest source of bugs is side effects. You change the behavior somewhere and it has an unintended consequence somewhere else. The bigger the project gets and the more potential code paths and switches and states you add, the more likely it is that you'll introduce unintended consequences. I don't really see a viable alternative to black box style regression testing on a somewhat regular basis which is what we do.

[TheCPUWizard]

I've never worked with third party testing apps, but I'm skeptical. I'm responsible for a very large suite of apps. No idea how many lines of code we have other than I'm sure it's measured in millions. The type of bugs we run into aren't usually failures in individual functions. .

The testing frameworks have little to do with the testing logic itself. The primary purpose is to provide an infrastructure for testing. This typically includes:

1) Helpers to compare results and log pass/fail along with supporting information.

2) Historical analysis of testing (e.g. "Test YadaYada started failing in Build #2123 and was fixed in Build #126")

3) GUI interfaces for selecting the tests.

4) Instrumentation helpers to track the exact lines of code and/or variables as the test was run.

5) Automation of all of the above (except GUI) for integration with build processs

[Russco]

Private virtuals make absolutely NO sense.

Using non-virtual re-directors is also a distinct ANTI-pattern, and is usually frowned upon (or even explicitly prohibited.

I think you may find http://www.gotw.ca/publications/mill18.htm interesting reading.

[JohnW@Wessex]

A lot of the more complex code in our applications are to do with image analysis. There are often no real fixed pass/fail tests as such. What we do is run about 70000 images though the algorithms in an automated run and check which areas of detection have been improved and (often the case) which ones have got worse. The decision on whether to keep the 'improvement' is based on the relative value of the gains over the losses.
When the algorithm is being developed, it is often tested by using logged output data to annotate the image to show where and how the algorithm was applied. Artificially generated and simplified images help in development in allowing output data to be more predictable.

[TheCPUWizard]

I think you may find http://www.gotw.ca/publications/mill18.htm interesting reading.

I am familiar with Herb Sutters point of view on the matter and have been for many years.

A big part of the difference is the focus on design vs. implementation. When using the Template Method pattern, one is stating that the conceptual behavour is inherent in the base class and does NOT vary based on derived classes. When using a public virtual mathod, there is the explicit statement that the behaviour is controlled by the derived class.

Using the sample, the code states that Widget and all derived classes will perform the same function to Process when the Template Method pattern is used. The fact that the implementation of different phases is controllable by derived classes is not implicit.

[laserlight]

A big part of the difference is the focus on design vs. implementation. When using the Template Method pattern, one is stating that the conceptual behavour is inherent in the base class and does NOT vary based on derived classes. When using a public virtual mathod, there is the explicit statement that the behaviour is controlled by the derived class.

Using the sample, the code states that Widget and all derived classes will perform the same function to Process when the Template Method pattern is used. The fact that the implementation of different phases is controllable by derived classes is not implicit.

How does that square with your statement that "Private virtuals make absolutely NO sense"?

[JohnW@Wessex]

Here's a comment about private virtuals. I don't whether there's any other downsides.

[TheCPUWizard]

How does that square with your statement that "Private virtuals make absolutely NO sense"?

It would have made better sense if I had actually posted the entire thought...

Private virtuals make absolutely NO sense when talking about design and object models, their usage is an implementation detail.

As originally posted, the statement is obviously false.

[Russco]

I will make a new thread soon to discuss the issue of accessibility of virtuals. Please keep the thread on topic, the non-publicness of virtuals I think deserves its own discussion.