How do you test your c++ code?

I am about to embark on a fair size project. Up until now I write classes/functions with testing functions as part of the same code controlled by #defines and ifdef/ifndef/endif blocks. This works ok for everything I've done so far but all projects have been fairly small and I'm not overly happy with the obfuscation to code readability that this method leads to.
So for you programmers who work mainly on larger projects, what is your testing regime. How do you test your classes and functions and what are the pros and cons of each method?
Do you make a separate project in the same solution?
Do you intergrate your tests within the main project?
Do you use some sort of test framework?
Do your tests cause obfuscation to code readability?

What is basically the best habit to get into for writing tested c++ code?

PS. Any suggested 3rd party frameworks etc. would have to be freely available for me to be interested in them.

I divide testing into (al least) two distinct arenas. BlackBox testing which only has access to the exposed functionallity defined by the specification, and WhiteBox testing which handles implementation specific details.

In an ideal world, if a program provides correct observable behaviour and state for all possible inputs, then it is in and of itself correct. (See signature for theory/practice). Unfortunately it is usually not possible/practical to provide all of the necessary environments to fully test in this manner.

Unfortunately [IMPO] too many people immediately jump to WhiteBox and even the addition of "testing methods". In many cases this can be avoided by a better class design. Consider the following code:

Code:

---

class Foo:
{
    private int SomeWork(int j, int k) {...}
}

---

Depending onhow thid method is called from within Foo, it can be impossible to check behaviour for all (significant) combinations of j,k.

Now look at the following:

Code:

---

class Worker
{
  public static int SomeWork(int j, int k) {...}
}

class Foo:
{
    private int SomeWork(int j, int k) {return Worker.SomeWork(j,k);}
}

---

The computational ligic is now 100% testable, and all that needs to be verified is the proper passing of the three values.

In most projects I have worked on, this can be used to increase the explicit test coverage to about the 97% level. The remaining 3% many well be suited for "examination" type testing for a variety of uses, but there will be situations where 100% testing is really required.

At this point, it will become necessary to introduce test methods into the objects themselves. This can be done with a single #include to minimize the polution:

Code:

---

// Foo_Tests.inh
public:
  int TestSomething();

// Foo.h
class Foo:
{
    private int SomeWork(int j, int k) {return Worker.SomeWork(j,k);}
    #include "Foo_Tests.inh"
}

---

I do use a number of commercial frameworks (especially for client projects), but actually prefer a proprietary framework I developed starting back in the 1990's.

I run everything I write through the debugger and watch every line execute to be sure it's doing what I think it's doing. After that, we have a test group that does black box testing.

As to all the #ifdef stuff, I would bend over backwards to avoid it and prohibit anybody that works for me from using it. Code should execute the same way each time. Having behavior depend on switches evaluated at compile time is asking for trouble as it's way too easy to have one set in the wrong state and it makes the code harder to read.

Quote:

---

Originally Posted by GCDEF

As to all the #ifdef stuff, I would bend over backwards to avoid it and prohibit anybody that works for me from using it. Code should execute the same way each time. Having behavior depend on switches evaluated at compile time is asking for trouble as it's way too easy to have one set in the wrong state and it makes the code harder to read.

---

:thumb::thumb::thumb:(Although I stop one step shy of prohibition, and simply make it a process requiring documentation and approval...)

ANYTHING that changes the code, in any way, invalidates at least a portion of the tests. Consider:

Your code has a buffer overrun of 4 bytes. The buffer is normally the last member in the memory layout. During testing you add contidionally add a member that is specific for testing. The overrun will corrupt the "Test" member, and probably be undetected.

However as soon ars you re-compile with that member conditionalized out, the overrun goes outside the bounds of your object, posbbily leading to crashes, or undefined behaviour of another object that happens to be next to it.... :eek::eek:

Quote:

---

Originally Posted by GCDEF

I run everything I write through the debugger and watch every line execute to be sure it's doing what I think it's doing. After that, we have a test group that does black box testing.

As to all the #ifdef stuff, I would bend over backwards to avoid it and prohibit anybody that works for me from using it. Code should execute the same way each time. Having behavior depend on switches evaluated at compile time is asking for trouble as it's way too easy to have one set in the wrong state and it makes the code harder to read.

---

I tend to watch my variables closely too while single stepping code. I think we all do, but is that enough?

And I totally agree on the conditional compilation. It was the major impetus for this thread.

Generally my virtuals are always private. I use a non virtual invarient in the base class in most cases which calls the private virtual, so this would give me a good hook-point for inserting test code for virtual functions.
Should I maybe do the same with the invarients or value types which almost never have virtuals? Have a public function call a private non virtual that does the real work. Does this really buy me anything?

What do you guys think of the unit testing frameworks that are freely available?

My understanding is these can help greatly but I've never used one. Are they easy to use and understand? I dont like to use code I cant understand myself. Which would you recommend?

Quote:

---

Originally Posted by Russco

Generally my virtuals are always private.

---

Private virtuals make absolutely NO sense.

Using non-virtual re-directors is also a distinct ANTI-pattern, and is usually frowned upon (or even explicitly prohibited.

I've never worked with third party testing apps, but I'm skeptical. I'm responsible for a very large suite of apps. No idea how many lines of code we have other than I'm sure it's measured in millions. The type of bugs we run into aren't usually failures in individual functions. I catch those when I'm testing in the debugger.

The biggest source of bugs is side effects. You change the behavior somewhere and it has an unintended consequence somewhere else. The bigger the project gets and the more potential code paths and switches and states you add, the more likely it is that you'll introduce unintended consequences. I don't really see a viable alternative to black box style regression testing on a somewhat regular basis which is what we do.

Quote:

---

Originally Posted by GCDEF

I've never worked with third party testing apps, but I'm skeptical. I'm responsible for a very large suite of apps. No idea how many lines of code we have other than I'm sure it's measured in millions. The type of bugs we run into aren't usually failures in individual functions. .

---

The testing frameworks have little to do with the testing logic itself. The primary purpose is to provide an infrastructure for testing. This typically includes:

1) Helpers to compare results and log pass/fail along with supporting information.

2) Historical analysis of testing (e.g. "Test YadaYada started failing in Build #2123 and was fixed in Build #126")

3) GUI interfaces for selecting the tests.

4) Instrumentation helpers to track the exact lines of code and/or variables as the test was run.

5) Automation of all of the above (except GUI) for integration with build processs

Quote:

---

Originally Posted by TheCPUWizard

Private virtuals make absolutely NO sense.

Using non-virtual re-directors is also a distinct ANTI-pattern, and is usually frowned upon (or even explicitly prohibited.

---

I think you may find http://www.gotw.ca/publications/mill18.htm interesting reading.

A lot of the more complex code in our applications are to do with image analysis. There are often no real fixed pass/fail tests as such. What we do is run about 70000 images though the algorithms in an automated run and check which areas of detection have been improved and (often the case) which ones have got worse. The decision on whether to keep the 'improvement' is based on the relative value of the gains over the losses.
When the algorithm is being developed, it is often tested by using logged output data to annotate the image to show where and how the algorithm was applied. Artificially generated and simplified images help in development in allowing output data to be more predictable.

Quote:

---

Originally Posted by Russco

I think you may find http://www.gotw.ca/publications/mill18.htm interesting reading.

---

I am familiar with Herb Sutters point of view on the matter and have been for many years.

A big part of the difference is the focus on design vs. implementation. When using the Template Method pattern, one is stating that the conceptual behavour is inherent in the base class and does NOT vary based on derived classes. When using a public virtual mathod, there is the explicit statement that the behaviour is controlled by the derived class.

Using the sample, the code states that Widget and all derived classes will perform the same function to Process when the Template Method pattern is used. The fact that the implementation of different phases is controllable by derived classes is not implicit.

Quote:

---

Originally Posted by TheCPUWizard

A big part of the difference is the focus on design vs. implementation. When using the Template Method pattern, one is stating that the conceptual behavour is inherent in the base class and does NOT vary based on derived classes. When using a public virtual mathod, there is the explicit statement that the behaviour is controlled by the derived class.

Using the sample, the code states that Widget and all derived classes will perform the same function to Process when the Template Method pattern is used. The fact that the implementation of different phases is controllable by derived classes is not implicit.

---

How does that square with your statement that "Private virtuals make absolutely NO sense"?

Here's a comment about private virtuals. I don't whether there's any other downsides.

Quote:

---

Originally Posted by laserlight

How does that square with your statement that "Private virtuals make absolutely NO sense"?

---

It would have made better sense if I had actually posted the entire thought... :blush::blush:

Private virtuals make absolutely NO sense when talking about design and object models, their usage is an implementation detail.

As originally posted, the statement is obviously false.

I will make a new thread soon to discuss the issue of accessibility of virtuals. Please keep the thread on topic, the non-publicness of virtuals I think deserves its own discussion.

Quote:

---

Originally Posted by JohnW@Wessex

Here's a comment about private virtuals. I don't whether there's any other downsides.

---

IMHO, the very URL is sufficient: http://www.parashift.com/c++-faq-lite/strange-inheritance.html#faq-23.4

Quote:

---

Almost never.

Protected virtuals are okay, but private virtuals are usually a net loss. Reason: private virtuals confuse new C++ programmers, and confusion increases cost, delays schedule, and degrades risk.

New C++ programmers get confused by private virtuals because they think a private virtual cannot be overridden. After all, a derived class cannot access members that are private in its base class so how, they ask, could it override a private virtual from its base class? There are explanations for the above, but that's academic. The real issue is that almost everyone gets confused the first time they run into private virtuals, and confusion is bad.

Unless there is a compelling reason to the contrary, avoid private virtuals.

---

You can try cppunit - it's a good framework to easily keep adding test cases to suites. Try to keep test code seperate from the actual code in order to not make them too coupled. Then there is boost::test as well. Haven't tried it but it should be good.

For user interface tests you can try Gui Automated Tests (from osec). I've started to use it in my current project and I've found it is very handy.

I support for cppunit and QUnit.

I'm planning to give UnitTest++ a try out at some point.

Hundreds and hundreds of ASSERT calls which write to a debug log. When something crashes, I have a log of exactly what went wrong. The best part is that when I compiler a release version, all of that debugging code is removed so there is no slowdown or bloatware. :D

Unfortunately, our applications can usually only be debugged at full speed (release build) as there are many time critical features. Therefore all debugging aids must have minimal impact on the running of the application.

You mean you're looking for thread race situations? Yeah, those are a pain in the *** to debug.

Quote:

---

Unfortunately, our applications can usually only be debugged at full speed (release build) as there are many time critical features. Therefore all debugging aids must have minimal impact on the running of the application.

---

I do a good amount of industrial automation, and "true" real-time systems (requiring response times measured in the sub-millisecond range), yet 80% or more of you code testing is done in conditions that are NOT constrained by time.

Perhaps even more important is that our bug rate for problems which can ONLY be found by "running at speed" is less than 2% of the total issues we discover.....

John, Do you have any metrics on the number/percentage of bugs/defects that you have actually found in your code over a period of time (say one year) which could NOT have been discovered by any other means?

No, talking to external hardware, though we do use multiple threads.

It's difficult to say offhand what the percentages are, but what you say sounds about right.
I've been debugging the messaging protocol with customer the last couple of weeks and tracking down a couple of timing errors, so real-time debugging has been important lately and I probably overstated its importance in my post.
Real-time debugging is important to us, but we can do a fair amount offline.

Quote:

---

Originally Posted by JohnW@Wessex

It's difficult to say offhand what the percentages are, but what you say sounds about right.
I've been debugging the messaging protocol with customer the last couple of weeks and tracking down a couple of timing errors, so real-time debugging has been important lately and I probably overstated its importance in my post.
Real-time debugging is important to us, but we can do a fair amount offline.

---

That makes alot more sense :wave::wave:

You mentioned "talking to hardware" in a previous post. I have long (since the 1970's ) been a big fan of using "simulators/emulators" where ever possible. One major client of mine [1996-2002] did not think the investment (estimated at $30K) was worth it. About 2 months later a pair of software bugs (that had been in the code for quite a while) surfaced in a slightly different scenario. The first time it blew a $60K turbine, the second time it nearly caused a gas explosion. Both had potential for serious damage and injury (death).

Quote:

---

Originally Posted by TheCPUWizard

I have long (since the 1970's ) been a big fan of using "simulators/emulators" where ever possible.

---

That's exactly what we use for our soak testing and the first call when 'real time' debugging. Going to our customer and taking one of their machines out of action to debug is usually a non-starter, so we try to do as much testing with our simulation hardware/software as possible. It also looks better when you can turn up on the day with something that's 99% correct, even when it's at the earlier stages of development. Actually, most of my site visits are a bit boring as there's usually not a lot for me to do. :rolleyes:

Quote:

---

The first time it blew a $60K turbine, the second time it nearly caused a gas explosion.

---

Scary! Thankfully bugs and timing glitches in our code only result in a mail item landing in the wrong box. A small chance of a paper cut maybe. :eek:

Quote:

---

Originally Posted by JohnW@Wessex

Scary! Thankfully bugs and timing glitches in our code only result in a mail item landing in the wrong box. A small chance of a paper cut maybe. :eek:

---

No kiddiny, after the turbo blew (it shattered the housing - which sould not have happened), they were pulling shrapnel out of the walls for weeks..

Back (at least closer) to the topic.

The issue of which tools to use to test code is a serious one, but the selection usually depends on the persons needs and goals. I have found it very difficult to qualify one as "better" than the others across the possibilities of situations. Fortunately many of them are free, and a few hours is typically all it takes to make a personal evaluation.

This does not really address the question of "HOW" do you test your code. WhiteBox, BlackBox, GreyBox are completely different approaches. There are also domain bounded test methods (done test numbers like 10^20 for single item prices in a shopping cart application) and full perumation test methods (which can often run for a very very long time).

Since much of my early experience was in demanding applications (Military, Space, etc) where a defect was simply too cost prohibitive to consider, comprehensive (whatever that really means) testing was the norm.

When people who do not come from a high testing background look at serious test methodologies, they are often concerned with the cost ($$$ and time is money) involved. Many reject it out of hand without even making in depth studies.

OVer the past 25 years (since the founding of my company) we have often evaluated places where we could "scale back" on some of our testing from the (sometime extreme) methodologies that served us well in the past. Because we are alread invested in the methodologies and have large suites of test application, the "ramp up" costs are mainly ancient history, and we have to actually consider the costs of creating "lighter" test platforms. Our repeated experience [YMMV] has been that there is NO payoff to decreasing the level of testing.

The conclusion (from my point of view) is that a driving factor is if you are in this for the "long haul" or is software development a transient or secondary activity. If the former, then any investment in testing will payoff; if the latter then the risks of releasing buggy code may be more tolerable, and indepth testing may not be a financial win.

There is a quote from one of Richard P. Feynman's books where he talks about his involvement in the Challenger disaster inquiry. He was very impressed with the rigour that the software team applied to testing the flight control software. They used the quite sensible method of building the code up from small and testable components, right up to the full simulation.

Apparently a manager from NASA had wanted to try to save money by cutting back on the testing and had said to the team. "Why bother with so many tests? You always pass them anyway!". :rolleyes:

Quote:

---

There is a quote from one of Richard P. Feynman's books where he talks about his involvement in the Challenger disaster inquiry.

---

Do you have the article ?

No article, but here's the book

What_Do_You_Care_What_Other_People_Think

I write small modules which can be tested easily with little overhead.
When dealing with small chunks of code, most errors are caught by eye
before they are ever even ran in the simulation and debugger.
While coding my game engine over the last four years i have written
and debugged each and every chunk in this manner.
When a module is finished i put it aside and create the next part.
I have ran into a few quirks along the way but nothing serious.
And if i decide to do something different with a chunk in the future,
it can be easily swapped out with the new chunk of code.
I am not a proffesional programmer with a deadline so i have
plenty of time so sit around and ponder my ideas.
Being in a hurry causes us to make bad decisions, thus introducing bugs.
I think the time spent pondering is time well spent in the long run.

Quote:

---

Originally Posted by bitshifter420

Being in a hurry causes us to make bad decisions, thus introducing bugs.
I think the time spent pondering is time well spent in the long run.

---

I'd go along with that. I've met too many programmers where code appears to be approached with a 'throw-away quick fix' mentality.

If you start work as a professional programmer, pin that above your desk, and probably your managers too. :D

I plan to never take on programming as a profession mainly beacuse
i believe if you do something for long enough you will learn to hate it.
(programming is fun, and i plan to keep it that way forever)
For the last 10 years i have been turning wrenches at same motorcycle dealership
and never once has my boss told me to hurry up this job, to go faster, to get 'er done.
Since allowing me to work at my own pace and in my own mindset
i have not had a machine comeback to the shop for erronous work.
He sees this and we have a mutual unverbalized agreement as to how it works.
He understands that it takes this long to do job, and twice as long if mistakes are made.
And in this buisness, time is truly money ($100.00 an hour shop rate)

Quote:

---

Originally Posted by bitshifter420

I plan to never take on programming as a profession mainly beacuse
i believe if you do something for long enough you will learn to hate it.

---

Does that mean you'll be giving up the 'turning wrenches' soon?

I've been programming in various fields for just over 20 years and I still really enjoy it. As long as you keep learning and taking on new challenges , it stays fresh.

I think the learning curve has a lot to do with it.
Once you stop learning its time to move on.
I have been a wrench monkey for 25 years and its getting old.
Most technology advancement here is in the 'black box'.

Quote:

---

Originally Posted by bitshifter420

Most technology advancement here is in the 'black box'.

---

Yes. Gone are the days when you set the points with a cigarette paper and balanced the carbs with a pair of lollipop sticks. :cry:

Quote:

---

Originally Posted by bitshifter420

I plan to never take on programming as a profession mainly beacusei believe if you do something for long enough you will learn to hate it.

---

* Started programming (more than "playing" with) computers in 1972
* First paid programing work in 1975
* First fulltime programming job in 1977

Now (almost 37 years later) I still love it...... :D :D

Quote:

---

I've been programming in various fields for just over 20 years and I still really enjoy it. As long as you keep learning and taking on new challenges , it stays fresh.

---

I agree.