How do you test your c++ code?

[TheCPUWizard]

Here's a comment about private virtuals. I don't whether there's any other downsides.

IMHO, the very URL is sufficient: http://www.parashift.com/c++-faq-lite/strange-inheritance.html#faq-23.4

Almost never.

Protected virtuals are okay, but private virtuals are usually a net loss. Reason: private virtuals confuse new C++ programmers, and confusion increases cost, delays schedule, and degrades risk.

New C++ programmers get confused by private virtuals because they think a private virtual cannot be overridden. After all, a derived class cannot access members that are private in its base class so how, they ask, could it override a private virtual from its base class? There are explanations for the above, but that's academic. The real issue is that almost everyone gets confused the first time they run into private virtuals, and confusion is bad.

Unless there is a compelling reason to the contrary, avoid private virtuals.

[exterminator]

You can try cppunit - it's a good framework to easily keep adding test cases to suites. Try to keep test code seperate from the actual code in order to not make them too coupled. Then there is boost::test as well. Haven't tried it but it should be good.

[cromul]

For user interface tests you can try Gui Automated Tests (from osec). I've started to use it in my current project and I've found it is very handy.

[Peter_APIIT]

I support for cppunit and QUnit.

[JohnW@Wessex]

I'm planning to give UnitTest++ a try out at some point.

[ninja9578]

Hundreds and hundreds of ASSERT calls which write to a debug log. When something crashes, I have a log of exactly what went wrong. The best part is that when I compiler a release version, all of that debugging code is removed so there is no slowdown or bloatware.

[JohnW@Wessex]

Unfortunately, our applications can usually only be debugged at full speed (release build) as there are many time critical features. Therefore all debugging aids must have minimal impact on the running of the application.

[ninja9578]

You mean you're looking for thread race situations? Yeah, those are a pain in the *** to debug.

[TheCPUWizard]

Unfortunately, our applications can usually only be debugged at full speed (release build) as there are many time critical features. Therefore all debugging aids must have minimal impact on the running of the application.

I do a good amount of industrial automation, and "true" real-time systems (requiring response times measured in the sub-millisecond range), yet 80% or more of you code testing is done in conditions that are NOT constrained by time.

Perhaps even more important is that our bug rate for problems which can ONLY be found by "running at speed" is less than 2% of the total issues we discover.....

John, Do you have any metrics on the number/percentage of bugs/defects that you have actually found in your code over a period of time (say one year) which could NOT have been discovered by any other means?

[JohnW@Wessex]

No, talking to external hardware, though we do use multiple threads.

[JohnW@Wessex]

It's difficult to say offhand what the percentages are, but what you say sounds about right.
I've been debugging the messaging protocol with customer the last couple of weeks and tracking down a couple of timing errors, so real-time debugging has been important lately and I probably overstated its importance in my post.
Real-time debugging is important to us, but we can do a fair amount offline.

[TheCPUWizard]

It's difficult to say offhand what the percentages are, but what you say sounds about right.
I've been debugging the messaging protocol with customer the last couple of weeks and tracking down a couple of timing errors, so real-time debugging has been important lately and I probably overstated its importance in my post.
Real-time debugging is important to us, but we can do a fair amount offline.

That makes alot more sense

You mentioned "talking to hardware" in a previous post. I have long (since the 1970's ) been a big fan of using "simulators/emulators" where ever possible. One major client of mine [1996-2002] did not think the investment (estimated at $30K) was worth it. About 2 months later a pair of software bugs (that had been in the code for quite a while) surfaced in a slightly different scenario. The first time it blew a $60K turbine, the second time it nearly caused a gas explosion. Both had potential for serious damage and injury (death).

[JohnW@Wessex]

I have long (since the 1970's ) been a big fan of using "simulators/emulators" where ever possible.

That's exactly what we use for our soak testing and the first call when 'real time' debugging. Going to our customer and taking one of their machines out of action to debug is usually a non-starter, so we try to do as much testing with our simulation hardware/software as possible. It also looks better when you can turn up on the day with something that's 99% correct, even when it's at the earlier stages of development. Actually, most of my site visits are a bit boring as there's usually not a lot for me to do.

[JohnW@Wessex]

The first time it blew a $60K turbine, the second time it nearly caused a gas explosion.

Scary! Thankfully bugs and timing glitches in our code only result in a mail item landing in the wrong box. A small chance of a paper cut maybe.

[TheCPUWizard]

Scary! Thankfully bugs and timing glitches in our code only result in a mail item landing in the wrong box. A small chance of a paper cut maybe.

No kiddiny, after the turbo blew (it shattered the housing - which sould not have happened), they were pulling shrapnel out of the walls for weeks..

Back (at least closer) to the topic.

The issue of which tools to use to test code is a serious one, but the selection usually depends on the persons needs and goals. I have found it very difficult to qualify one as "better" than the others across the possibilities of situations. Fortunately many of them are free, and a few hours is typically all it takes to make a personal evaluation.

This does not really address the question of "HOW" do you test your code. WhiteBox, BlackBox, GreyBox are completely different approaches. There are also domain bounded test methods (done test numbers like 10^20 for single item prices in a shopping cart application) and full perumation test methods (which can often run for a very very long time).

Since much of my early experience was in demanding applications (Military, Space, etc) where a defect was simply too cost prohibitive to consider, comprehensive (whatever that really means) testing was the norm.

When people who do not come from a high testing background look at serious test methodologies, they are often concerned with the cost ($$$ and time is money) involved. Many reject it out of hand without even making in depth studies.

OVer the past 25 years (since the founding of my company) we have often evaluated places where we could "scale back" on some of our testing from the (sometime extreme) methodologies that served us well in the past. Because we are alread invested in the methodologies and have large suites of test application, the "ramp up" costs are mainly ancient history, and we have to actually consider the costs of creating "lighter" test platforms. Our repeated experience [YMMV] has been that there is NO payoff to decreasing the level of testing.

The conclusion (from my point of view) is that a driving factor is if you are in this for the "long haul" or is software development a transient or secondary activity. If the former, then any investment in testing will payoff; if the latter then the risks of releasing buggy code may be more tolerable, and indepth testing may not be a financial win.