Do you always create test cases for the code path less travelled – the unusual code path that rarely occurs. Are you sure? On all but the simplest projects it can be hard to know. Like many programmers, I use a code coverage logger to provide an indication of how well my tests checked the product code. This was a mistake. I discovered that, even though the code coverage log indicated that the tests had forced execution of 100% of the source lines in many modules, an investigation revealed that I had failed to test up to 25% of the code paths. How was that possible?
Actually, it is very easy to create code where a code path has no unique source lines. Years ago, I often would write code that checked variables for range or validity and then did nothing if the value was not proper. Consider the following incredibly simple example.
// Fetch 1st char in receive buffer to determine
// if message start
if( rxBufrPtr != CONST_NULLPTR)
{
// Valid receive buffer
firstChar = *rxBufrPtr;
if( firstChar == CONST_STARTOFMSG)
{
// Message Start detected
// Process Message
Message_Process( rxBufrPtr);
}
}
// done
return;
If I write just one test having a message beginning with CONST_STARTOFMSG and run it, the coverage log would show that all lines have been tested. As can be clearly seen, several important code paths have not been tested. The 100% reported code coverage has provided a false sense of accomplishment. I understand that it’s hard to be concerned about not testing all code paths in this simple example, but I’m sure you can conceive of situations where concern would be warranted.
Several years ago, I built firmware for a certified IEC61508 Functional Safety device. The testing level required depended upon the SIL (Safety Integrity Level) certification of the device. At the lowest level – SIL 1 – all module entry points must be tested. The next higher SIL certification requires 100% code statement coverage. Even higher SIL levels require testing of all branches or testing of all branch conditions. (High levels of testing are needed only for code deemed critical for safety.)
We intended to use the code coverage report as one of the indicators that proper testing had been done. To avoid the situation demonstrated above, the code was written a bit differently, shown in this modified example.
// Fetch 1st char in receive buffer to determine
// if message start
if( rxBufrPtr != CONST_NULLPTR)
{
// Valid receive buffer
firstChar = *rxBufrPtr;
if( firstChar == CONST_STARTOFMSG)
{
// Message start detected
// Process Message
Message_Process( rxBufrPtr);
}
else
{
// Message start not detected
// — do nothing
MACRO_CODEPATH_TRACKING();
}
}
else
{
// Invalid pointer
// — do nothing
MACRO_CODEPATH_TRACKING();
}
// done
return;
We used “#define” to create a code macro – MACRO_CODEPATH_TRACKING. When testing, this macro compiles to some statement that will not be removed by the optimizer. When not testing, the macro can be defined to compile to nothing so there is no effect on the code.
Now, the code coverage log correctly shows whether or not all paths through the code have been tested.
I suggest that programmers make it a habit to add the extra else clause with such a macro when writing the initial code. I can personally testify that adding the else clause with the code tracking macro later in the development is tedious.
Note: Some compiler tools provide a built-in “__nop()” which puts a NOP instruction into the generated code that is never optimized out; if this is not available, you will need to use something that will remain after optimization like incrementing a volatile variable.
Note: In full disclosure, I often admonish programmers to avoid defined macros; they are a bad programming practice, a poor substitute for real functions and sometimes cause horribly difficult to debug problems. Obviously, I find the above use acceptable.