Testing with Complex Data Dependencies: Untie the Gordian Knot

Testing software functionality is a fairly straightforward idea: set up pre-conditions, perform a series of actions, verify the results. In practice, however, those three seemingly simple steps can turn out to be extremely difficult and time-consuming.

In general, there are two approaches to software testing:

Black Box: the tester uses the software pretty much as an end-user would, ignorant of the implementation details. This approach is best used on fairly simple software with finite combinations of inputs. When applied to more complex systems, testing is usually limited to "happy path" scenarios.

White Box: the tester has full access to, and knowledge of, the inner-workings of the code. This approach allows for more targeted testing because the tester can avoid running test cases that are effectively duplicates of others. Please note, however, that it can sometimes be difficult to tell which test cases are duplicate effort.

Software systems that apply business rules to large sets of highly inter-related data are notoriously difficult to test. Test cases in this kind of system can easily call for pre-conditions that are difficult to describe, much less set up. Frankly, neither method, used alone, will ensure a high level of quality. White Box is impossible to apply to complex systems with 100% coverage. Black Box helps to target testing efforts, but in systems where the dependency is in the data rather than in the code, tracing the lines of dependency can be exceptionally difficult.

Instead, apply a hybrid approach. Turn on full debug logging and take a swing through the happy paths as in the White Box approach. By looking at the logs, you should be able to identify the test cases which hit the code that needs testing. Then, apply the Black Box method to those test cases and any that, from a low level, are closely related. The results will be much less time-consuming than pure White Box, and much more complete than pure Black Box. You might even learn something new about how your application really works. If you do, document it!