Variant Black Box Technique

The first two test suites were developed using a variant black box testing technique. It is less a new technique
than a codification of good practice. Its first stage follows these steps:
Black 1: Test conditions are methodically generated from the form of the specification. For example, a
precondition of the form "X must be true" generates two test conditions: "X is true" and "X is false",
regardless of what X actually is. These test conditions are further refined by processing connectives like
AND and OR (using rules similar to cause-effect testing [Myers79]). For example, "X AND Y" generates
three test cases:
X is true, Y is true.
X is false, Y is true.
X is true, Y is false.
Black 2: Next, test conditions are generated from the content of the specification. Specifications contain
cliches [Rich90]. A search of a circular list is a typical cliche. Certain data types are also used in a cliched
way. For example, the UNIX pathname as a slash-separated string is implicit in many specifications.
Cliches are identified by looking at the nouns and verbs in the specification: "search", "list", "pathname".
The implementations of these cliches often contain cliched faults.1 For example:
(1) If the specification includes a search for an element of a circular list, one test condition is that the list
does not include the element. The expectation is that the search might go into an infinite loop.
(2) If a function decomposes and reconstructs UNIX pathnames, one test condition is that it be given a
pathname of the form "X//Y", because programmers often fail to remember that two slashes are
equivalent to one.
Because experienced testers know cliched faults, they use them when generating tests. However, writing
the cliches and faults down in a catalog reduces dependency on experience and memory. Such a catalog
has been written; sample entries are given in Appendix B.
Black 3: These test conditions are combined into test cases. A test case is a precise description of particular
input values and expected results.
The next stage is called broken box testing2. It exposes information that the specification hides from the
user, but that the tester needs to know. For example, a user needn’t know that a routine uses a hash table,
but a tester would want to probe hash table collision handling. There are two steps:
Broken 1: The code is scanned, looking for important operations and types that aren’t visible in the
specification. Types are often recognized because of comments about the use of a variable. (A variable’s
declaration does not contain all the information needed; an integer may be a count, a range, a percentage,
or an index, each of which produces different test conditions.) Cliched operations are often distinct blocks
of code separated by blank lines or comments. Of course, the key way you recognize a cliche is by having
seen it often before. Once found, these cliches are then treated exactly as if they had been found in the
specification. No attempt is made to find and satisfy coverage conditions. (The name indicates this: the
box is broken open enough for us to see gross features, but we don’t look at detail.)
Broken 2: These new test conditions are combined into new test cases.
In production use, a tester presented with a specification and a finished program would omit step Black3.
Test conditions would be derived from both the specification and the code, then combined together. This
would minimize the size of the test suite. For this experiment, the two test suites were kept separate, in
order to see what the contribution of looking at the code would be. This also simulates the more desirable
case where the tests are designed before the code is written. (Doing so reduces elapsed time, since tests
can be written while the code is being written. Further, the act of writing concrete tests often discovers
errors in the specification, and it’s best to discover those early.)
In this experiment, the separation is artificial. I wrote the specifications for six of the programs, laid them
aside for a month, then wrote the test cases, hoping that willful forgetfulness would make the black box
testing less informed by the implementation.