Black-Box Testing: Black-box testing assumes an external perspective of the system under test (SUT). It uses concrete input test data to obtain results from the SUT, which are then checked against some oracle. Usually, the oracle uses functional properties, comparing the output data with expected values. The approach is easy in general, but quickly becomes problematic for complex SUTs, especially if one of the following conditions exists:
a. The set of possible, relevant inputs is large (this can be aggravated by particular implementations of algorithms that lead to numeric instability; i.e., the data might not seem to be suspicious from a domain perspective but still causes failure in a given implementation).
b. The SUT behavior cannot be completely controlled by input data (for example, its behavior depends on scheduling in concurrent programs).
c. The result data are complex and requires expensive analysis to verify.
d. The result data can be subject to “accidental correctness”; i.e., the SUT produces a right result for wrong reasons. This usually is caused by a small set of possible outputs, computed from a large input space.
e. Running the test requires manual action to enter inputs, start the SUT, or analyze the output.
f. There is no measure of relevant test coverage.