Random Testing: Theoretical Results and Practical Implications

• Published in: IEEE transactions on software engineering Vol. 38; no. 2; pp. 258 - 277
• Main Authors: Arcuri, A., Iqbal, M. Z., Briand, L.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-03-2012; IEEE Computer Society
• Subjects: adaptive random testing; Algorithm design and analysis; Analysis; Color; Coupon collector; Digital Object Identifier; Failure; Generators; partition testing; predictability; random testing; Random variables; Schur function; Software; Software engineering; Studies; Testing; theory; Upper bound
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2011.121

A substantial amount of work has shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications have been reported in the literature. Although it is not going to solve all possible testing problems, random testing appears to be an essential tool in the hands of software testers. In this paper, we review and analyze the debate about random testing. Its benefits and drawbacks are discussed. Novel results addressing general questions about random testing are also presented, such as how long does random testing need, on average, to achieve testing targets (e.g., coverage), how does it scale, and how likely is it to yield similar results if we rerun it on the same testing problem (predictability). Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Results show that there are practical situations in which random testing is a viable option. Our theorems are backed up by simulations and we show how they can be applied to most types of software and testing criteria. In light of these results, we then assess the validity of empirical analyzes reported in the literature and derive guidelines for both practitioners and scientists.