Measuring the Diversity of a Test Set With Distance Entropy

• Published in: IEEE transactions on reliability Vol. 65; no. 1; pp. 19 - 27
• Main Authors: Shi, Qingkai, Chen, Zhenyu, Fang, Chunrong, Feng, Yang, Xu, Baowen
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-03-2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Complexity; diversity; Empirical analysis; Entropy; Euclidean distance; Fault detection; Fault detection capability; Historical structures; Image edge detection; metrics; minimum spanning tree; Run time (computers); Test sets; Testing; Time complexity
• ISSN: 0018-9529; 1558-1721
• DOI: 10.1109/TR.2015.2434953

Most existing metrics that we call white-box metrics, such as coverage metrics, require white-box information, like program structure information, and historical runtime information, to evaluate the fault detection capability of a test set. In practice, such white-box information is usually unavailable or difficult to obtain, which means they often cannot be used. In this paper, we propose a black-box metric, distance entropy, based on the diversification idea behind many published diversity-based techniques. Distance entropy provides a possible solution for test set evaluation when white-box information is not available. The empirical study illustrates that distance entropy can effectively evaluate test sets if the distance metric between tests is well defined. Meanwhile, distance entropy outperforms simple diversity metrics without increasing time complexity.