Automatic test report augmentation to assist crowdsourced testing

• Published in: Frontiers of Computer Science Vol. 13; no. 5; pp. 943 - 959
• Main Authors: CHEN, Xin, JIANG, He, CHEN, Zhenyu, HE, Tieke, NIE, Liming
• Format: Journal Article
• Language: English
• Published: Beijing Higher Education Press 01-10-2019; Springer Nature B.V
• Subjects: Applications programs; Computer Science; crowdsourced testing; Crowdsourcing; Descriptions; Effectiveness; Mobile computing; Natural language; Natural language processing; Research Article; Software testing; test report; test report augmentation; TF-IDF; natural language processing; test report augmentation; test report; TF-IDF; crowdsourced testing
• ISSN: 2095-2228; 2095-2236
• DOI: 10.1007/s11704-018-7308-5

In crowdsourced mobile application testing, workers are often inexperienced in and unfamiliar with software testing. Meanwhile, workers edit test reports in descriptive natural language on mobile devices. Thus, these test reports generally lack important details and challenge developers in understanding the bugs. To improve the quality of inspected test reports, we issue a new problem of test report augmentation by leveraging the additional useful information contained in duplicate test reports. In this paper, we propose a new framework named test report augmentation framework (TRAF) towards resolving the problem. First, natural language processing (NLP) techniques are adopted to preprocess the crowdsourced test reports. Then, three strategies are proposed to augment the environments, inputs, and descriptions of the inspected test reports, respectively. Finally, we visualize the augmented test reports to help developers distinguish the added information. To evaluate TRAF, we conduct experiments over five industrial datasets with 757 crowdsourced test reports. Experimental results show that TRAF can recommend relevant inputs to augment the inspected test reports with 98.49% in terms of NDCG and 88.65% in terms of precision on average, and identify valuable sentences from the descriptions of duplicates to augment the inspected test reports with 83.58% in terms of precision, 77.76% in terms of recall, and 78.72% in terms of F-measure on average. Meanwhile, empirical evaluation also demonstrates that augmented test reports can help developers understand and fix bugs better.