LIRAT: Layout and Image Recognition Driving Automated Mobile Testing of Cross-Platform

• Published in: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) pp. 1066 - 1069
• Main Authors: Yu, Shengcheng, Fang, Chunrong, Feng, Yang, Zhao, Wenyuan, Chen, Zhenyu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-11-2019
• Subjects: Cross-Platform Testing; Feature extraction; Image recognition; Layout; Optical character recognition software; Record and Replay; Smart phones; Testing
• ISSN: 2643-1572
• DOI: 10.1109/ASE.2019.00103

The fragmentation issue spreads over multiple mobile platforms such as Android, iOS, mobile web, and WeChat, which hinders test scripts from running across platforms. To reduce the cost of adapting scripts for various platforms, some existing tools apply conventional computer vision techniques to replay the same script on multiple platforms. However, because these solutions can hardly identify dynamic or similar widgets. It becomes difficult for engineers to apply them in practice. In this paper, we present an image-driven tool, namely LIRAT, to record and replay test scripts cross platforms, solving the problem of test script cross-platform replay for the first time. LIRAT records screenshots and layouts of the widgets, and leverages image understanding techniques to locate them in the replay process. Based on accurate widget localization, LIRAT supports replaying test scripts across devices and platforms. We employed LIRAT to replay 25 scripts from 5 application across 8 Android devices and 2 iOS devices. The results show that LIRAT can replay 88% scripts on Android platforms and 60% on iOS platforms. The demo can be found at: https: //github.com/YSC9848/LIRAT.