MLPerf Mobile Inference Benchmark

• Main Authors: Reddi, Vijay Janapa, Kanter, David, Mattson, Peter, Duke, Jared, Nguyen, Thai, Chukka, Ramesh, Shiring, Ken, Tan, Koan-Sin, Charlebois, Mark, Chou, William, El-Khamy, Mostafa, Hong, Jungwook, John, Tom St, Trinh, Cindy, Buch, Michael, Mazumder, Mark, Markovic, Relia, Atta, Thomas, Cakir, Fatih, Charkhabi, Masoud, Chen, Xiaodong, Chiang, Cheng-Ming, Dexter, Dave, Heo, Terry, Schmuelling, Gunther, Shabani, Maryam, Zika, Dylan
• Format: Journal Article
• Language: English
• Published: 03-12-2020
• Subjects: Computer Science - Distributed, Parallel, and Cluster Computing; Computer Science - Learning
• DOI: 10.48550/arxiv.2012.02328

This paper presents the first industry-standard open-source machine learning
(ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices
with different AI chips and software stacks. The benchmark draws from the
expertise of leading mobile-SoC vendors, ML-framework providers, and model
producers. It comprises a suite of models that operate with standard data sets,
quality metrics and run rules. We describe the design and implementation of
this domain-specific ML benchmark. The current benchmark version comes as a
mobile app for different computer vision and natural language processing tasks.
The benchmark also supports non-smartphone devices, such as laptops and mobile
PCs. Benchmark results from the first two rounds reveal the overwhelming
complexity of the underlying mobile ML system stack, emphasizing the need for
transparency in mobile ML performance analysis. The results also show that the
strides being made all through the ML stack improve performance. Within six
months, offline throughput improved by 3x, while latency reduced by as much as
12x. ML is an evolving field with changing use cases, models, data sets and
quality targets. MLPerf Mobile will evolve and serve as an open-source
community framework to guide research and innovation for mobile AI.