MLPerf Mobile Inference Benchmark

This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks. The benchmark draws from the expertise of leading mobile-SoC vendors, ML-framework providers, and m...

Full description

Saved in:

Bibliographic Details
Main Authors:	Reddi, Vijay Janapa, Kanter, David, Mattson, Peter, Duke, Jared, Nguyen, Thai, Chukka, Ramesh, Shiring, Ken, Tan, Koan-Sin, Charlebois, Mark, Chou, William, El-Khamy, Mostafa, Hong, Jungwook, John, Tom St, Trinh, Cindy, Buch, Michael, Mazumder, Mark, Markovic, Relia, Atta, Thomas, Cakir, Fatih, Charkhabi, Masoud, Chen, Xiaodong, Chiang, Cheng-Ming, Dexter, Dave, Heo, Terry, Schmuelling, Gunther, Shabani, Maryam, Zika, Dylan
Format:	Journal Article
Language:	English
Published:	03-12-2020
Subjects:	Computer Science - Distributed, Parallel, and Cluster Computing Computer Science - Learning
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks. The benchmark draws from the expertise of leading mobile-SoC vendors, ML-framework providers, and model producers. It comprises a suite of models that operate with standard data sets, quality metrics and run rules. We describe the design and implementation of this domain-specific ML benchmark. The current benchmark version comes as a mobile app for different computer vision and natural language processing tasks. The benchmark also supports non-smartphone devices, such as laptops and mobile PCs. Benchmark results from the first two rounds reveal the overwhelming complexity of the underlying mobile ML system stack, emphasizing the need for transparency in mobile ML performance analysis. The results also show that the strides being made all through the ML stack improve performance. Within six months, offline throughput improved by 3x, while latency reduced by as much as 12x. ML is an evolving field with changing use cases, models, data sets and quality targets. MLPerf Mobile will evolve and serve as an open-source community framework to guide research and innovation for mobile AI.
DOI:	10.48550/arxiv.2012.02328