Benchmarking TinyML Systems: Challenges and Direction
Authors:
Colby R. Banbury,
Vijay Janapa Reddi,
Max Lam,
William Fu,
Amin Fazel,
Jeremy Holleman,
Xinyuan Huang,
Robert Hurtado,
David Kanter,
Anton Lokhmotov,
David Patterson,
Danilo Pau,
Jae-sun Seo,
Jeff Sieracki,
Urmish Thakker,
Marian Verhelst,
Poonam Yadav
Abstract:
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field re…
▽ More
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards develo** a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.
△ Less
Submitted 29 January, 2021; v1 submitted 10 March, 2020;
originally announced March 2020.
