VPU-EM: An Event-based Modeling Framework to Evaluate NPU Performance and Power Efficiency at Scale
Authors:
Charles Qi,
Yi Wang,
Hui Wang,
Yang Lu,
Shiva Shankar Subramanian,
Finola Cahill,
Conall Tuohy,
Victor Li,
Xu Qian,
Darren Crews,
Ling Wang,
Shivaji Roy,
Andrea Deidda,
Martin Power,
Niall Hanrahan,
Rick Richmond,
Umer Cheema,
Arnab Raha,
Alessandro Palla,
Gary Baugh,
Deepak Mathaikutty
Abstract:
State-of-art NPUs are typically architected as a self-contained sub-system with multiple heterogeneous hardware computing modules, and a dataflow-driven programming model. There lacks well-established methodology and tools in the industry to evaluate and compare the performance of NPUs from different architectures. We present an event-based performance modeling framework, VPU-EM, targeting scalabl…
▽ More
State-of-art NPUs are typically architected as a self-contained sub-system with multiple heterogeneous hardware computing modules, and a dataflow-driven programming model. There lacks well-established methodology and tools in the industry to evaluate and compare the performance of NPUs from different architectures. We present an event-based performance modeling framework, VPU-EM, targeting scalable performance evaluation of modern NPUs across diversified AI workloads. The framework adopts high-level event-based system-simulation methodology to abstract away design details for speed, while maintaining hardware pipelining, concurrency and interaction with software task scheduling. It is natively developed in Python and built to interface directly with AI frameworks such as Tensorflow, PyTorch, ONNX and OpenVINO, linking various in-house NPU graph compilers to achieve optimized full model performance. Furthermore, VPU-EM also provides the capability to model power characteristics of NPU in Power-EM mode to enable joint performance/power analysis. Using VPU-EM, we conduct performance/power analysis of models from representative neural network architecture. We demonstrate that even though this framework is developed for Intel VPU, an Intel in-house NPU IP technology, the methodology can be generalized for analysis of modern NPUs.
△ Less
Submitted 17 March, 2023;
originally announced March 2023.
