VPU-EM: An Event-based Modeling Framework to Evaluate NPU Performance and Power Efficiency at Scale
Authors:
Charles Qi,
Yi Wang,
Hui Wang,
Yang Lu,
Shiva Shankar Subramanian,
Finola Cahill,
Conall Tuohy,
Victor Li,
Xu Qian,
Darren Crews,
Ling Wang,
Shivaji Roy,
Andrea Deidda,
Martin Power,
Niall Hanrahan,
Rick Richmond,
Umer Cheema,
Arnab Raha,
Alessandro Palla,
Gary Baugh,
Deepak Mathaikutty
Abstract:
State-of-art NPUs are typically architected as a self-contained sub-system with multiple heterogeneous hardware computing modules, and a dataflow-driven programming model. There lacks well-established methodology and tools in the industry to evaluate and compare the performance of NPUs from different architectures. We present an event-based performance modeling framework, VPU-EM, targeting scalabl…
▽ More
State-of-art NPUs are typically architected as a self-contained sub-system with multiple heterogeneous hardware computing modules, and a dataflow-driven programming model. There lacks well-established methodology and tools in the industry to evaluate and compare the performance of NPUs from different architectures. We present an event-based performance modeling framework, VPU-EM, targeting scalable performance evaluation of modern NPUs across diversified AI workloads. The framework adopts high-level event-based system-simulation methodology to abstract away design details for speed, while maintaining hardware pipelining, concurrency and interaction with software task scheduling. It is natively developed in Python and built to interface directly with AI frameworks such as Tensorflow, PyTorch, ONNX and OpenVINO, linking various in-house NPU graph compilers to achieve optimized full model performance. Furthermore, VPU-EM also provides the capability to model power characteristics of NPU in Power-EM mode to enable joint performance/power analysis. Using VPU-EM, we conduct performance/power analysis of models from representative neural network architecture. We demonstrate that even though this framework is developed for Intel VPU, an Intel in-house NPU IP technology, the methodology can be generalized for analysis of modern NPUs.
△ Less
Submitted 17 March, 2023;
originally announced March 2023.
Towards Optimal VPU Compiler Cost Modeling by using Neural Networks to Infer Hardware Performances
Authors:
Ian Frederick Vigogne Goodbody Hunter,
Alessandro Palla,
Sebastian Eusebiu Nagy,
Richard Richmond,
Kyle McAdoo
Abstract:
Calculating the most efficient schedule of work in a neural network compiler is a difficult task. There are many parameters to be accounted for that can positively or adversely affect that schedule depending on their configuration - How work is shared between distributed targets, the subdivision of tensors to fit in memory, toggling the enablement of optimizations, etc. Traditionally, neural netwo…
▽ More
Calculating the most efficient schedule of work in a neural network compiler is a difficult task. There are many parameters to be accounted for that can positively or adversely affect that schedule depending on their configuration - How work is shared between distributed targets, the subdivision of tensors to fit in memory, toggling the enablement of optimizations, etc. Traditionally, neural network compilers determine how to set these values by building a graph of choices and choosing the path with minimal 'cost'. These choices and their corresponding costs are usually determined by an algorithm crafted by engineers with a deep knowledge of the target platform. However, when the amount of options available to a compiler is large, it is very difficult to ensure that these models consistently produce an optimal schedule for all scenarios, whilst still completing compilation in an acceptable timeframe. This paper presents 'VPUNN' - a neural network-based cost model trained on low-level task profiling that consistently outperforms the state-of-the-art cost modeling in Intel's line of VPU processors.
△ Less
Submitted 9 May, 2022;
originally announced May 2022.
