Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning
Authors:
Scott Cyphers,
Arjun K. Bansal,
Anahita Bhiwandiwalla,
Jayaram Bobba,
Matthew Brookhart,
Avijit Chakraborty,
Will Constable,
Christian Convey,
Leona Cook,
Omar Kanawi,
Robert Kimball,
Jason Knight,
Nikolay Korovaiko,
Varun Kumar,
Yixing Lao,
Christopher R. Lishka,
Jaikrishnan Menon,
Jennifer Myers,
Sandeep Aswath Narayana,
Adam Procter,
Tristan J. Webb
Abstract:
The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the tr…
▽ More
The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of frameworks and $p$ is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).
△ Less
Submitted 29 January, 2018; v1 submitted 24 January, 2018;
originally announced January 2018.
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Authors:
Urs Köster,
Tristan J. Webb,
Xin Wang,
Marcel Nassar,
Arjun K. Bansal,
William H. Constable,
Oğuz H. Elibol,
Scott Gray,
Stewart Hall,
Luke Hornof,
Amir Khosrowshahi,
Carey Kloss,
Ruby J. Pai,
Naveen Rao
Abstract:
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the F…
▽ More
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet, a deep residual network and a generative adversarial network, using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.
△ Less
Submitted 2 December, 2017; v1 submitted 6 November, 2017;
originally announced November 2017.
