-
Schrödinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training
Authors:
Miloš Nikolić,
Enrique Torres Sanchez,
Jiahui Wang,
Ali Hadi Zadeh,
Mostafa Mahmoud,
Ameer Abdelhadi,
Kareem Ibrahim,
Andreas Moshovos
Abstract:
The transfer of tensors from/to memory during neural network training dominates time and energy. To improve energy efficiency and performance, research has been exploring ways to use narrower data representations. So far, these attempts relied on user-directed trial-and-error to achieve convergence. We present methods that relieve users from this responsibility. Our methods dynamically adjust the…
▽ More
The transfer of tensors from/to memory during neural network training dominates time and energy. To improve energy efficiency and performance, research has been exploring ways to use narrower data representations. So far, these attempts relied on user-directed trial-and-error to achieve convergence. We present methods that relieve users from this responsibility. Our methods dynamically adjust the size and format of the floating-point containers used for activations and weights during training, achieving adaptivity across three dimensions: i) which datatype to use, ii) on which tensor, and iii) how it changes over time. The different meanings and distributions of exponent and mantissas lead us to tailored approaches for each. We present two lossy pairs of methods to eliminate as many mantissa and exponent bits as possible without affecting accuracy. Quantum Mantissa and Quantum Exponent are machine learning compression methods that tap into the gradient descent algorithm to learn the minimal mantissa and exponent bitlengths on a per-layer granularity. They automatically learn that many tensors can use just 1 or 2 mantissa bits and 3 or 4 exponent bits. Overall, the two machine learning methods reduce the footprint by $4.74\times$. Alternatively, BitWave observes changes in the loss function during training to adjust mantissa and exponent bitlengths network-wide, yielding a $3.19\times$ reduction in footprint. Finally, we present an optional method, Gecko, to exploit the naturally emerging, lop-sided exponent distribution to losslessly compress resulting exponents from Quantum Exponent or BitWave and, on average, improve compression rates to $5.64\times$ and $4.56\times$.
△ Less
Submitted 16 May, 2024; v1 submitted 28 April, 2022;
originally announced April 2022.
-
Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
Authors:
Ali Hadi Zadeh,
Mostafa Mahmoud,
Ameer Abdelhadi,
Andreas Moshovos
Abstract:
Increasingly larger and better Transformer models keep advancing state-of-the-art accuracy and capability for Natural Language Processing applications. These models demand more computational power, storage, and energy. Mokey reduces the footprint of state-of-the-art 32-bit or 16-bit floating-point transformer models by quantizing all values to 4-bit indexes into dictionaries of representative 16-b…
▽ More
Increasingly larger and better Transformer models keep advancing state-of-the-art accuracy and capability for Natural Language Processing applications. These models demand more computational power, storage, and energy. Mokey reduces the footprint of state-of-the-art 32-bit or 16-bit floating-point transformer models by quantizing all values to 4-bit indexes into dictionaries of representative 16-bit fixed-point centroids. Mokey does not need fine-tuning, an essential feature as often the training resources or datasets are not available to many. Exploiting the range of values that naturally occur in transformer models, Mokey selects centroid values to also fit an exponential curve. This unique feature enables Mokey to replace the bulk of the original multiply-accumulate operations with narrow 3b fixed-point additions resulting in an area- and energy-efficient hardware accelerator design. Over a set of state-of-the-art transformer models, the Mokey accelerator delivers an order of magnitude improvements in energy efficiency over a Tensor Cores-based accelerator while improving performance by at least $4\times$ and as much as $15\times$ depending on the model and on-chip buffering capacity. Optionally, Mokey can be used as a memory compression assist for any other accelerator, transparently stashing wide floating-point or fixed-point activations or weights into narrow 4-bit indexes. Mokey proves superior to prior state-of-the-art quantization methods for Transformers.
△ Less
Submitted 23 March, 2022;
originally announced March 2022.
-
Some applications of phase-type distributions in recurrent events
Authors:
Roufeh Asghari,
Amin Hassan Zadeh
Abstract:
In this paper, the recurrent events that can occur more than one over the follow-up time have been modeled by phase-type distributions. We use the finite-state continuous-time Markov process with multi states for patients with recurrent events. The number of recurrences until time $t$, the time stay for every state and the time till death are of importances. The time till death is assumed to have…
▽ More
In this paper, the recurrent events that can occur more than one over the follow-up time have been modeled by phase-type distributions. We use the finite-state continuous-time Markov process with multi states for patients with recurrent events. The number of recurrences until time $t$, the time stay for every state and the time till death are of importances. The time till death is assumed to have a phase-type distribution (which is defined in a Markov chain environment) with interpretable parameters. The underlying continuous-time Markov chain has one absorbing state (death) and transient states to reflect recovery and disease stages. A system of differential equations is obtained to calculate the probability of various number of transitions, the conditional expected time to stay in a disease stage and the probability of transition from a stage to another. The model has been calibrated via a real and simulated datasets. The bootstrap techniques have been used to construct the confidence intervals for the parameters.
△ Less
Submitted 24 January, 2022;
originally announced January 2022.
-
FPRaker: A Processing Element For Accelerating Neural Network Training
Authors:
Omar Mohamed Awad,
Mostafa Mahmoud,
Isak Edo,
Ali Hadi Zadeh,
Ciaran Bannon,
Anand Jayarajan,
Gennady Pekhimenko,
Andreas Moshovos
Abstract:
We present FPRaker, a processing element for composing training accelerators. FPRaker processes several floating-point multiply-accumulation operations concurrently and accumulates their result into a higher precision accumulator. FPRaker boosts performance and energy efficiency during training by taking advantage of the values that naturally appear during training. Specifically, it processes the…
▽ More
We present FPRaker, a processing element for composing training accelerators. FPRaker processes several floating-point multiply-accumulation operations concurrently and accumulates their result into a higher precision accumulator. FPRaker boosts performance and energy efficiency during training by taking advantage of the values that naturally appear during training. Specifically, it processes the significand of the operands of each multiply-accumulate as a series of signed powers of two. The conversion to this form is done on-the-fly. This exposes ineffectual work that can be skipped: values when encoded have few terms and some of them can be discarded as they would fall outside the range of the accumulator given the limited precision of floating-point. We demonstrate that FPRaker can be used to compose an accelerator for training and that it can improve performance and energy efficiency compared to using conventional floating-point units under ISO-compute area constraints. We also demonstrate that FPRaker delivers additional benefits when training incorporates pruning and quantization. Finally, we show that FPRaker naturally amplifies performance with training methods that use a different precision per layer.
△ Less
Submitted 15 October, 2020;
originally announced October 2020.
-
TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
Authors:
Mostafa Mahmoud,
Isak Edo,
Ali Hadi Zadeh,
Omar Mohamed Awad,
Gennady Pekhimenko,
Jorge Albericio,
Andreas Moshovos
Abstract:
TensorDash is a hardware level technique for enabling data-parallel MAC units to take advantage of sparsity in their input operand streams. When used to compose a hardware accelerator for deep learning, TensorDash can speedup the training process while also increasing energy efficiency. TensorDash combines a low-cost, sparse input operand interconnect comprising an 8-input multiplexer per multipli…
▽ More
TensorDash is a hardware level technique for enabling data-parallel MAC units to take advantage of sparsity in their input operand streams. When used to compose a hardware accelerator for deep learning, TensorDash can speedup the training process while also increasing energy efficiency. TensorDash combines a low-cost, sparse input operand interconnect comprising an 8-input multiplexer per multiplier input, with an area-efficient hardware scheduler. While the interconnect allows a very limited set of movements per operand, the scheduler can effectively extract sparsity when it is present in the activations, weights or gradients of neural networks. Over a wide set of models covering various applications, TensorDash accelerates the training process by $1.95{\times}$ while being $1.89\times$ more energy-efficient, $1.6\times$ more energy efficient when taking on-chip and off-chip memory accesses into account. While TensorDash works with any datatype, we demonstrate it with both single-precision floating-point units and bfloat16.
△ Less
Submitted 1 September, 2020;
originally announced September 2020.
-
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
Authors:
Ali Hadi Zadeh,
Isak Edo,
Omar Mohamed Awad,
Andreas Moshovos
Abstract:
Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art…
▽ More
Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (1) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (2) replaces most multiply-accumulations with additions, and (3) reduces the off-chip traffic by amplifying on-chip memory capacity.
△ Less
Submitted 26 September, 2020; v1 submitted 7 May, 2020;
originally announced May 2020.
-
Modelling Joint Lifetimes of Couples by Using Bivariate Phase-type Distributions
Authors:
Amin Hassan Zadeh,
Soroush Amirhashchi
Abstract:
Many insurance products and pension plans provide benefits which are related to couples, and thus under influence of the survival status of two lives. Some studies show the future lifetime of couples is correlated. Three reasons are available to confirm this fact: (1) catastrophe events that affect both lives, (2) the impact of spousal death and (3) the long-term association due to common life sty…
▽ More
Many insurance products and pension plans provide benefits which are related to couples, and thus under influence of the survival status of two lives. Some studies show the future lifetime of couples is correlated. Three reasons are available to confirm this fact: (1) catastrophe events that affect both lives, (2) the impact of spousal death and (3) the long-term association due to common life style. Dependence between lifetimes of couples could have a financial impact on insurance companies and pension plans providers. In this paper, we use a health index called physiological age in a Markov process context by that we model aging process of joint and last survivor statuses. Under this model, future joint lifetime of couples follows a bivariate phase-type distribution. The model has physical interpretation and closed-form expressions for actuarial quantities and owns tractable computation for the other ones. We use the model to pricing products relevant to couples annuities and life insurances.
△ Less
Submitted 26 June, 2018;
originally announced June 2018.
