-
Throughput Maximization of DNN Inference: Batching or Multi-Tenancy?
Authors:
Seyed Morteza Nabavinejad,
Masoumeh Ebrahimi,
Sherief Reda
Abstract:
Deployment of real-time ML services on warehouse-scale infrastructures is on the increase. Therefore, decreasing latency and increasing throughput of deep neural network (DNN) inference applications that empower those services have attracted attention from both academia and industry. A common solution to address this challenge is leveraging hardware accelerators such as GPUs. To improve the infere…
▽ More
Deployment of real-time ML services on warehouse-scale infrastructures is on the increase. Therefore, decreasing latency and increasing throughput of deep neural network (DNN) inference applications that empower those services have attracted attention from both academia and industry. A common solution to address this challenge is leveraging hardware accelerators such as GPUs. To improve the inference throughput of DNNs deployed on GPU accelerators, two common approaches are employed: Batching and Multi-Tenancy. Our preliminary experiments show that the effect of these approaches on the throughput depends on the DNN architecture. Taking this observation into account, we design and implement DNNScaler which aims to maximize the throughput of interactive AI-powered services while meeting their latency requirements. DNNScaler first detects the suitable approach (Batching or Multi-Tenancy) that would be most beneficial for a DNN regarding throughput improvement. Then, it adjusts the control knob of the detected approach (batch size for Batching and number of co-located instances for Multi-Tenancy) to maintain the latency while increasing the throughput. Conducting an extensive set of experiments using well-known DNNs from a variety of domains, several popular datasets, and a cutting-edge GPU, the results indicate that DNNScaler can improve the throughput by up to 14x (218% on average) compared with the previously proposed approach, while meeting the latency requirements of the services.
△ Less
Submitted 26 August, 2023;
originally announced August 2023.
-
Opportunities of Renewable Energy Powered DNN Inference
Authors:
Seyed Morteza Nabavinejad,
Tian Guo
Abstract:
With the proliferation of the adoption of renewable energy in powering data centers, addressing the challenges of such energy sources has attracted researchers from academia and industry. One of the challenging characteristics of data centers with renewable energy is the intrinsic power fluctuation. Fluctuation in renewable power supply inevitably requires adjusting applications' power consumption…
▽ More
With the proliferation of the adoption of renewable energy in powering data centers, addressing the challenges of such energy sources has attracted researchers from academia and industry. One of the challenging characteristics of data centers with renewable energy is the intrinsic power fluctuation. Fluctuation in renewable power supply inevitably requires adjusting applications' power consumption, which can lead to undesirable performance degradation. This paper investigates the possible control knobs to manage the power and performance of a popular cloud workload, i.e., deep neural network inference, under the fluctuating power supply. Through empirical profiling and trace-driven simulations, we observe the different impact levels associated with inference control knobs on throughput, under varying power supplies. Based on our observations, we provide a list of future research directions to leverage the control knobs to achieve high throughput.
△ Less
Submitted 21 June, 2023;
originally announced June 2023.
-
On the Impact of Device-Level Techniques on Energy-Efficiency of Neural Network Accelerators
Authors:
Seyed Morteza Nabavinejad,
Behzad Salami
Abstract:
Energy-efficiency is a key concern for neural network applications. To alleviate this issue, hardware acceleration using FPGAs or GPUs can provide better energy-efficiency than general-purpose processors. However, further improvement of the energy-efficiency of such accelerators will be extremely beneficial specially to deploy neural network in power-constrained edge computing environments. In thi…
▽ More
Energy-efficiency is a key concern for neural network applications. To alleviate this issue, hardware acceleration using FPGAs or GPUs can provide better energy-efficiency than general-purpose processors. However, further improvement of the energy-efficiency of such accelerators will be extremely beneficial specially to deploy neural network in power-constrained edge computing environments. In this paper, we experimentally explore the potential of device-level energy-efficiency techniques (e.g.,supply voltage underscaling, frequency scaling, and data quantization) for representative off-the-shelf FPGAs compared to GPUs. Frequency scaling in both platforms can improve the power and energy consumption but with performance overhead, e.g.,in GPUs it improves the power consumption and GOPs/J by up to 34% and 28%, respectively. However, leveraging reduced-precision instructions improves power (up to 13%), energy (up to 20%), and performance (up to 7%) simultaneously, with negligible reduction in accuracy of neural network accuracy.
△ Less
Submitted 26 June, 2021;
originally announced June 2021.
