-
MTrainS: Improving DLRM training efficiency using heterogeneous memories
Authors:
Hiwot Tadese Kassa,
Paul Johnson,
Jason Akers,
Mrinmoy Ghosh,
Andrew Tulloch,
Dheevatsa Mudigere,
Jongsoo Park,
Xing Liu,
Ronald Dreslinski,
Ehsan K. Ardestani
Abstract:
Recommendation models are very large, requiring terabytes (TB) of memory during training. In pursuit of better quality, the model size and complexity grow over time, which requires additional training data to avoid overfitting. This model growth demands a large number of resources in data centers. Hence, training efficiency is becoming considerably more important to keep the data center power dema…
▽ More
Recommendation models are very large, requiring terabytes (TB) of memory during training. In pursuit of better quality, the model size and complexity grow over time, which requires additional training data to avoid overfitting. This model growth demands a large number of resources in data centers. Hence, training efficiency is becoming considerably more important to keep the data center power demand manageable. In Deep Learning Recommendation Models (DLRM), sparse features capturing categorical inputs through embedding tables are the major contributors to model size and require high memory bandwidth. In this paper, we study the bandwidth requirement and locality of embedding tables in real-world deployed models. We observe that the bandwidth requirement is not uniform across different tables and that embedding tables show high temporal locality. We then design MTrainS, which leverages heterogeneous memory, including byte and block addressable Storage Class Memory for DLRM hierarchically. MTrainS allows for higher memory capacity per node and increases training efficiency by lowering the need to scale out to multiple hosts in memory capacity bound use cases. By optimizing the platform memory hierarchy, we reduce the number of nodes for training by 4-8X, saving power and cost of training while meeting our target training performance.
△ Less
Submitted 19 April, 2023;
originally announced May 2023.
-
Building a Performance Model for Deep Learning Recommendation Model Training on GPUs
Authors:
Zhongyi Lin,
Louis Feng,
Ehsan K. Ardestani,
Jaewon Lee,
John Lundell,
Changkyu Kim,
Arun Kejariwal,
John D. Owens
Abstract:
We devise a performance model for GPU training of Deep Learning Recommendation Models (DLRM), whose GPU utilization is low compared to other well-optimized CV and NLP models. We show that both the device active time (the sum of kernel runtimes) but also the device idle time are important components of the overall device time. We therefore tackle them separately by (1) flexibly adopting heuristic-b…
▽ More
We devise a performance model for GPU training of Deep Learning Recommendation Models (DLRM), whose GPU utilization is low compared to other well-optimized CV and NLP models. We show that both the device active time (the sum of kernel runtimes) but also the device idle time are important components of the overall device time. We therefore tackle them separately by (1) flexibly adopting heuristic-based and ML-based kernel performance models for operators that dominate the device active time, and (2) categorizing operator overheads into five types to determine quantitatively their contribution to the device active time. Combining these two parts, we propose a critical-path-based algorithm to predict the per-batch training time of DLRM by traversing its execution graph. We achieve less than 10% geometric mean average error (GMAE) in all kernel performance modeling, and 4.61% and 7.96% geomean errors for GPU active time and overall E2E per-batch training time prediction with overheads from individual workloads, respectively. A slight increase of 2.19% incurred in E2E prediction error with shared overheads across workloads suggests the feasibility of using shared overheads in large-scale prediction. We show that our general performance model not only achieves low prediction error on DLRM, which has highly customized configurations and is dominated by multiple factors but also yields comparable accuracy on other compute-bound ML models targeted by most previous methods. Using this performance model and graph-level data and task dependency analysis, we show our system can provide more general model-system co-design than previous methods.
△ Less
Submitted 16 November, 2022; v1 submitted 19 January, 2022;
originally announced January 2022.
-
Supporting Massive DLRM Inference Through Software Defined Memory
Authors:
Ehsan K. Ardestani,
Changkyu Kim,
Seung Jae Lee,
Luoshang Pan,
Valmiki Rampersad,
Jens Axboe,
Banit Agrawal,
Fuxun Yu,
Ansha Yu,
Trung Le,
Hector Yuen,
Shishir Juluri,
Akshat Nanda,
Manoj Wodekar,
Dheevatsa Mudigere,
Krishnakumar Nair,
Maxim Naumov,
Chris Peterson,
Mikhail Smelyanskiy,
Vijay Rao
Abstract:
Deep Learning Recommendation Models (DLRM) are widespread, account for a considerable data center footprint, and grow by more than 1.5x per year. With model size soon to be in terabytes range, leveraging Storage ClassMemory (SCM) for inference enables lower power consumption and cost. This paper evaluates the major challenges in extending the memory hierarchy to SCM for DLRM, and presents differen…
▽ More
Deep Learning Recommendation Models (DLRM) are widespread, account for a considerable data center footprint, and grow by more than 1.5x per year. With model size soon to be in terabytes range, leveraging Storage ClassMemory (SCM) for inference enables lower power consumption and cost. This paper evaluates the major challenges in extending the memory hierarchy to SCM for DLRM, and presents different techniques to improve performance through a Software Defined Memory. We show how underlying technologies such as Nand Flash and 3DXP differentiate, and relate to real world scenarios, enabling from 5% to 29% power savings.
△ Less
Submitted 8 November, 2021; v1 submitted 21 October, 2021;
originally announced October 2021.
-
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models
Authors:
Dheevatsa Mudigere,
Yuchen Hao,
Jianyu Huang,
Zhihao Jia,
Andrew Tulloch,
Srinivas Sridharan,
Xing Liu,
Mustafa Ozdal,
Jade Nie,
Jongsoo Park,
Liang Luo,
Jie Amy Yang,
Leon Gao,
Dmytro Ivchenko,
Aarti Basant,
Yuxi Hu,
Jiyan Yang,
Ehsan K. Ardestani,
Xiaodong Wang,
Rakesh Komuravelli,
Ching-Hsiang Chu,
Serhat Yilmaz,
Huayu Li,
Jiyuan Qian,
Zhuobo Feng
, et al. (28 additional authors not shown)
Abstract:
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pa…
▽ More
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pair it with the new evolution of Zion platform, namely ZionEX. We demonstrate the capability to train very large DLRMs with up to 12 Trillion parameters and show that we can attain 40X speedup in terms of time to solution over previous systems. We achieve this by (i) designing the ZionEX platform with dedicated scale-out network, provisioned with high bandwidth, optimal topology and efficient transport (ii) implementing an optimized PyTorch-based training stack supporting both model and data parallelism (iii) develo** sharding algorithms capable of hierarchical partitioning of the embedding tables along row, column dimensions and load balancing them across multiple workers; (iv) adding high-performance core operators while retaining flexibility to support optimizers with fully deterministic updates (v) leveraging reduced precision communications, multi-level memory hierarchy (HBM+DDR+SSD) and pipelining. Furthermore, we develop and briefly comment on distributed data ingestion and other supporting services that are required for the robust and efficient end-to-end training in production environments.
△ Less
Submitted 26 February, 2023; v1 submitted 11 April, 2021;
originally announced April 2021.
