Wukong: Towards a Scaling Law for Large-Scale Recommendation
Authors:
Buyun Zhang,
Liang Luo,
Yuxin Chen,
Jade Nie,
Xi Liu,
Daifeng Guo,
Yanli Zhao,
Shen Li,
Yuchen Hao,
Yantao Yao,
Guna Lakshminarayanan,
Ellie Dingqiao Wen,
Jongsoo Park,
Maxim Naumov,
Wenlin Chen
Abstract:
Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets.…
▽ More
Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 GFLOP/example, where prior arts fall short.
△ Less
Submitted 4 June, 2024; v1 submitted 4 March, 2024;
originally announced March 2024.
Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation
Authors:
Liang Luo,
Buyun Zhang,
Michael Tsang,
Yinbin Ma,
Ching-Hsiang Chu,
Yuxin Chen,
Shen Li,
Yuchen Hao,
Yanli Zhao,
Guna Lakshminarayanan,
Ellie Dingqiao Wen,
Jongsoo Park,
Dheevatsa Mudigere,
Maxim Naumov
Abstract:
We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global…
▽ More
We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.
△ Less
Submitted 2 May, 2024; v1 submitted 1 March, 2024;
originally announced March 2024.
