-
Time-based Sequence Model for Personalization and Recommendation Systems
Authors:
Tigran Ishkhanov,
Maxim Naumov,
Xianjie Chen,
Yan Zhu,
Yuan Zhong,
Alisson Gusatti Azzolini,
Chonglin Sun,
Frank Jiang,
Andrey Malevich,
Liang Xiong
Abstract:
In this paper we develop a novel recommendation model that explicitly incorporates time information. The model relies on an embedding layer and TSL attention-like mechanism with inner products in different vector spaces, that can be thought of as a modification of multi-headed attention. This mechanism allows the model to efficiently treat sequences of user behavior of different length. We study t…
▽ More
In this paper we develop a novel recommendation model that explicitly incorporates time information. The model relies on an embedding layer and TSL attention-like mechanism with inner products in different vector spaces, that can be thought of as a modification of multi-headed attention. This mechanism allows the model to efficiently treat sequences of user behavior of different length. We study the properties of our state-of-the-art model on statistically designed data set. Also, we show that it outperforms more complex models with longer sequence length on the Taobao User Behavior dataset.
△ Less
Submitted 27 August, 2020;
originally announced August 2020.
-
ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
Authors:
Qinqing Zheng,
Bor-Yiing Su,
Jiyan Yang,
Alisson Azzolini,
Qiang Wu,
Ou **,
Shri Karandikar,
Hagay Lupesko,
Liang Xiong,
Eric Zhou
Abstract:
Recommendation systems are often trained with a tremendous amount of data, and distributed training is the workhorse to shorten the training time. While the training throughput can be increased by simply adding more workers, it is also increasingly challenging to preserve the model quality. In this paper, we present \shadowsync, a distributed framework specifically tailored to modern scale recomme…
▽ More
Recommendation systems are often trained with a tremendous amount of data, and distributed training is the workhorse to shorten the training time. While the training throughput can be increased by simply adding more workers, it is also increasingly challenging to preserve the model quality. In this paper, we present \shadowsync, a distributed framework specifically tailored to modern scale recommendation system training. In contrast to previous works where synchronization happens as part of the training process, \shadowsync separates the synchronization from training and runs it in the background. Such isolation significantly reduces the synchronization overhead and increases the synchronization frequency, so that we are able to obtain both high throughput and excellent model quality when training at scale. The superiority of our procedure is confirmed by experiments on training deep neural networks for click-through-rate prediction tasks. Our framework is capable to express data parallelism and/or model parallelism, generic to host various types of synchronization algorithms, and readily applicable to large scale problems in other areas.
△ Less
Submitted 23 February, 2021; v1 submitted 6 March, 2020;
originally announced March 2020.
-
Deep Learning Recommendation Model for Personalization and Recommendation Systems
Authors:
Maxim Naumov,
Dheevatsa Mudigere,
Hao-Jun Michael Shi,
Jianyu Huang,
Narayanan Sundaraman,
Jongsoo Park,
Xiaodong Wang,
Udit Gupta,
Carole-Jean Wu,
Alisson G. Azzolini,
Dmytro Dzhulgakov,
Andrey Mallevich,
Ilia Cherniavskii,
Yinghai Lu,
Raghuraman Krishnamoorthi,
Ansha Yu,
Volodymyr Kondratenko,
Stephanie Pereira,
Xianjie Chen,
Wenlin Chen,
Vijay Rao,
Bill Jia,
Liang Xiong,
Misha Smelyanskiy
Abstract:
With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation m…
▽ More
With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.
△ Less
Submitted 31 May, 2019;
originally announced June 2019.