-
LLMs for User Interest Exploration in Large-scale Recommendation Systems
Authors:
Jianling Wang,
Haokai Lu,
Yifan Liu,
He Ma,
Yueqi Wang,
Yang Gu,
Shuzhou Zhang,
Ningren Han,
Shuchao Bi,
Lexi Baugher,
Ed Chi,
Minmin Chen
Abstract:
Traditional recommendation systems are subject to a strong feedback loop by learning from and reinforcing past user-item interactions, which in turn limits the discovery of novel user interests. To address this, we introduce a hybrid hierarchical framework combining Large Language Models (LLMs) and classic recommendation models for user interest exploration. The framework controls the interfacing…
▽ More
Traditional recommendation systems are subject to a strong feedback loop by learning from and reinforcing past user-item interactions, which in turn limits the discovery of novel user interests. To address this, we introduce a hybrid hierarchical framework combining Large Language Models (LLMs) and classic recommendation models for user interest exploration. The framework controls the interfacing between the LLMs and the classic recommendation models through "interest clusters", the granularity of which can be explicitly determined by algorithm designers. It recommends the next novel interests by first representing "interest clusters" using language, and employs a fine-tuned LLM to generate novel interest descriptions that are strictly within these predefined clusters. At the low level, it grounds these generated interests to an item-level policy by restricting classic recommendation models, in this case a transformer-based sequence recommender to return items that fall within the novel clusters generated at the high level. We showcase the efficacy of this approach on an industrial-scale commercial platform serving billions of users. Live experiments show a significant increase in both exploration of novel interests and overall user enjoyment of the platform.
△ Less
Submitted 7 June, 2024; v1 submitted 25 May, 2024;
originally announced May 2024.
-
Fresh Content Needs More Attention: Multi-funnel Fresh Content Recommendation
Authors:
Jianling Wang,
Haokai Lu,
Sai zhang,
Bart Locanthi,
Haoting Wang,
Dylan Greaves,
Benjamin Lipshitz,
Sriraj Badam,
Ed H. Chi,
Cristos Goodrow,
Su-Lin Wu,
Lexi Baugher,
Minmin Chen
Abstract:
Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platfo…
▽ More
Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platform. To nominate fresh contents, we built a multi-funnel nomination system that combines (i) a two-tower model with strong generalization power for coverage, and (ii) a sequence model with near real-time update on user feedback for relevance. The multi-funnel setup effectively balances between coverage and relevance. An in-depth study uncovers the relationship between user activity level and their proximity toward fresh contents, which further motivates a contextual multi-funnel setup. Nominated fresh candidates are then scored and ranked by systems considering prediction uncertainty to further bootstrap content with less exposure. We evaluate the benefits of the dedicated fresh content recommendation stack, and the multi-funnel nomination system in particular, through user corpus co-diverted live experiments. We conduct multiple rounds of live experiments on a commercial platform serving billion of users demonstrating efficacy of our proposed methods.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
Long-Term Value of Exploration: Measurements, Findings and Algorithms
Authors:
Yi Su,
Xiangyu Wang,
Elaine Ya Le,
Liang Liu,
Yuening Li,
Haokai Lu,
Benjamin Lipshitz,
Sriraj Badam,
Lukasz Heldt,
Shuchao Bi,
Ed Chi,
Cristos Goodrow,
Su-Lin Wu,
Lexi Baugher,
Minmin Chen
Abstract:
Effective exploration is believed to positively influence the long-term user experience on recommendation platforms. Determining its exact benefits, however, has been challenging. Regular A/B tests on exploration often measure neutral or even negative engagement metrics while failing to capture its long-term benefits. We here introduce new experiment designs to formally quantify the long-term valu…
▽ More
Effective exploration is believed to positively influence the long-term user experience on recommendation platforms. Determining its exact benefits, however, has been challenging. Regular A/B tests on exploration often measure neutral or even negative engagement metrics while failing to capture its long-term benefits. We here introduce new experiment designs to formally quantify the long-term value of exploration by examining its effects on content corpus, and connecting content corpus growth to the long-term user experience from real-world experiments. Once established the values of exploration, we investigate the Neural Linear Bandit algorithm as a general framework to introduce exploration into any deep learning based ranking systems. We conduct live experiments on one of the largest short-form video recommendation platforms that serves billions of users to validate the new experiment designs, quantify the long-term values of exploration, and to verify the effectiveness of the adopted neural linear bandit algorithm for exploration.
△ Less
Submitted 25 February, 2024; v1 submitted 12 May, 2023;
originally announced May 2023.
-
Small-N collisional dynamics IV: Order in the realm of not-so-small-N
Authors:
Nathan W. C. Leigh,
Aaron M. Geller,
Michael M. Shara,
Lukas Baugher,
Vianny Hierro,
De'Andre Ferreira,
Elizabeth Teperino
Abstract:
In this paper, the fourth in the series, we continue our study of combinatorics in chaotic Newtonian dynamics. We focus once again on the chaotic four-body problem in Newtonian gravity assuming finite-sized particles, and interactions that produce direct collisions between any two particles. Our long-term goal is to predict the probability of a given collision event occurring over the course of an…
▽ More
In this paper, the fourth in the series, we continue our study of combinatorics in chaotic Newtonian dynamics. We focus once again on the chaotic four-body problem in Newtonian gravity assuming finite-sized particles, and interactions that produce direct collisions between any two particles. Our long-term goal is to predict the probability of a given collision event occurring over the course of an interaction, as a function of the numbers and properties of the particles. In previous papers, we varied the number of interacting particles, as well as the distributions of particle radii and masses. Here, we refine the methods developed in these preceding studies, and arrive at a final and robust methodology that can be used to study collisional dynamics in a variety of astrophysical contexts, ranging from stars in star clusters, galaxies in galaxy groups and clusters and even the collisional growth of planetesimals in protoplanetary disks. We further present and refine the concept of a Collision Rate Diagram (CRD), the primary tool we use to quantify the relative rates for different collision scenarios to occur. The agreement between our final theoretical model and the results of numerical scattering simulations is excellent.
△ Less
Submitted 27 July, 2018;
originally announced July 2018.
