-
Fresh Content Needs More Attention: Multi-funnel Fresh Content Recommendation
Authors:
Jianling Wang,
Haokai Lu,
Sai zhang,
Bart Locanthi,
Haoting Wang,
Dylan Greaves,
Benjamin Lipshitz,
Sriraj Badam,
Ed H. Chi,
Cristos Goodrow,
Su-Lin Wu,
Lexi Baugher,
Minmin Chen
Abstract:
Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platfo…
▽ More
Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platform. To nominate fresh contents, we built a multi-funnel nomination system that combines (i) a two-tower model with strong generalization power for coverage, and (ii) a sequence model with near real-time update on user feedback for relevance. The multi-funnel setup effectively balances between coverage and relevance. An in-depth study uncovers the relationship between user activity level and their proximity toward fresh contents, which further motivates a contextual multi-funnel setup. Nominated fresh candidates are then scored and ranked by systems considering prediction uncertainty to further bootstrap content with less exposure. We evaluate the benefits of the dedicated fresh content recommendation stack, and the multi-funnel nomination system in particular, through user corpus co-diverted live experiments. We conduct multiple rounds of live experiments on a commercial platform serving billion of users demonstrating efficacy of our proposed methods.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
Long-Term Value of Exploration: Measurements, Findings and Algorithms
Authors:
Yi Su,
Xiangyu Wang,
Elaine Ya Le,
Liang Liu,
Yuening Li,
Haokai Lu,
Benjamin Lipshitz,
Sriraj Badam,
Lukasz Heldt,
Shuchao Bi,
Ed Chi,
Cristos Goodrow,
Su-Lin Wu,
Lexi Baugher,
Minmin Chen
Abstract:
Effective exploration is believed to positively influence the long-term user experience on recommendation platforms. Determining its exact benefits, however, has been challenging. Regular A/B tests on exploration often measure neutral or even negative engagement metrics while failing to capture its long-term benefits. We here introduce new experiment designs to formally quantify the long-term valu…
▽ More
Effective exploration is believed to positively influence the long-term user experience on recommendation platforms. Determining its exact benefits, however, has been challenging. Regular A/B tests on exploration often measure neutral or even negative engagement metrics while failing to capture its long-term benefits. We here introduce new experiment designs to formally quantify the long-term value of exploration by examining its effects on content corpus, and connecting content corpus growth to the long-term user experience from real-world experiments. Once established the values of exploration, we investigate the Neural Linear Bandit algorithm as a general framework to introduce exploration into any deep learning based ranking systems. We conduct live experiments on one of the largest short-form video recommendation platforms that serves billions of users to validate the new experiment designs, quantify the long-term values of exploration, and to verify the effectiveness of the adopted neural linear bandit algorithm for exploration.
△ Less
Submitted 25 February, 2024; v1 submitted 12 May, 2023;
originally announced May 2023.
-
Fairness in Recommendation Ranking through Pairwise Comparisons
Authors:
Alex Beutel,
Jilin Chen,
Tulsee Doshi,
Hai Qian,
Li Wei,
Yi Wu,
Lukasz Heldt,
Zhe Zhao,
Lichan Hong,
Ed H. Chi,
Cristos Goodrow
Abstract:
Recommender systems are one of the most pervasive applications of machine learning in industry, with many services using them to match users to products or information. As such it is important to ask: what are the possible fairness risks, how can we quantify them, and how should we address them? In this paper we offer a set of novel metrics for evaluating algorithmic fairness concerns in recommend…
▽ More
Recommender systems are one of the most pervasive applications of machine learning in industry, with many services using them to match users to products or information. As such it is important to ask: what are the possible fairness risks, how can we quantify them, and how should we address them? In this paper we offer a set of novel metrics for evaluating algorithmic fairness concerns in recommender systems. In particular we show how measuring fairness based on pairwise comparisons from randomized experiments provides a tractable means to reason about fairness in rankings from recommender systems. Building on this metric, we offer a new regularizer to encourage improving this metric during model training and thus improve fairness in the resulting rankings. We apply this pairwise regularization to a large-scale, production recommender system and show that we are able to significantly improve the system's pairwise fairness.
△ Less
Submitted 2 March, 2019;
originally announced March 2019.
