AutoML for Large Capacity Modeling of Meta's Ranking Systems
Authors:
Hang Yin,
Kuang-Hung Liu,
Mengying Sun,
Yuxin Chen,
Buyun Zhang,
Jiang Liu,
Vivek Sehgal,
Rudresh Rajnikant Panchal,
Eugen Hotaj,
Xi Liu,
Daifeng Guo,
Jamey Zhang,
Zhou Wang,
Shali Jiang,
Huayu Li,
Zhengxing Chen,
Wen-Yen Chen,
Jiyan Yang,
Wei Wen
Abstract:
Web-scale ranking systems at Meta serving billions of users is complex. Improving ranking models is essential but engineering heavy. Automated Machine Learning (AutoML) can release engineers from labor intensive work of tuning ranking models; however, it is unknown if AutoML is efficient enough to meet tight production timeline in real-world and, at the same time, bring additional improvements to…
▽ More
Web-scale ranking systems at Meta serving billions of users is complex. Improving ranking models is essential but engineering heavy. Automated Machine Learning (AutoML) can release engineers from labor intensive work of tuning ranking models; however, it is unknown if AutoML is efficient enough to meet tight production timeline in real-world and, at the same time, bring additional improvements to the strong baselines. Moreover, to achieve higher ranking performance, there is an ever-increasing demand to scale up ranking models to even larger capacity, which imposes more challenges on the efficiency. The large scale of models and tight production schedule requires AutoML to outperform human baselines by only using a small number of model evaluation trials (around 100). We presents a sampling-based AutoML method, focusing on neural architecture search and hyperparameter optimization, addressing these challenges in Meta-scale production when building large capacity models. Our approach efficiently handles large-scale data demands. It leverages a lightweight predictor-based searcher and reinforcement learning to explore vast search spaces, significantly reducing the number of model evaluations. Through experiments in large capacity modeling for CTR and CVR applications, we show that our method achieves outstanding Return on Investment (ROI) versus human tuned baselines, with up to 0.09% Normalized Entropy (NE) loss reduction or $25\%$ Query per Second (QPS) increase by only sampling one hundred models on average from a curated search space. The proposed AutoML method has already made real-world impact where a discovered Instagram CTR model with up to -0.36% NE gain (over existing production baseline) was selected for large-scale online A/B test and show statistically significant gain. These production results proved AutoML efficacy and accelerated its adoption in ranking systems at Meta.
△ Less
Submitted 16 November, 2023; v1 submitted 13 November, 2023;
originally announced November 2023.
AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles
Authors:
Charles Weill,
Javier Gonzalvo,
Vitaly Kuznetsov,
Scott Yang,
Scott Yak,
Hanna Mazzawi,
Eugen Hotaj,
Ghassen Jerfel,
Vladimir Macko,
Ben Adlam,
Mehryar Mohri,
Corinna Cortes
Abstract:
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspired by the AdaNet algorithm (Cortes et al., 2017) which learns the structure of a neural network as an ensemble of subnetworks. We designed it to: (1) integrate with the existing TensorFlow ecosystem, (2) offer sensible de…
▽ More
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspired by the AdaNet algorithm (Cortes et al., 2017) which learns the structure of a neural network as an ensemble of subnetworks. We designed it to: (1) integrate with the existing TensorFlow ecosystem, (2) offer sensible default search spaces to perform well on novel datasets, (3) present a flexible API to utilize expert information when available, and (4) efficiently accelerate training with distributed CPU, GPU, and TPU hardware. The code is open-source and available at: https://github.com/tensorflow/adanet.
△ Less
Submitted 30 April, 2019;
originally announced May 2019.
