-
Online Matching: A Real-time Bandit System for Large-scale Recommendations
Authors:
Xinyang Yi,
Shao-Chuan Wang,
Ruining He,
Hariharan Chandrasekaran,
Charles Wu,
Lukasz Heldt,
Lichan Hong,
Minmin Chen,
Ed H. Chi
Abstract:
The last decade has witnessed many successes of deep learning-based models for industry-scale recommender systems. These models are typically trained offline in a batch manner. While being effective in capturing users' past interactions with recommendation platforms, batch learning suffers from long model-update latency and is vulnerable to system biases, making it hard to adapt to distribution sh…
▽ More
The last decade has witnessed many successes of deep learning-based models for industry-scale recommender systems. These models are typically trained offline in a batch manner. While being effective in capturing users' past interactions with recommendation platforms, batch learning suffers from long model-update latency and is vulnerable to system biases, making it hard to adapt to distribution shift and explore new items or user interests. Although online learning-based approaches (e.g., multi-armed bandits) have demonstrated promising theoretical results in tackling these challenges, their practical real-time implementation in large-scale recommender systems remains limited. First, the scalability of online approaches in servicing a massive online traffic while ensuring timely updates of bandit parameters poses a significant challenge. Additionally, exploring uncertainty in recommender systems can easily result in unfavorable user experience, highlighting the need for devising intricate strategies that effectively balance the trade-off between exploitation and exploration. In this paper, we introduce Online Matching: a scalable closed-loop bandit system learning from users' direct feedback on items in real time. We present a hybrid "offline + online" approach for constructing this system, accompanied by a comprehensive exposition of the end-to-end system architecture. We propose Diag-LinUCB -- a novel extension of the LinUCB algorithm -- to enable distributed updates of bandits parameter in a scalable and timely manner. We conduct live experiments in YouTube and show that Online Matching is able to enhance the capabilities of fresh content discovery and item exploration in the present platform.
△ Less
Submitted 29 July, 2023;
originally announced July 2023.
-
A New Model to Analyze Power and Communication System Intra-and-Inter Dependencies
Authors:
Sohini Roy,
Harish Chandrasekaran,
Anamitra Pal,
Arunabha Sen
Abstract:
The reliable and resilient operation of the smart grid necessitates a clear understanding of the intra-and-inter dependencies of its power and communication systems. This understanding can only be achieved by accurately depicting the interactions between the different components of these two systems. This paper presents a model, called modified implicative interdependency model (MIIM), for capturi…
▽ More
The reliable and resilient operation of the smart grid necessitates a clear understanding of the intra-and-inter dependencies of its power and communication systems. This understanding can only be achieved by accurately depicting the interactions between the different components of these two systems. This paper presents a model, called modified implicative interdependency model (MIIM), for capturing these interactions. Data obtained from a power utility in the U.S. Southwest is used to ensure the validity of the model. The performance of the model for a specific power system application namely, state estimation, is demonstrated using the IEEE 118-bus system. The results indicate that the proposed model is more accurate than its predecessor, the implicative interdependency model (IIM) [1], in predicting the system state in case of failures in the power and/or communication systems.
△ Less
Submitted 18 January, 2020;
originally announced January 2020.
-
Scalable K-Medoids via True Error Bound and Familywise Bandits
Authors:
Aravindakshan Babu,
Saurabh Agarwal,
Sudarshan Babu,
Hariharan Chandrasekaran
Abstract:
K-Medoids(KM) is a standard clustering method, used extensively on semi-metric data.Error analyses of KM have traditionally used an in-sample notion of error,which can be far from the true error and suffer from generalization gap. We formalize the true K-Medoid error based on the underlying data distribution.We decompose the true error into fundamental statistical problems of: minimum estimation (…
▽ More
K-Medoids(KM) is a standard clustering method, used extensively on semi-metric data.Error analyses of KM have traditionally used an in-sample notion of error,which can be far from the true error and suffer from generalization gap. We formalize the true K-Medoid error based on the underlying data distribution.We decompose the true error into fundamental statistical problems of: minimum estimation (ME) and minimum mean estimation (MME). We provide a convergence result for MME. We show $\errMME$ decreases no slower than $Θ(\frac{1}{n^{\frac{2}{3}}})$, where $n$ is a measure of sample size. Inspired by this bound, we propose a computationally efficient, distributed KM algorithm namely MCPAM. MCPAM has expected runtime $\mathcal{O}(km)$,where $k$ is the number of medoids and $m$ is number of samples. MCPAM provides massive computational savings for a small tradeoff in accuracy. We verify the quality and scaling properties of MCPAM on various datasets. And achieve the hitherto unachieved feat of calculating the KM of 1 billion points on semi-metric spaces.
△ Less
Submitted 29 October, 2019; v1 submitted 27 May, 2019;
originally announced May 2019.
-
Discovery and Rossiter-McLaughlin Effect of Exoplanet Kepler-8b
Authors:
Jon M. Jenkins,
William J. Borucki,
David G. Koch,
Geoffrey W. Marcy,
William D. Cochran,
Gibor Basri,
Natalie M. Batalha,
Lars A. Buchhave,
Tim M. Brown,
Douglas A. Caldwell,
Edward W. Dunham,
Michael Endl,
Debra A. Fischer,
Thomas N. Gautier III,
John C. Geary,
Ronald L. Gilliland,
Steve B. Howell,
Howard Isaacson,
John Asher Johnson,
David W. Latham,
Jack J. Lissauer,
David G. Monet,
Jason F. Rowe,
Dimitar D. Sasselov,
William F. Welsh
, et al. (28 additional authors not shown)
Abstract:
We report the discovery and the Rossiter-McLaughlin effect of Kepler-8b, a transiting planet identified by the NASA Kepler Mission. Kepler photometry and Keck-HIRES radial velocities yield the radius and mass of the planet around this F8IV subgiant host star. The planet has a radius RP = 1.419 RJ and a mass, MP = 0.60 MJ, yielding a density of 0.26 g cm^-3, among the lowest density planets known…
▽ More
We report the discovery and the Rossiter-McLaughlin effect of Kepler-8b, a transiting planet identified by the NASA Kepler Mission. Kepler photometry and Keck-HIRES radial velocities yield the radius and mass of the planet around this F8IV subgiant host star. The planet has a radius RP = 1.419 RJ and a mass, MP = 0.60 MJ, yielding a density of 0.26 g cm^-3, among the lowest density planets known. The orbital period is P = 3.523 days and orbital semima jor axis is 0.0483+0.0006/-0.0012 AU. The star has a large rotational v sin i of 10.5 +/- 0.7 km s^-1 and is relatively faint (V = 13.89 mag), both properties deleterious to precise Doppler measurements. The velocities are indeed noisy, with scatter of 30 m s^-1, but exhibit a period and phase consistent with the planet implied by the photometry. We securely detect the Rossiter-McLaughlin effect, confirming the planet's existence and establishing its orbit as prograde. We measure an inclination between the projected planetary orbital axis and the projected stellar rotation axis of lambda = -26.9 +/- 4.6 deg, indicating a moderate inclination of the planetary orbit. Rossiter-McLaughlin measurements of a large sample of transiting planets from Kepler will provide a statistically robust measure of the true distribution of spin-orbit orientations for hot jupiters in general.
△ Less
Submitted 4 January, 2010;
originally announced January 2010.
-
The Kepler Pixel Response Function
Authors:
Stephen T. Bryson,
Peter Tenenbaum,
Jon M. Jenkins,
Hema Chandrasekaran,
Todd Klaus,
Douglas A. Caldwell,
Ronald L. Gilliland,
Michael R. Haas,
Jessie L. Dotson,
David G. Koch,
William J. Borucki
Abstract:
Kepler seeks to detect sequences of transits of Earth-size exoplanets orbiting Solar-like stars. Such transit signals are on the order of 100 ppm. The high photometric precision demanded by Kepler requires detailed knowledge of how the Kepler pixels respond to starlight during a nominal observation. This information is provided by the Kepler pixel response function (PRF), defined as the composit…
▽ More
Kepler seeks to detect sequences of transits of Earth-size exoplanets orbiting Solar-like stars. Such transit signals are on the order of 100 ppm. The high photometric precision demanded by Kepler requires detailed knowledge of how the Kepler pixels respond to starlight during a nominal observation. This information is provided by the Kepler pixel response function (PRF), defined as the composite of Kepler's optical point spread function, integrated spacecraft pointing jitter during a nominal cadence and other systematic effects. To provide sub-pixel resolution, the PRF is represented as a piecewise-continuous polynomial on a sub-pixel mesh. This continuous representation allows the prediction of a star's flux value on any pixel given the star's pixel position. The advantages and difficulties of this polynomial representation are discussed, including characterization of spatial variation in the PRF and the smoothing of discontinuities between sub-pixel polynomial patches. On-orbit super-resolution measurements of the PRF across the Kepler field of view are described. Two uses of the PRF are presented: the selection of pixels for each star that maximizes the photometric signal to noise ratio for that star, and PRF-fitted centroids which provide robust and accurate stellar positions on the CCD, primarily used for attitude and plate scale tracking. Good knowledge of the PRF has been a critical component for the successful collection of high-precision photometry by Kepler.
△ Less
Submitted 24 March, 2010; v1 submitted 2 January, 2010;
originally announced January 2010.
-
Kepler Mission Design, Realized Photometric Performance, and Early Science
Authors:
David G. Koch,
William J. Borucki,
Gibor Basri,
Natalie M. Batalha,
Timothy M. Brown,
Douglas Caldwell,
Joergen Christensen-Dalsgaard,
William D. Cochran,
Edna DeVore,
Edward W. Dunham,
Thomas N. Gautier III,
John C. Geary,
Ronald L. Gilliland,
Alan Gould,
Jon Jenkins,
Yoji Kondo,
David W. Latham,
Jack J. Lissauer,
Geoffrey Marcy,
David Monet,
Dimitar Sasselov,
Alan Boss,
Donald Brownlee,
John Caldwell,
Andrea K. Dupree
, et al. (22 additional authors not shown)
Abstract:
The Kepler Mission, launched on Mar 6, 2009 was designed with the explicit capability to detect Earth-size planets in the habitable zone of solar-like stars using the transit photometry method. Results from just forty-three days of data along with ground-based follow-up observations have identified five new transiting planets with measurements of their masses, radii, and orbital periods. Many as…
▽ More
The Kepler Mission, launched on Mar 6, 2009 was designed with the explicit capability to detect Earth-size planets in the habitable zone of solar-like stars using the transit photometry method. Results from just forty-three days of data along with ground-based follow-up observations have identified five new transiting planets with measurements of their masses, radii, and orbital periods. Many aspects of stellar astrophysics also benefit from the unique, precise, extended and nearly continuous data set for a large number and variety of stars. Early results for classical variables and eclipsing stars show great promise. To fully understand the methodology, processes and eventually the results from the mission, we present the underlying rationale that ultimately led to the flight and ground system designs used to achieve the exquisite photometric performance. As an example of the initial photometric results, we present variability measurements that can be used to distinguish dwarf stars from red giants.
△ Less
Submitted 26 January, 2010; v1 submitted 1 January, 2010;
originally announced January 2010.
-
Overview of the Kepler Science Processing Pipeline
Authors:
Jon M. Jenkins,
Douglas A. Caldwell,
Hema Chandrasekaran,
Joseph D. Twicken,
Stephen T. Bryson,
Elisa V. Quintana,
Bruce D. Clarke,
Jie Li,
Christopher Allen,
Peter Tenenbaum,
Hayley Wu,
Todd C. Klaus,
Christopher K. Middour,
Miles T. Cote,
Sean McCauliff,
Forrest R. Girouard,
Jay P. Gunter,
Bill Wohler,
Jeneen Sommers,
Jennifer R. Hall,
Kamal Uddin,
Michael S. Wu,
Paresh A. Bhavsar,
Jeffrey Van Cleve,
David L. Pletcher
, et al. (5 additional authors not shown)
Abstract:
The Kepler Mission Science Operations Center (SOC) performs several critical functions including managing the ~156,000 target stars, associated target tables, science data compression tables and parameters, as well as processing the raw photometric data downlinked from the spacecraft each month. The raw data are first calibrated at the pixel level to correct for bias, smear induced by a shutterl…
▽ More
The Kepler Mission Science Operations Center (SOC) performs several critical functions including managing the ~156,000 target stars, associated target tables, science data compression tables and parameters, as well as processing the raw photometric data downlinked from the spacecraft each month. The raw data are first calibrated at the pixel level to correct for bias, smear induced by a shutterless readout, and other detector and electronic effects. A background sky flux is estimated from ~4500 pixels on each of the 84 CCD readout channels, and simple aperture photometry is performed on an optimal aperture for each star. Ancillary engineering data and diagnostic information extracted from the science data are used to remove systematic errors in the flux time series that are correlated with these data prior to searching for signatures of transiting planets with a wavelet-based, adaptive matched filter. Stars with signatures exceeding 7.1 sigma are subjected to a suite of statistical tests including an examination of each star's centroid motion to reject false positives caused by background eclipsing binaries. Physical parameters for each planetary candidate are fitted to the transit signature, and signatures of additional transiting planets are sought in the residual light curve. The pipeline is operational, finding planetary signatures and providing robust eliminations of false positives.
△ Less
Submitted 1 January, 2010;
originally announced January 2010.
-
Initial Characteristics of Kepler Long Cadence Data For Detecting Transiting Planets
Authors:
Jon M. Jenkins,
Douglas A. Caldwell,
Hema Chandrasekaran,
Joseph D. Twicken,
Stephen T. Bryson,
Elisa V. Quintana,
Bruce D. Clarke,
Jie Li,
Christopher Allen,
Peter Tenenbaum,
Hayley Wu,
Todd C. Klaus,
Jeffrey Van Cleve,
Jessie A. Dotson,
Michael R. Haas,
Ronald L. Gilliland,
David G. Koch,
William J. Borucki
Abstract:
The Kepler Mission seeks to detect Earth-size planets transiting solar-like stars in its ~115 deg^2 field of view over the course of its 3.5 year primary mission by monitoring the brightness of each of ~156,000 Long Cadence stellar targets with a time resolution of 29.4 minutes. We discuss the photometric precision achieved on timescales relevant to transit detection for data obtained in the 33.…
▽ More
The Kepler Mission seeks to detect Earth-size planets transiting solar-like stars in its ~115 deg^2 field of view over the course of its 3.5 year primary mission by monitoring the brightness of each of ~156,000 Long Cadence stellar targets with a time resolution of 29.4 minutes. We discuss the photometric precision achieved on timescales relevant to transit detection for data obtained in the 33.5-day long Quarter 1 (Q1) observations that ended 2009 June 15. The lower envelope of the photometric precision obtained at various timescales is consistent with expected random noise sources, indicating that Kepler has the capability to fulfill its mission. The Kepler light curves exhibit high precision over a large dynamic range, which will surely permit their use for a large variety of investigations in addition to finding and characterizing planets. We discuss the temporal characteristics of both the raw flux time series and the systematic error-corrected flux time series produced by the Kepler Science Pipeline, and give examples illustrating Kepler's large dynamic range and the variety of light curves obtained from the Q1 observations.
△ Less
Submitted 1 January, 2010;
originally announced January 2010.
-
Instrument Performance in Kepler's First Months
Authors:
Douglas A. Caldwell,
Jeffery J. Kolodziejczak,
Jeffrey E. Van Cleve,
Jon M. Jenkins,
Paul R. Gazis,
Vic S. Argabright,
Eric E. Bachtell,
Edward W. Dunham,
John C. Geary,
Ronald L. Gilliland,
Hema Chandrasekaran,
Jie Li,
Peter Tenenbaum,
Hayley Wu,
William J. Borucki,
Stephen T. Bryson,
Jessie L. Dotson,
Michael R. Haas,
David G. Koch
Abstract:
The Kepler Mission relies on precise differential photometry to detect the 80 parts per million (ppm) signal from an Earth-Sun equivalent transit. Such precision requires superb instrument stability on time scales up to ~2 days and systematic error removal to better than 20 ppm. To this end, the spacecraft and photometer underwent 67 days of commissioning, which included several data sets taken…
▽ More
The Kepler Mission relies on precise differential photometry to detect the 80 parts per million (ppm) signal from an Earth-Sun equivalent transit. Such precision requires superb instrument stability on time scales up to ~2 days and systematic error removal to better than 20 ppm. To this end, the spacecraft and photometer underwent 67 days of commissioning, which included several data sets taken to characterize the photometer performance. Because Kepler has no shutter, we took a series of dark images prior to the dust cover ejection, from which we measured the bias levels, dark current, and read noise. These basic detector properties are essentially unchanged from ground-based tests, indicating that the photometer is working as expected. Several image artifacts have proven more complex than when observed during ground testing, as a result of their interactions with starlight and the greater thermal stability in flight, which causes the temperature-dependent artifact variations to be on the timescales of transits. Because of Kepler's unprecedented sensitivity and stability, we have also seen several unexpected systematics that affect photometric precision. We are using the first 43 days of science data to characterize these effects and to develop detection and mitigation methods that will be implemented in the calibration pipeline. Based on early testing, we expect to attain Kepler's planned photometric precision over 80%-90% of the field of view.
△ Less
Submitted 1 January, 2010;
originally announced January 2010.
