-
Pareto optimal proxy metrics
Authors:
Lee Richardson,
Alessandro Zito,
Dylan Greaves,
Jacopo Soriano
Abstract:
North star metrics and online experimentation play a central role in how technology companies improve their products. In many practical settings, however, evaluating experiments based on the north star metric directly can be difficult. The two most significant issues are 1) low sensitivity of the north star metric and 2) differences between the short-term and long-term impact on the north star met…
▽ More
North star metrics and online experimentation play a central role in how technology companies improve their products. In many practical settings, however, evaluating experiments based on the north star metric directly can be difficult. The two most significant issues are 1) low sensitivity of the north star metric and 2) differences between the short-term and long-term impact on the north star metric. A common solution is to rely on proxy metrics rather than the north star in experiment evaluation and launch decisions. Existing literature on proxy metrics concentrates mainly on the estimation of the long-term impact from short-term experimental data. In this paper, instead, we focus on the trade-off between the estimation of the long-term impact and the sensitivity in the short term. In particular, we propose the Pareto optimal proxy metrics method, which simultaneously optimizes prediction accuracy and sensitivity. In addition, we give an efficient multi-objective optimization algorithm that outperforms standard methods. We applied our methodology to experiments from a large industrial recommendation system, and found proxy metrics that are eight times more sensitive than the north star and consistently moved in the same direction, increasing the velocity and the quality of the decisions to launch new features.
△ Less
Submitted 3 July, 2023;
originally announced July 2023.
-
Fresh Content Needs More Attention: Multi-funnel Fresh Content Recommendation
Authors:
Jianling Wang,
Haokai Lu,
Sai zhang,
Bart Locanthi,
Haoting Wang,
Dylan Greaves,
Benjamin Lipshitz,
Sriraj Badam,
Ed H. Chi,
Cristos Goodrow,
Su-Lin Wu,
Lexi Baugher,
Minmin Chen
Abstract:
Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platfo…
▽ More
Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platform. To nominate fresh contents, we built a multi-funnel nomination system that combines (i) a two-tower model with strong generalization power for coverage, and (ii) a sequence model with near real-time update on user feedback for relevance. The multi-funnel setup effectively balances between coverage and relevance. An in-depth study uncovers the relationship between user activity level and their proximity toward fresh contents, which further motivates a contextual multi-funnel setup. Nominated fresh candidates are then scored and ranked by systems considering prediction uncertainty to further bootstrap content with less exposure. We evaluate the benefits of the dedicated fresh content recommendation stack, and the multi-funnel nomination system in particular, through user corpus co-diverted live experiments. We conduct multiple rounds of live experiments on a commercial platform serving billion of users demonstrating efficacy of our proposed methods.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
Research Note: An Open Source Bluespec Compiler
Authors:
David J. Greaves
Abstract:
In this Research Note we report on an open-source compiler for the Bluespec hardware description language.
In this Research Note we report on an open-source compiler for the Bluespec hardware description language.
△ Less
Submitted 8 May, 2019;
originally announced May 2019.
-
FRAMER: A Software-based Capability Model
Authors:
Myoung ** Nam,
Periklis Akritidis,
David J Greaves
Abstract:
Fine-grained memory protection for C and C++ programs must track individual objects (or pointers), and store bounds information per object (pointer). Its cost is dominated by metadata updates and lookups, making efficient metadata management the key for minimizing performance impact. Existing approaches reduce metadata management overheads by sacrificing precision, breaking binary compatibility by…
▽ More
Fine-grained memory protection for C and C++ programs must track individual objects (or pointers), and store bounds information per object (pointer). Its cost is dominated by metadata updates and lookups, making efficient metadata management the key for minimizing performance impact. Existing approaches reduce metadata management overheads by sacrificing precision, breaking binary compatibility by changing object memory layout, or wasting space by excessive alignment or large shadow memory spaces. We propose FRAMER, a software capability model for object-granularity memory protection. Its efficient per-object metadata management mechanism enables direct access to metadata by calculating their location from a tagged pointer to the object and, for large objects, a compact supplementary table. The number of bits in this tag and the size of the supplementary table are balanced to minimize both using a novel technique. FRAMER is a general proposal for object metadata management with potential applications in memory safety, type safety, thread safety and garbage collection that improves over previous solutions by (1) increasing locality of reference by having objects carry their metadata, (2) streamlining expensive metadata lookups, (3) saving space by avoiding superfluous alignment and padding, (4) avoiding internal object memory layout changes.
△ Less
Submitted 1 March, 2019; v1 submitted 27 October, 2018;
originally announced October 2018.
-
Extending programs with debug-related features, with application to hardware development
Authors:
Nik Sultana,
Salvator Galea,
David Greaves,
Marcin Wojcik,
Noa Zilberman,
Richard Clegg,
Luo Mai,
Richard Mortier,
Peter Pietzuch,
Jon Crowcroft,
Andrew W Moore
Abstract:
The capacity and programmability of reconfigurable hardware such as FPGAs has improved steadily over the years, but they do not readily provide any mechanisms for monitoring or debugging running programs. Such mechanisms need to be written into the program itself. This is done using ad hoc methods and primitive tools when compared to CPU programming. This complicates the programming and debugging…
▽ More
The capacity and programmability of reconfigurable hardware such as FPGAs has improved steadily over the years, but they do not readily provide any mechanisms for monitoring or debugging running programs. Such mechanisms need to be written into the program itself. This is done using ad hoc methods and primitive tools when compared to CPU programming. This complicates the programming and debugging of reconfigurable hardware. We introduce Program-hosted Directability (PhD), the extension of programs to interpret direction commands at runtime to enable debugging, monitoring and profiling. Normally in hardware development such features are fixed at compile time. We present a language of directing commands, specify its semantics in terms of a simple controller that is embedded with programs, and implement a prototype for directing network programs running in hardware. We show that this approach affords significant flexibility with low impact on hardware utilisation and performance.
△ Less
Submitted 28 May, 2017;
originally announced May 2017.
-
Non-contact hemodynamic imaging reveals the jugular venous pulse waveform
Authors:
Robert Amelard,
Richard L Hughson,
Danielle K Greaves,
Kaylen J Pfisterer,
Jason Leung,
David A Clausi,
Alexander Wong
Abstract:
Cardiovascular monitoring is important to prevent diseases from progressing. The jugular venous pulse (JVP) waveform offers important clinical information about cardiac health, but is not routinely examined due to its invasive catheterisation procedure. Here, we demonstrate for the first time that the JVP can be consistently observed in a non-contact manner using a novel light-based photoplethysmo…
▽ More
Cardiovascular monitoring is important to prevent diseases from progressing. The jugular venous pulse (JVP) waveform offers important clinical information about cardiac health, but is not routinely examined due to its invasive catheterisation procedure. Here, we demonstrate for the first time that the JVP can be consistently observed in a non-contact manner using a novel light-based photoplethysmographic imaging system, coded hemodynamic imaging (CHI). While traditional monitoring methods measure the JVP at a single location, CHI's wide-field imaging capabilities were able to observe the jugular venous pulse's spatial flow profile for the first time. The important inflection points in the JVP were observed, meaning that cardiac abnormalities can be assessed through JVP distortions. CHI provides a new way to assess cardiac health through non-contact light-based JVP monitoring, and can be used in non-surgical environments for cardiac assessment.
△ Less
Submitted 21 April, 2016; v1 submitted 15 April, 2016;
originally announced April 2016.
