-
Vision-Language Models as a Source of Rewards
Authors:
Kate Baumli,
Satinder Baveja,
Feryal Behbahani,
Harris Chan,
Gheorghe Comanici,
Sebastian Flennerhag,
Maxime Gazeau,
Kristian Holsheimer,
Dan Horgan,
Michael Laskin,
Clare Lyle,
Hussain Masoom,
Kay McKinney,
Volodymyr Mnih,
Alexander Neitz,
Fabio Pardo,
Jack Parker-Holder,
John Quan,
Tim Rocktäschel,
Himanshu Sahni,
Tom Schaul,
Yannick Schroecker,
Stephen Spencer,
Richie Steigerwald,
Luyu Wang
, et al. (1 additional authors not shown)
Abstract:
Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been the need for a large number of reward functions for achieving different goals. We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of…
▽ More
Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been the need for a large number of reward functions for achieving different goals. We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals. We showcase this approach in two distinct visual domains and present a scaling trend showing how larger VLMs lead to more accurate rewards for visual goal achievement, which in turn produces more capable RL agents.
△ Less
Submitted 21 February, 2024; v1 submitted 14 December, 2023;
originally announced December 2023.
-
Uncertainty Quantification for Local Model Explanations Without Model Access
Authors:
Surin Ahn,
Justin Grana,
Yafet Tamene,
Kristian Holsheimer
Abstract:
We present a model-agnostic algorithm for generating post-hoc explanations and uncertainty intervals for a machine learning model when only a static sample of inputs and outputs from the model is available, rather than direct access to the model itself. This situation may arise when model evaluations are expensive; when privacy, security and bandwidth constraints are imposed; or when there is a ne…
▽ More
We present a model-agnostic algorithm for generating post-hoc explanations and uncertainty intervals for a machine learning model when only a static sample of inputs and outputs from the model is available, rather than direct access to the model itself. This situation may arise when model evaluations are expensive; when privacy, security and bandwidth constraints are imposed; or when there is a need for real-time, on-device explanations. Our algorithm uses a bootstrap** approach to quantify the uncertainty that inevitably arises when generating explanations from a finite sample of model queries. Through a simulation study, we show that the uncertainty intervals generated by our algorithm exhibit a favorable trade-off between interval width and coverage probability compared to the naive confidence intervals from classical regression analysis as well as current Bayesian approaches for quantifying explanation uncertainty. We further demonstrate the capabilities of our method by applying it to black-box models, including a deep neural network, trained on three real-world datasets.
△ Less
Submitted 24 June, 2023; v1 submitted 13 January, 2023;
originally announced January 2023.
-
On the Marginally Relevant Operator in z=2 Lifshitz Holography
Authors:
Kristian Holsheimer
Abstract:
We study holographic renormalization and RG flow in a strongly-coupled Lifshitz-type theory in 2+1 dimensions with dynamical exponent z=2. The bottom-up gravity dual we use is 3+1 dimensional Einstein gravity coupled to a massive vector field. This model contains a marginally relevant operator around the Lifshitz fixed point. We show how holographic renormalization works in the presence of this ma…
▽ More
We study holographic renormalization and RG flow in a strongly-coupled Lifshitz-type theory in 2+1 dimensions with dynamical exponent z=2. The bottom-up gravity dual we use is 3+1 dimensional Einstein gravity coupled to a massive vector field. This model contains a marginally relevant operator around the Lifshitz fixed point. We show how holographic renormalization works in the presence of this marginally relevant operator without the need to introduce explicitly cutoff-dependent counterterms. A simple closed-form expression is found for the renormalized on-shell action. We also discuss how asymptotically Lifshitz geometries flow to AdS in the interior due to the marginally relevant operator. We study the behavior of the renormalized entanglement entropy and confirm that it decreases monotonically along the Lifshitz-to-AdS RG flow.
△ Less
Submitted 6 January, 2014; v1 submitted 18 November, 2013;
originally announced November 2013.
-
Anomalous Breaking of Anisotropic Scaling Symmetry in the Quantum Lifshitz Model
Authors:
Marco Baggio,
Jan de Boer,
Kristian Holsheimer
Abstract:
In this note we investigate the anomalous breaking of anisotropic scaling symmetry in a non-relativistic field theory with dynamical exponent z=2. On general grounds, one can show that there exist two possible "central charges" which characterize the breaking of scale invariance. Using heat kernel methods, we compute these two central charges in the quantum Lifshitz model, a free field theory whic…
▽ More
In this note we investigate the anomalous breaking of anisotropic scaling symmetry in a non-relativistic field theory with dynamical exponent z=2. On general grounds, one can show that there exist two possible "central charges" which characterize the breaking of scale invariance. Using heat kernel methods, we compute these two central charges in the quantum Lifshitz model, a free field theory which is second order in time and fourth order in spatial derivatives. We find that one of the two central charges vanishes. Interestingly, this is also true for strongly coupled non-relativistic field theories with a geometric dual described by a metric and a massive vector field.
△ Less
Submitted 12 September, 2012; v1 submitted 29 December, 2011;
originally announced December 2011.
-
Hamilton-Jacobi Renormalization for Lifshitz Spacetime
Authors:
Marco Baggio,
Jan de Boer,
Kristian Holsheimer
Abstract:
Just like AdS spacetimes, Lifshitz spacetimes require counterterms in order to make the on-shell value of the bulk action finite. We study these counterterms using the Hamilton-Jacobi method. Rather than imposing boundary conditions from the start, we will derive suitable boundary conditions by requiring that divergences can be canceled using only local counterterms. We will demonstrate in example…
▽ More
Just like AdS spacetimes, Lifshitz spacetimes require counterterms in order to make the on-shell value of the bulk action finite. We study these counterterms using the Hamilton-Jacobi method. Rather than imposing boundary conditions from the start, we will derive suitable boundary conditions by requiring that divergences can be canceled using only local counterterms. We will demonstrate in examples that this procedure indeed leads to a finite bulk action while at the same time it determines the asymptotic behavior of the fields. This puts more substance to the belief that Lifshitz spacetimes are dual to well-behaved field theories. As a byproduct, we will find the analogue of the conformal anomaly for Lifshitz spacetimes.
△ Less
Submitted 12 September, 2012; v1 submitted 27 July, 2011;
originally announced July 2011.
