-
Explaining CLIP's performance disparities on data from blind/low vision users
Authors:
Daniela Massiceti,
Camilla Longden,
Agnieszka Słowik,
Samuel Wills,
Martin Grayson,
Cecily Morrison
Abstract:
Large multi-modal models (LMMs) hold the potential to usher in a new era of automated visual assistance for people who are blind or low vision (BLV). Yet, these models have not been systematically evaluated on data captured by BLV users. We address this by empirically assessing CLIP, a widely-used LMM likely to underpin many assistive technologies. Testing 25 CLIP variants in a zero-shot classific…
▽ More
Large multi-modal models (LMMs) hold the potential to usher in a new era of automated visual assistance for people who are blind or low vision (BLV). Yet, these models have not been systematically evaluated on data captured by BLV users. We address this by empirically assessing CLIP, a widely-used LMM likely to underpin many assistive technologies. Testing 25 CLIP variants in a zero-shot classification task, we find that their accuracy is 15 percentage points lower on average for images captured by BLV users than web-crawled images. This disparity stems from CLIP's sensitivities to 1) image content (e.g. not recognizing disability objects as well as other objects); 2) image quality (e.g. not being robust to lighting variation); and 3) text content (e.g. not recognizing objects described by tactile adjectives as well as visual ones). We delve deeper with a textual analysis of three common pre-training datasets: LAION-400M, LAION-2B and DataComp-1B, showing that disability content is rarely mentioned. We then provide three examples that illustrate how the performance disparities extend to three downstream models underpinned by CLIP: OWL-ViT, CLIPSeg and DALL-E2. We find that few-shot learning with as few as 5 images can mitigate CLIP's quality-of-service disparities for BLV users in some scenarios, which we discuss alongside a set of other possible mitigations.
△ Less
Submitted 25 March, 2024; v1 submitted 28 November, 2023;
originally announced November 2023.
-
Contextual HyperNetworks for Novel Feature Adaptation
Authors:
Angus Lamb,
Evgeny Saveliev,
Yingzhen Li,
Sebastian Tschiatschek,
Camilla Longden,
Simon Woodhead,
José Miguel Hernández-Lobato,
Richard E. Turner,
Pashmina Cameron,
Cheng Zhang
Abstract:
While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new output features remains a challenge, as neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe in online learning settings, where new output features, such as items in a recommender system, are added c…
▽ More
While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new output features remains a challenge, as neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe in online learning settings, where new output features, such as items in a recommender system, are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), an auxiliary model which generates parameters for extending the base model to a new feature, by utilizing both existing data as well as any observations and/or metadata associated with the new feature. At prediction time, the CHN requires only a single forward pass through a neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches.
To assess the performance of CHNs, we use a CHN to augment a partial variational autoencoder (P-VAE), a deep generative model which can impute the values of missing features in sparsely-observed data. We show that this system obtains improved few-shot learning performance for novel features over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.
△ Less
Submitted 12 April, 2021;
originally announced April 2021.
-
Einstein-Gauss-Bonnet gravity with extra dimensions
Authors:
Carsten van de Bruck,
Chris Longden
Abstract:
We consider a theory of modified gravity possessing d extra spatial dimensions with a maximally symmetric metric and a scale factor, whose (4+d)-dimensional gravitational action contains terms proportional to quadratic curvature scalars. Constructing the 4D effective field theory by dimensional reduction, we find that a special case of our action where the additional terms appear in the well-known…
▽ More
We consider a theory of modified gravity possessing d extra spatial dimensions with a maximally symmetric metric and a scale factor, whose (4+d)-dimensional gravitational action contains terms proportional to quadratic curvature scalars. Constructing the 4D effective field theory by dimensional reduction, we find that a special case of our action where the additional terms appear in the well-known Gauss-Bonnet combination is of special interest as it uniquely produces a Horndeski scalar-tensor theory in the 4D effective action. We further consider the possibility of achieving stabilised extra dimensions in this scenario, as a function of the number and curvature of extra dimensions, as well as the strength of the Gauss-Bonnet coupling. Further questions that remain to be answered such as the influence of matter-coupling are briefly discussed.
△ Less
Submitted 4 September, 2018;
originally announced September 2018.
-
Gauss-Bonnet-coupled Quintessential Inflation
Authors:
Carsten van de Bruck,
Konstantinos Dimopoulos,
Chris Longden,
Charlotte Owen
Abstract:
We study in detail a new model of quintessential inflation where the inflaton field is coupled to the Gauss-Bonnet term. This coupling ensures that the variation of the field is kept sub-Planckian, which avoids the 5th force problem as well as the lifting of the flatness of the quintessential tail in the runaway scalar potential due to radiative corrections. We find that the inflationary predictio…
▽ More
We study in detail a new model of quintessential inflation where the inflaton field is coupled to the Gauss-Bonnet term. This coupling ensures that the variation of the field is kept sub-Planckian, which avoids the 5th force problem as well as the lifting of the flatness of the quintessential tail in the runaway scalar potential due to radiative corrections. We find that the inflationary predictions of the model are in excellent agreement with CMB observations, while the coincidence requirement of dark energy is satisfied with natural values of the parameters, overcoming thereby the extreme fine-tuning of the cosmological constant in $Λ$CDM.
△ Less
Submitted 21 July, 2017;
originally announced July 2017.
-
Non-standard hierarchies of the runnings of the spectral index in inflation
Authors:
Chris Longden
Abstract:
Recent analyses of cosmic microwave background surveys have revealed hints that there may be a non-trivial running of the running of the spectral index. If future experiments were to confirm these hints, it would prove a powerful discriminator of inflationary models, ruling out simple single field models. We discuss how isocurvature perturbations in multi-field models can be invoked to generate la…
▽ More
Recent analyses of cosmic microwave background surveys have revealed hints that there may be a non-trivial running of the running of the spectral index. If future experiments were to confirm these hints, it would prove a powerful discriminator of inflationary models, ruling out simple single field models. We discuss how isocurvature perturbations in multi-field models can be invoked to generate large runnings in a non-standard hierarchy, and find that a minimal model capable of practically realising this would be a two-field model with a non-canonical kinetic structure. We also consider alternative scenarios such as variable speed of light models and canonical quantum gravity effects and their implications for runnings of the spectral index.
△ Less
Submitted 1 February, 2017;
originally announced February 2017.
-
The adiabatic/entropy decomposition in $P(φ^I,X^{IJ})$ theories with multiple sound speeds
Authors:
Chris Longden
Abstract:
We consider $P(φ^I,X^{IJ})$ theories of multi-field inflation and ask the question of how to define the adiabatic and entropy perturbations, widely used in calculating the curvature and isocurvature power spectra, in this general context. It is found that when the field perturbations propagate with different speeds, these adiabatic and entropy modes are not generally the fundamental (most natural…
▽ More
We consider $P(φ^I,X^{IJ})$ theories of multi-field inflation and ask the question of how to define the adiabatic and entropy perturbations, widely used in calculating the curvature and isocurvature power spectra, in this general context. It is found that when the field perturbations propagate with different speeds, these adiabatic and entropy modes are not generally the fundamental (most natural to canonically quantise) degrees of freedom that propagate with a single speed. The alternative fields which do propagate with a single speed are found to be a rotation in field space of the adiabatic and entropy perturbations. We show how this affects the form of the horizon-crossing power spectrum, when there is not a single "adiabatic sound speed" sourcing the curvature perturbation. Special cases of our results are discussed, including $P(X)$ theories where the adiabatic and entropy perturbations are fundamental. We finally look at physical motivations for considering multi-speed models of inflation, particularly showing that disformal couplings can naturally lead to the kind of kinetic interactions which cause fields to have different sound speeds.
△ Less
Submitted 9 January, 2017; v1 submitted 8 November, 2016;
originally announced November 2016.
-
Non-Gaussianity in multi-sound-speed disformally coupled inflation
Authors:
Carsten van de Bruck,
Tomi Koivisto,
Chris Longden
Abstract:
Most, if not all, scalar-tensor theories are equivalent to General Relativity with a disformally coupled matter sector. In extra-dimensional theories such a coupling can be understood as a result of induction of the metric on a brane that matter is confined to. This article presents a first look at the non-Gaussianities in disformally coupled inflation, a simple two-field model that features a nov…
▽ More
Most, if not all, scalar-tensor theories are equivalent to General Relativity with a disformally coupled matter sector. In extra-dimensional theories such a coupling can be understood as a result of induction of the metric on a brane that matter is confined to. This article presents a first look at the non-Gaussianities in disformally coupled inflation, a simple two-field model that features a novel kinetic interaction. Cases with both canonical and Dirac-Born-Infeld (DBI) kinetic terms are taken into account, the latter motivated by the possible extra-dimensional origin of the disformality. The computations are carried out for the equilateral configuration in the slow-roll regime, wherein it is found that the non-Gaussianity is typically rather small and negative. This is despite the fact that the new kinetic interaction causes the perturbation modes to propagate with different sounds speeds, which may both significantly deviate from unity during inflation.
△ Less
Submitted 10 February, 2017; v1 submitted 31 August, 2016;
originally announced August 2016.
-
Running of the Running and Entropy Perturbations During Inflation
Authors:
Carsten van de Bruck,
Chris Longden
Abstract:
In single field slow-roll inflation, one expects that the spectral index $n_s -1$ is first order in slow-roll parameters. Similarly, its running $α_s = dn_s/d \log k$ and the running of the running $β_s = dα_s/d \log k$ are second and third order and therefore expected to be progressively smaller, and usually negative. Hence, such models of inflation are in considerable tension with a recent analy…
▽ More
In single field slow-roll inflation, one expects that the spectral index $n_s -1$ is first order in slow-roll parameters. Similarly, its running $α_s = dn_s/d \log k$ and the running of the running $β_s = dα_s/d \log k$ are second and third order and therefore expected to be progressively smaller, and usually negative. Hence, such models of inflation are in considerable tension with a recent analysis hinting that $β_s$ may actually be positive, and larger than $α_s$. Motivated by this, in this work we ask the question of what kinds of inflationary models may be useful in achieving such a hierarchy of runnings, particularly focusing on two--field models of inflation in which the late-time transfer of power from isocurvature to curvature modes allows for a much more diverse range of phenomenology. We calculate the runnings due to this effect and briefly apply our results to assessing the feasibility of finding $|β_s| \gtrsim |α_s|$ in some specific models.
△ Less
Submitted 30 September, 2016; v1 submitted 7 June, 2016;
originally announced June 2016.
-
Reheating in Gauss-Bonnet-coupled inflation
Authors:
Carsten van de Bruck,
Konstantinos Dimopoulos,
Chris Longden
Abstract:
We investigate the feasibility of models of inflation with a large Gauss-Bonnet coupling at late times, which have been shown to modify and prevent the end of inflation. Despite the potential of Gauss-Bonnet models in predicting favourable power spectra, capable of greatly lowering the tensor-to-scalar-ratio compared to now-disfavoured models of standard chaotic inflation, it is important to also…
▽ More
We investigate the feasibility of models of inflation with a large Gauss-Bonnet coupling at late times, which have been shown to modify and prevent the end of inflation. Despite the potential of Gauss-Bonnet models in predicting favourable power spectra, capable of greatly lowering the tensor-to-scalar-ratio compared to now-disfavoured models of standard chaotic inflation, it is important to also understand in what context it is possible for post-inflationary (p)reheating to proceed and hence recover an acceptable late-time cosmology. We argue that in the previously-studied inverse power law coupling case, reheating cannot happen due to a lack of oscillatory solutions for the inflaton, and that neither instant preheating nor gravitational particle production would avoid this problem due to the persistence of the inflaton's energy density, even if it were to partially decay. Hence we proceed to define a minimal generalisation of the model which can permit perturbative reheating and study the consequences of this, including heavily modified dynamics during reheating and predictions of the power spectra.
△ Less
Submitted 20 May, 2016;
originally announced May 2016.
-
Higgs Inflation with a Gauss-Bonnet term in the Jordan Frame
Authors:
Carsten van de Bruck,
Chris Longden
Abstract:
We consider an extension of Higgs inflation in which the Higgs field is coupled to the Gauss-Bonnet term. Working solely in the Jordan frame, we firstly recover the standard predictions of Higgs inflation without a Gauss-Bonnet term. We then calculate the power spectra for scalar and tensor perturbations in the presence of a coupling to a Gauss-Bonnet term. We show that generically the predictions…
▽ More
We consider an extension of Higgs inflation in which the Higgs field is coupled to the Gauss-Bonnet term. Working solely in the Jordan frame, we firstly recover the standard predictions of Higgs inflation without a Gauss-Bonnet term. We then calculate the power spectra for scalar and tensor perturbations in the presence of a coupling to a Gauss-Bonnet term. We show that generically the predictions of Higgs inflation are robust and the contributions to the power spectra coming from the Gauss-Bonnet term are negligible. We find, however, that the end of inflation can be strongly modified and that we hence expect the details of (p)reheating to be significantly altered, leading to some concerns over the feasibility of the model which require further investigations.
△ Less
Submitted 20 May, 2016; v1 submitted 15 December, 2015;
originally announced December 2015.
-
Disformally coupled inflation
Authors:
Carsten van de Bruck,
Tomi Koivisto,
Chris Longden
Abstract:
A disformal coupling between two scalar fields is considered in the context of cosmological inflation. The coupling introduces novel derivative interactions mixing the kinetic terms of the fields but without introducing superluminal or unstable propagation of the two scalar fluctuation modes. Though the typical effect of the disformal coupling is to inhibit one of the fields from inflating the uni…
▽ More
A disformal coupling between two scalar fields is considered in the context of cosmological inflation. The coupling introduces novel derivative interactions mixing the kinetic terms of the fields but without introducing superluminal or unstable propagation of the two scalar fluctuation modes. Though the typical effect of the disformal coupling is to inhibit one of the fields from inflating the universe, the energy density of the other field can drive viable near Sitter -inflation in the presence of nontrivial disformal dynamics, in particular when one assumes exponential instead of power-law form for the couplings. The linear perturbation equations are written for the two-field system, its canonical degrees of freedom are quantised, their spectra are derived and the inflationary predictions are reported for numerically solved exponential models. A generic prediction is low tensor-to-scalar ratio.
△ Less
Submitted 20 May, 2016; v1 submitted 5 October, 2015;
originally announced October 2015.
