-
Large-Scale Gaussian Processes via Alternating Projection
Authors:
Kaiwen Wu,
Jonathan Wenger,
Haydn Jones,
Geoff Pleiss,
Jacob R. Gardner
Abstract:
Training and inference in Gaussian processes (GPs) require solving linear systems with $n\times n$ kernel matrices. To address the prohibitive $\mathcal{O}(n^3)$ time complexity, recent work has employed fast iterative methods, like conjugate gradients (CG). However, as datasets increase in magnitude, the kernel matrices become increasingly ill-conditioned and still require $\mathcal{O}(n^2)$ spac…
▽ More
Training and inference in Gaussian processes (GPs) require solving linear systems with $n\times n$ kernel matrices. To address the prohibitive $\mathcal{O}(n^3)$ time complexity, recent work has employed fast iterative methods, like conjugate gradients (CG). However, as datasets increase in magnitude, the kernel matrices become increasingly ill-conditioned and still require $\mathcal{O}(n^2)$ space without partitioning. Thus, while CG increases the size of datasets GPs can be trained on, modern datasets reach scales beyond its applicability. In this work, we propose an iterative method which only accesses subblocks of the kernel matrix, effectively enabling mini-batching. Our algorithm, based on alternating projection, has $\mathcal{O}(n)$ per-iteration time and space complexity, solving many of the practical challenges of scaling GPs to very large datasets. Theoretically, we prove the method enjoys linear convergence. Empirically, we demonstrate its fast convergence in practice and robustness to ill-conditioning. On large-scale benchmark datasets with up to four million data points, our approach accelerates GP training and inference by speed-up factors up to $27\times$ and $72 \times$, respectively, compared to CG.
△ Less
Submitted 8 March, 2024; v1 submitted 26 October, 2023;
originally announced October 2023.
-
MetaBayesDTA: Codeless Bayesian meta-analysis of test accuracy, with or without a gold standard
Authors:
Enzo Cerullo,
Alex J. Sutton,
Hayley E. Jones,
Olivia Wu,
Terry J. Quinn,
Nicola J. Cooper
Abstract:
Introduction: Despite their applicability, statistical models used for the meta-analysis of test accuracy require specialised knowledge to implement, with the necessary level of expertise having recently increased. This is due to the development and recommendation to use more sophisticated methods; such as those in Version 2 of the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accura…
▽ More
Introduction: Despite their applicability, statistical models used for the meta-analysis of test accuracy require specialised knowledge to implement, with the necessary level of expertise having recently increased. This is due to the development and recommendation to use more sophisticated methods; such as those in Version 2 of the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. This paper describes a web-based application that extends the functionality of previous applications, making many advanced analysis methods more accessible.
Methods: We sought to create an extended, stand-alone, Bayesian version of MetaDTA, which (i) has the benefits of previously proposed applications and addresses key limitations of them, (ii) is accessible to researchers who do not have the specific expertise required to fit such models, and (iii) is suitable for experienced analysts. We created the application using Shiny and Stan.
Results: We created MetaBayesDTA (https://crsu.shinyapps.io/MetaBayesDTA/), which allows users to conduct meta-analysis of test accuracy, with or without a gold standard. The application addresses several key limitations of other applications. For instance, for the bivariate model, one can conduct subgroup analysis, univariate meta-regression, and comparative test accuracy evaluation. Meanwhile, for the model which does not assume a perfect gold standard, the application can account for the fact that studies use different reference tests.
Conclusions: Due to its user-friendliness and broad array of features, MetaBayesDTA should appeal to a wide variety of researchers. We anticipate that the application will encourage wider use of more advanced methods, which ultimately should improve the quality of test accuracy reviews.
△ Less
Submitted 15 November, 2022; v1 submitted 10 November, 2022;
originally announced November 2022.
-
Data-driven Approaches to Surrogate Machine Learning Model Development
Authors:
H. Rhys Jones,
Tingting Mu,
Andrei C. Popescu,
Yusuf Sulehman
Abstract:
We demonstrate the adaption of three established methods to the field of surrogate machine learning model development. These methods are data augmentation, custom loss functions and transfer learning. Each of these methods have seen widespread use in the field of machine learning, however, here we apply them specifically to surrogate machine learning model development. The machine learning model t…
▽ More
We demonstrate the adaption of three established methods to the field of surrogate machine learning model development. These methods are data augmentation, custom loss functions and transfer learning. Each of these methods have seen widespread use in the field of machine learning, however, here we apply them specifically to surrogate machine learning model development. The machine learning model that forms the basis behind this work was intended to surrogate a traditional engineering model used in the UK nuclear industry. Previous performance of this model has been hampered by poor performance due to limited training data. Here, we demonstrate that through a combination of additional techniques, model performance can be significantly improved. We show that each of the aforementioned techniques have utility in their own right and in combination with one another. However, we see them best applied as part of a transfer learning operation. Five pre-trained surrogate models produced prior to this research were further trained with an augmented dataset and with our custom loss function. Through the combination of all three techniques, we see an improvement of at least $38\%$ in performance across the five models.
△ Less
Submitted 3 November, 2022; v1 submitted 5 October, 2022;
originally announced October 2022.
-
Local Latent Space Bayesian Optimization over Structured Inputs
Authors:
Natalie Maus,
Haydn T. Jones,
Juston S. Moore,
Matt J. Kusner,
John Bradshaw,
Jacob R. Gardner
Abstract:
Bayesian optimization over the latent spaces of deep autoencoder models (DAEs) has recently emerged as a promising new approach for optimizing challenging black-box functions over structured, discrete, hard-to-enumerate search spaces (e.g., molecules). Here the DAE dramatically simplifies the search space by map** inputs into a continuous latent space where familiar Bayesian optimization tools c…
▽ More
Bayesian optimization over the latent spaces of deep autoencoder models (DAEs) has recently emerged as a promising new approach for optimizing challenging black-box functions over structured, discrete, hard-to-enumerate search spaces (e.g., molecules). Here the DAE dramatically simplifies the search space by map** inputs into a continuous latent space where familiar Bayesian optimization tools can be more readily applied. Despite this simplification, the latent space typically remains high-dimensional. Thus, even with a well-suited latent space, these approaches do not necessarily provide a complete solution, but may rather shift the structured optimization problem to a high-dimensional one. In this paper, we propose LOL-BO, which adapts the notion of trust regions explored in recent work on high-dimensional Bayesian optimization to the structured setting. By reformulating the encoder to function as both an encoder for the DAE globally and as a deep kernel for the surrogate model within a trust region, we better align the notion of local optimization in the latent space with local optimization in the input space. LOL-BO achieves as much as 20 times improvement over state-of-the-art latent space Bayesian optimization methods across six real-world benchmarks, demonstrating that improvement in optimization strategies is as important as develo** better DAE models.
△ Less
Submitted 22 February, 2023; v1 submitted 27 January, 2022;
originally announced January 2022.
-
Meta-analysis of dichotomous and ordinal tests without a gold standard
Authors:
Enzo Cerullo,
Hayley E. Jones,
Olivia Carter,
Terry J. Quinn,
Nicola J. Cooper,
Alex J. Sutton
Abstract:
Standard methods for the meta-analysis of medical tests without a gold standard are limited to dichotomous data. Multivariate probit models are used to analyze correlated binary data, and can be extended to multivariate ordered probit models to model ordinal data. Within the context of an imperfect gold standard, they have previously been used for the analysis of dichotomous and ordinal tests in a…
▽ More
Standard methods for the meta-analysis of medical tests without a gold standard are limited to dichotomous data. Multivariate probit models are used to analyze correlated binary data, and can be extended to multivariate ordered probit models to model ordinal data. Within the context of an imperfect gold standard, they have previously been used for the analysis of dichotomous and ordinal tests in a single study, and for the meta-analysis of dichotomous tests. In this paper, we developed a hierarchical, latent class multivariate probit model for the simultaneous meta-analysis of ordinal and dichotomous tests without assuming a gold standard. The model can accommodate a hierarchical partial pooling model on the conditional within-study correlations, enabling one to obtain summary estimates of joint test accuracy. Dichotomous tests use probit regression likelihoods and ordinal tests use ordered probit regression likelihoods. We fitted the models using Stan, which uses a state-of-the-art Hamiltonian Monte Carlo algorithm. We applied the models to a dataset in which studies evaluated the accuracy of tests, and test combinations, for deep vein thrombosis. We first demonstrated the issues with dichotomising test accuracy data a priori without a gold standard by fitting models which dichotomised the ordinal test data, and then we applied models which do not dichotomise the data. Furthermore, we fitted and compared a variety of other models, including those which assumed conditional independence and dependence between tests, and those assuming perfect and an imperfect gold standard.
△ Less
Submitted 26 April, 2022; v1 submitted 11 March, 2021;
originally announced March 2021.
-
Agatha: disentangling periodic signals from correlated noise in a periodogram framework
Authors:
Fabo Feng,
Mikko Tuomi,
Hugh R. A. Jones
Abstract:
Periodograms are used as a key significance assessment and visualisation tool to display the significant periodicities in unevenly sampled time series. We introduce a framework of periodograms, called "Agatha", to disentangle periodic signals from correlated noise and to solve the 2-dimensional model selection problem: signal dimension and noise model dimension. These periodograms are calculated b…
▽ More
Periodograms are used as a key significance assessment and visualisation tool to display the significant periodicities in unevenly sampled time series. We introduce a framework of periodograms, called "Agatha", to disentangle periodic signals from correlated noise and to solve the 2-dimensional model selection problem: signal dimension and noise model dimension. These periodograms are calculated by applying likelihood maximization and marginalization and combined in a self-consistent way. We compare Agatha with other periodograms for the detection of Keplerian signals in synthetic radial velocity data produced for the Radial Velocity Challenge as well as in radial velocity datasets of several Sun-like stars. In our tests we find Agatha is able to recover signals to the adopted detection limit of the radial velocity challenge. Applied to real radial velocity, we use Agatha to confirm previous analysis of CoRoT-7 and to find two new planet candidates with minimum masses of 15.1 $M_\oplus$ and 7.08 $M_\oplus$ orbiting HD177565 and HD41248, with periods of 44.5 d and 13.4 d, respectively. We find that Agatha outperforms other periodograms in terms of removing correlated noise and assessing the significances of signals with more robust metrics. Moreover, it can be used to select the optimal noise model and to test the consistency of signals in time. Agatha is intended to be flexible enough to be applied to time series analyses in other astronomical and scientific disciplines. Agatha is available at http://www.agatha.herts.ac.uk.
△ Less
Submitted 8 May, 2017;
originally announced May 2017.
-
Between-trial heterogeneity in meta-analyses may be partially explained by reported design characteristics
Authors:
Kirsty Rhodes,
Rebecca Turner,
Jelena Savović,
Hayley Jones,
David Mawdsley,
Julian Higgins
Abstract:
Objective: We investigated the associations between risk of bias judgments from Cochrane reviews for sequence generation, allocation concealment and blinding and between-trial heterogeneity.
Study Design and Setting: Bayesian hierarchical models were fitted to binary data from 117 meta-analyses, to estimate the ratio λ by which heterogeneity changes for trials at high/unclear risk of bias, compa…
▽ More
Objective: We investigated the associations between risk of bias judgments from Cochrane reviews for sequence generation, allocation concealment and blinding and between-trial heterogeneity.
Study Design and Setting: Bayesian hierarchical models were fitted to binary data from 117 meta-analyses, to estimate the ratio λ by which heterogeneity changes for trials at high/unclear risk of bias, compared to trials at low risk of bias. We estimated the proportion of between-trial heterogeneity in each meta-analysis that could be explained by the bias associated with specific design characteristics.
Results: Univariable analyses showed that heterogeneity variances were, on average, increased among trials at high/unclear risk of bias for sequence generation (λ 1.14, 95% interval: 0.57 to 2.30) and blinding (λ 1.74, 95% interval: 0.85 to 3.47). Trials at high/unclear risk of bias for allocation concealment were on average less heterogeneous (λ 0.75, 95% interval: 0.35 to 1.61). Multivariable analyses showed that a median of 37% (95% interval: 0% to 71%) heterogeneity variance could be explained by trials at high/unclear risk of bias for sequence generation, allocation concealment and/or blinding. All 95% intervals for changes in heterogeneity were wide and included the null of no difference.
Conclusion: Our interpretation of the results is limited by imprecise estimates. There is some indication that between-trial heterogeneity could be partially explained by reported design characteristics, and hence adjustment for bias could potentially improve accuracy of meta-analysis results.
△ Less
Submitted 27 November, 2017; v1 submitted 21 April, 2017;
originally announced April 2017.
-
Label-invariant models for the analysis of meta-epidemiological data
Authors:
Kirsty Rhodes,
David Mawdsley,
Rebecca Turner,
Hayley Jones,
Jelena Savovic,
Julian Higgins
Abstract:
Rich meta-epidemiological data sets have been collected to explore associations between intervention effect estimates and study-level characteristics. Welton et al. proposed models for the analysis of meta-epidemiological data, but these models are restrictive because they force heterogeneity among studies with a particular characteristic to be at least as large as that among studies without the c…
▽ More
Rich meta-epidemiological data sets have been collected to explore associations between intervention effect estimates and study-level characteristics. Welton et al. proposed models for the analysis of meta-epidemiological data, but these models are restrictive because they force heterogeneity among studies with a particular characteristic to be at least as large as that among studies without the characteristic. In this paper we present alternative models that are invariant to the labels defining the two categories of studies. To exemplify the methods, we use a collection of meta-analyses in which the Cochrane Risk of Bias tool has been implemented. We first investigate the influence of small trial sample sizes (less than 100 participants), before investigating the influence of multiple methodological flaws (inadequate or unclear sequence generation, allocation concealment and blinding). We fit both the Welton et al. model and our proposed label-invariant model and compare the results. Estimates of mean bias associated with the trial characteristics and of between-trial variances are not very sensitive to the choice of model. Results from fitting a univariable model show that heterogeneity variance is, on average, 88% greater among trials with less than 100 participants. Based on a multivariable model, heterogeneity variance is, on average, 25% greater among trials with inadequate/unclear sequence generation, 51% greater among trials with inadequate/unclear blinding, and 23% lower among trials with inadequate/unclear allocation concealment, though the 95% intervals for these ratios are very wide. Our proposed label-invariant models for meta-epidemiological data analysis facilitate investigations of between-study heterogeneity attributable to certain study characteristics.
△ Less
Submitted 27 November, 2017; v1 submitted 5 April, 2017;
originally announced April 2017.
-
A Goldilocks principle for modeling radial velocity noise
Authors:
Fabo Feng,
M. Tuomi,
H. R. A. Jones,
R. P. Butler,
S. Vogt
Abstract:
The doppler measurements of stars are diluted and distorted by stellar activity noise. Different choices of noise models and statistical methods have led to much controversy in the confirmation of exoplanet candidates obtained through analysing radial velocity data. To quantify the limitation of various models and methods, we compare different noise models and signal detection criteria for various…
▽ More
The doppler measurements of stars are diluted and distorted by stellar activity noise. Different choices of noise models and statistical methods have led to much controversy in the confirmation of exoplanet candidates obtained through analysing radial velocity data. To quantify the limitation of various models and methods, we compare different noise models and signal detection criteria for various simulated and real data sets in the Bayesian framework. According to our analyses, the white noise model tend to interpret noise as signal, leading to false positives. On the other hand, the red noise models are likely to interprete signal as noise, resulting in false negatives. We find that the Bayesian information criterion combined with a Bayes factor threshold of 150 can efficiently rule out false positives and confirm true detections. We further propose a Goldilocks principle aimed at modeling radial velocity noise to avoid too many false positives and too many false negatives. We propose that the noise model with RHK-dependent jitter is used in combination with the moving average model to detect planetary signals for M dwarfs. Our work may also shed light on the noise modeling for hotter stars, and provide a valid approach for finding similar principles in other disciplines.
△ Less
Submitted 16 June, 2016;
originally announced June 2016.
-
Probabilities of exoplanet signals from posterior samplings
Authors:
Mikko Tuomi,
Hugh R. A. Jones
Abstract:
Estimating the marginal likelihoods is an essential feature of model selection in the Bayesian context. It is especially crucial to have good estimates when assessing the number of planets orbiting stars when the models explain the noisy data with different numbers of Keplerian signals. We introduce a simple method for approximating the marginal likelihoods in practice when a statistically represe…
▽ More
Estimating the marginal likelihoods is an essential feature of model selection in the Bayesian context. It is especially crucial to have good estimates when assessing the number of planets orbiting stars when the models explain the noisy data with different numbers of Keplerian signals. We introduce a simple method for approximating the marginal likelihoods in practice when a statistically representative sample from the parameter posterior density is available.
We use our truncated posterior mixture estimate to receive accurate model probabilities for models with differing number of Keplerian signals in radial velocity data. We test this estimate in simple scenarios to assess its accuracy and rate of convergence in practice when the corresponding estimates calculated using deviance information criterion can be applied to receive trustworthy results for reliable comparison. As a test case, we determine the posterior probability of a planet orbiting HD 3651 given Lick and Keck radial velocity data.
The posterior mixture estimate appears to be a simple and an accurate way of calculating marginal integrals from posterior samples. We show, that it can be used to estimate the marginal integrals reliably in practice, given a suitable selection of parameter λ, that controls its accuracy and convergence rate. It is also more accurate than the one block Metropolis-Hastings estimate and can be used in any application because it is not based on assumptions on the nature of the posterior density nor the amount of data or parameters in the statistical model.
△ Less
Submitted 25 June, 2012; v1 submitted 27 December, 2011;
originally announced December 2011.
-
Application of Bayesian model inadequacy criterion for multiple data sets to radial velocity models of exoplanet systems
Authors:
Mikko Tuomi,
David Pinfield,
Hugh R. A. Jones
Abstract:
We present a simple mathematical criterion for determining whether a given statistical model does not describe several independent sets of measurements, or data modes, adequately. We derive this criterion for two data sets and generalise it to several sets by using the Bayesian updating of the posterior probability density. To demonstrate the usage of the criterion, we apply it to observations of…
▽ More
We present a simple mathematical criterion for determining whether a given statistical model does not describe several independent sets of measurements, or data modes, adequately. We derive this criterion for two data sets and generalise it to several sets by using the Bayesian updating of the posterior probability density. To demonstrate the usage of the criterion, we apply it to observations of exoplanet host stars by re-analysing the radial velocities of HD 217107, Gliese 581, and \u{psion} Andromedae and show that the currently used models are not necessarily adequate in describing the properties of these measurements. We show that while the two data sets of Gliese 581 can be modelled reasonably well, the noise model of HD 217107 needs to be revised. We also reveal some biases in the radial velocities of \u{psion} Andromedae and report updated orbital parameters for the recently proposed 4-planet model. Because of the generality of our criterion, no assumptions are needed on the nature of the measurements, models, or model parameters. The method we propose can be applied to any astronomical problems, as well as outside the field of astronomy, because it is a simple consequence of the Bayes' rule of conditional probabilities.
△ Less
Submitted 29 June, 2011;
originally announced June 2011.
