-
Fast Gibbs sampling for the local and global trend Bayesian exponential smoothing model
Authors:
Xueying Long,
Daniel F. Schmidt,
Christoph Bergmeir,
Slawek Smyl
Abstract:
In Smyl et al. [Local and global trend Bayesian exponential smoothing models. International Journal of Forecasting, 2024.], a generalised exponential smoothing model was proposed that is able to capture strong trends and volatility in time series. This method achieved state-of-the-art performance in many forecasting tasks, but its fitting procedure, which is based on the NUTS sampler, is very comp…
▽ More
In Smyl et al. [Local and global trend Bayesian exponential smoothing models. International Journal of Forecasting, 2024.], a generalised exponential smoothing model was proposed that is able to capture strong trends and volatility in time series. This method achieved state-of-the-art performance in many forecasting tasks, but its fitting procedure, which is based on the NUTS sampler, is very computationally expensive. In this work, we propose several modifications to the original model, as well as a bespoke Gibbs sampler for posterior exploration; these changes improve sampling time by an order of magnitude, thus rendering the model much more practically relevant. The new model, and sampler, are evaluated on the M3 dataset and are shown to be competitive, or superior, in terms of accuracy to the original method, while being substantially faster to run.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
-
Roadmap on Data-Centric Materials Science
Authors:
Stefan Bauer,
Peter Benner,
Tristan Bereau,
Volker Blum,
Mario Boley,
Christian Carbogno,
C. Richard A. Catlow,
Gerhard Dehm,
Sebastian Eibl,
Ralph Ernstorfer,
Ádám Fekete,
Lucas Foppa,
Peter Fratzl,
Christoph Freysoldt,
Baptiste Gault,
Luca M. Ghiringhelli,
Sajal K. Giri,
Anton Gladyshev,
Pawan Goyal,
Jason Hattrick-Simpers,
Lara Kabalan,
Petr Karpov,
Mohammad S. Khorrami,
Christoph Koch,
Sebastian Kokott
, et al. (36 additional authors not shown)
Abstract:
Science is and always has been based on data, but the terms "data-centric" and the "4th paradigm of" materials research indicate a radical change in how information is retrieved, handled and research is performed. It signifies a transformative shift towards managing vast data collections, digital repositories, and innovative data analytics methods. The integration of Artificial Intelligence (AI) a…
▽ More
Science is and always has been based on data, but the terms "data-centric" and the "4th paradigm of" materials research indicate a radical change in how information is retrieved, handled and research is performed. It signifies a transformative shift towards managing vast data collections, digital repositories, and innovative data analytics methods. The integration of Artificial Intelligence (AI) and its subset Machine Learning (ML), has become pivotal in addressing all these challenges. This Roadmap on Data-Centric Materials Science explores fundamental concepts and methodologies, illustrating diverse applications in electronic-structure theory, soft matter theory, microstructure research, and experimental techniques like photoemission, atom probe tomography, and electron microscopy. While the roadmap delves into specific areas within the broad interdisciplinary field of materials science, the provided examples elucidate key concepts applicable to a wider range of topics. The discussed instances offer insights into addressing the multifaceted challenges encountered in contemporary materials research.
△ Less
Submitted 1 May, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Prevalidated ridge regression is a highly-efficient drop-in replacement for logistic regression for high-dimensional data
Authors:
Angus Dempster,
Geoffrey I. Webb,
Daniel F. Schmidt
Abstract:
Logistic regression is a ubiquitous method for probabilistic classification. However, the effectiveness of logistic regression depends upon careful and relatively computationally expensive tuning, especially for the regularisation hyperparameter, and especially in the context of high-dimensional data. We present a prevalidated ridge regression model that closely matches logistic regression in term…
▽ More
Logistic regression is a ubiquitous method for probabilistic classification. However, the effectiveness of logistic regression depends upon careful and relatively computationally expensive tuning, especially for the regularisation hyperparameter, and especially in the context of high-dimensional data. We present a prevalidated ridge regression model that closely matches logistic regression in terms of classification error and log-loss, particularly for high-dimensional data, while being significantly more computationally efficient and having effectively no hyperparameters beyond regularisation. We scale the coefficients of the model so as to minimise log-loss for a set of prevalidated predictions derived from the estimated leave-one-out cross-validation error. This exploits quantities already computed in the course of fitting the ridge regression model in order to find the scaling parameter with nominal additional computational expense.
△ Less
Submitted 28 January, 2024;
originally announced January 2024.
-
From Prediction to Action: Critical Role of Performance Estimation for Machine-Learning-Driven Materials Discovery
Authors:
Mario Boley,
Felix Luong,
Simon Teshuva,
Daniel F Schmidt,
Lucas Foppa,
Matthias Scheffler
Abstract:
Materials discovery driven by statistical property models is an iterative decision process, during which an initial data collection is extended with new data proposed by a model-informed acquisition function--with the goal to maximize a certain "reward" over time, such as the maximum property value discovered so far. While the materials science community achieved much progress in develo** proper…
▽ More
Materials discovery driven by statistical property models is an iterative decision process, during which an initial data collection is extended with new data proposed by a model-informed acquisition function--with the goal to maximize a certain "reward" over time, such as the maximum property value discovered so far. While the materials science community achieved much progress in develo** property models that predict well on average with respect to the training distribution, this form of in-distribution performance measurement is not directly coupled with the discovery reward. This is because an iterative discovery process has a shifting reward distribution that is over-proportionally determined by the model performance for exceptional materials. We demonstrate this problem using the example of bulk modulus maximization among double perovskite oxides. We find that the in-distribution predictive performance suggests random forests as superior to Gaussian process regression, while the results are inverse in terms of the discovery rewards. We argue that the lack of proper performance estimation methods from pre-computed data collections is a fundamental problem for improving data-driven materials discovery, and we propose a novel such estimator that, in contrast to naïve reward estimation, successfully predicts Gaussian processes with the "expected improvement" acquisition function as the best out of four options in our demonstrational study for double perovskites. Importantly, it does so without requiring the over thousand ab initio computations that were needed to confirm this prediction.
△ Less
Submitted 6 December, 2023; v1 submitted 27 November, 2023;
originally announced November 2023.
-
Scalable Probabilistic Forecasting in Retail with Gradient Boosted Trees: A Practitioner's Approach
Authors:
Xueying Long,
Quang Bui,
Grady Oktavian,
Daniel F. Schmidt,
Christoph Bergmeir,
Rakshitha Godahewa,
Seong Per Lee,
Kaifeng Zhao,
Paul Condylis
Abstract:
The recent M5 competition has advanced the state-of-the-art in retail forecasting. However, we notice important differences between the competition challenge and the challenges we face in a large e-commerce company. The datasets in our scenario are larger (hundreds of thousands of time series), and e-commerce can afford to have a larger assortment than brick-and-mortar retailers, leading to more i…
▽ More
The recent M5 competition has advanced the state-of-the-art in retail forecasting. However, we notice important differences between the competition challenge and the challenges we face in a large e-commerce company. The datasets in our scenario are larger (hundreds of thousands of time series), and e-commerce can afford to have a larger assortment than brick-and-mortar retailers, leading to more intermittent data. To scale to larger dataset sizes with feasible computational effort, firstly, we investigate a two-layer hierarchy and propose a top-down approach to forecasting at an aggregated level with less amount of series and intermittency, and then disaggregating to obtain the decision-level forecasts. Probabilistic forecasts are generated under distributional assumptions. Secondly, direct training at the lower level with subsamples can also be an alternative way of scaling. Performance of modelling with subsets is evaluated with the main dataset. Apart from a proprietary dataset, the proposed scalable methods are evaluated using the Favorita dataset and the M5 dataset. We are able to show the differences in characteristics of the e-commerce and brick-and-mortar retail datasets. Notably, our top-down forecasting framework enters the top 50 of the original M5 competition, even with models trained at a higher level under a much simpler setting.
△ Less
Submitted 2 November, 2023;
originally announced November 2023.
-
Bayes beats Cross Validation: Efficient and Accurate Ridge Regression via Expectation Maximization
Authors:
Shu Yu Tew,
Mario Boley,
Daniel F. Schmidt
Abstract:
We present a novel method for tuning the regularization hyper-parameter, $λ$, of a ridge regression that is faster to compute than leave-one-out cross-validation (LOOCV) while yielding estimates of the regression parameters of equal, or particularly in the setting of sparse covariates, superior quality to those obtained by minimising the LOOCV risk. The LOOCV risk can suffer from multiple and bad…
▽ More
We present a novel method for tuning the regularization hyper-parameter, $λ$, of a ridge regression that is faster to compute than leave-one-out cross-validation (LOOCV) while yielding estimates of the regression parameters of equal, or particularly in the setting of sparse covariates, superior quality to those obtained by minimising the LOOCV risk. The LOOCV risk can suffer from multiple and bad local minima for finite $n$ and thus requires the specification of a set of candidate $λ$, which can fail to provide good solutions. In contrast, we show that the proposed method is guaranteed to find a unique optimal solution for large enough $n$, under relatively mild conditions, without requiring the specification of any difficult to determine hyper-parameters. This is based on a Bayesian formulation of ridge regression that we prove to have a unimodal posterior for large enough $n$, allowing for both the optimal $λ$ and the regression coefficients to be jointly learned within an iterative expectation maximization (EM) procedure. Importantly, we show that by utilizing an appropriate preprocessing step, a single iteration of the main EM loop can be implemented in $O(\min(n, p))$ operations, for input data with $n$ rows and $p$ columns. In contrast, evaluating a single value of $λ$ using fast LOOCV costs $O(n \min(n, p))$ operations when using the same preprocessing. This advantage amounts to an asymptotic improvement of a factor of $l$ for $l$ candidate values for $λ$ (in the regime $q, p \in O(\sqrt{n})$ where $q$ is the number of regression targets).
△ Less
Submitted 2 November, 2023; v1 submitted 28 October, 2023;
originally announced October 2023.
-
Computing Marginal and Conditional Divergences between Decomposable Models with Applications
Authors:
Loong Kuan Lee,
Geoffrey I. Webb,
Daniel F. Schmidt,
Nico Piatkowski
Abstract:
The ability to compute the exact divergence between two high-dimensional distributions is useful in many applications but doing so naively is intractable. Computing the alpha-beta divergence -- a family of divergences that includes the Kullback-Leibler divergence and Hellinger distance -- between the joint distribution of two decomposable models, i.e chordal Markov networks, can be done in time ex…
▽ More
The ability to compute the exact divergence between two high-dimensional distributions is useful in many applications but doing so naively is intractable. Computing the alpha-beta divergence -- a family of divergences that includes the Kullback-Leibler divergence and Hellinger distance -- between the joint distribution of two decomposable models, i.e chordal Markov networks, can be done in time exponential in the treewidth of these models. However, reducing the dissimilarity between two high-dimensional objects to a single scalar value can be uninformative. Furthermore, in applications such as supervised learning, the divergence over a conditional distribution might be of more interest. Therefore, we propose an approach to compute the exact alpha-beta divergence between any marginal or conditional distribution of two decomposable models. Doing so tractably is non-trivial as we need to decompose the divergence between these distributions and therefore, require a decomposition over the marginal and conditional distributions of these models. Consequently, we provide such a decomposition and also extend existing work to compute the marginal and conditional alpha-beta divergence between these decompositions. We then show how our method can be used to analyze distributional changes by first applying it to a benchmark image dataset. Finally, based on our framework, we propose a novel way to quantify the error in contemporary superconducting quantum computers. Code for all experiments is available at: https://lklee.dev/pub/2023-icdm/code
△ Less
Submitted 13 October, 2023;
originally announced October 2023.
-
QUANT: A Minimalist Interval Method for Time Series Classification
Authors:
Angus Dempster,
Daniel F. Schmidt,
Geoffrey I. Webb
Abstract:
We show that it is possible to achieve the same accuracy, on average, as the most accurate existing interval methods for time series classification on a standard set of benchmark datasets using a single type of feature (quantiles), fixed intervals, and an 'off the shelf' classifier. This distillation of interval-based approaches represents a fast and accurate method for time series classification,…
▽ More
We show that it is possible to achieve the same accuracy, on average, as the most accurate existing interval methods for time series classification on a standard set of benchmark datasets using a single type of feature (quantiles), fixed intervals, and an 'off the shelf' classifier. This distillation of interval-based approaches represents a fast and accurate method for time series classification, achieving state-of-the-art accuracy on the expanded set of 142 datasets in the UCR archive with a total compute time (training and inference) of less than 15 minutes using a single CPU core.
△ Less
Submitted 2 August, 2023;
originally announced August 2023.
-
An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set
Authors:
Ali Ismail-Fawaz,
Angus Dempster,
Chang Wei Tan,
Matthieu Herrmann,
Lynn Miller,
Daniel F. Schmidt,
Stefano Berretti,
Jonathan Weber,
Maxime Devanne,
Germain Forestier,
Geoffrey I. Webb
Abstract:
The measurement of progress using benchmarks evaluations is ubiquitous in computer science and machine learning. However, common approaches to analyzing and presenting the results of benchmark comparisons of multiple algorithms over multiple datasets, such as the critical difference diagram introduced by Demšar (2006), have important shortcomings and, we show, are open to both inadvertent and inte…
▽ More
The measurement of progress using benchmarks evaluations is ubiquitous in computer science and machine learning. However, common approaches to analyzing and presenting the results of benchmark comparisons of multiple algorithms over multiple datasets, such as the critical difference diagram introduced by Demšar (2006), have important shortcomings and, we show, are open to both inadvertent and intentional manipulation. To address these issues, we propose a new approach to presenting the results of benchmark comparisons, the Multiple Comparison Matrix (MCM), that prioritizes pairwise comparisons and precludes the means of manipulating experimental results in existing approaches. MCM can be used to show the results of an all-pairs comparison, or to show the results of a comparison between one or more selected algorithms and the state of the art. MCM is implemented in Python and is publicly available.
△ Less
Submitted 19 May, 2023;
originally announced May 2023.
-
Sparse Horseshoe Estimation via Expectation-Maximisation
Authors:
Shu Yu Tew,
Daniel F. Schmidt,
Enes Makalic
Abstract:
The horseshoe prior is known to possess many desirable properties for Bayesian estimation of sparse parameter vectors, yet its density function lacks an analytic form. As such, it is challenging to find a closed-form solution for the posterior mode. Conventional horseshoe estimators use the posterior mean to estimate the parameters, but these estimates are not sparse. We propose a novel expectatio…
▽ More
The horseshoe prior is known to possess many desirable properties for Bayesian estimation of sparse parameter vectors, yet its density function lacks an analytic form. As such, it is challenging to find a closed-form solution for the posterior mode. Conventional horseshoe estimators use the posterior mean to estimate the parameters, but these estimates are not sparse. We propose a novel expectation-maximisation (EM) procedure for computing the MAP estimates of the parameters in the case of the standard linear model. A particular strength of our approach is that the M-step depends only on the form of the prior and it is independent of the form of the likelihood. We introduce several simple modifications of this EM procedure that allow for straightforward extension to generalised linear models. In experiments performed on simulated and real data, our approach performs comparable, or superior to, state-of-the-art sparse estimation methods in terms of statistical performance and computational cost.
△ Less
Submitted 6 November, 2022;
originally announced November 2022.
-
Minimum message length inference of the Weibull distribution with complete and censored data
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
The Weibull distribution, with shape parameter $k>0$ and scale parameter $λ>0$, is one of the most popular parametric distributions in survival analysis with complete or censored data. Although inference of the parameters of the Weibull distribution is commonly done through maximum likelihood, it is well established that the maximum likelihood estimate of the shape parameter is inadequate due to t…
▽ More
The Weibull distribution, with shape parameter $k>0$ and scale parameter $λ>0$, is one of the most popular parametric distributions in survival analysis with complete or censored data. Although inference of the parameters of the Weibull distribution is commonly done through maximum likelihood, it is well established that the maximum likelihood estimate of the shape parameter is inadequate due to the associated large bias when the sample size is small or the proportion of censored data is large. This manuscript demonstrates how the Bayesian information-theoretic minimum message length principle coupled with a suitable choice of weakly informative prior distributions, can be used to infer Weibull distribution parameters given complete data or data with type I censoring. Empirical experiments show that the proposed minimum message length estimate of the shape parameter is superior to the maximum likelihood estimate and appears superior to other recently proposed modified maximum likelihood estimates in terms of Kullback-Leibler risk. Lastly, we derive an extension of the proposed method to data with type II censoring.
△ Less
Submitted 16 March, 2023; v1 submitted 29 September, 2022;
originally announced September 2022.
-
Introduction to minimum message length inference
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
The aim of this manuscript is to introduce the Bayesian minimum message length principle of inductive inference to a general statistical audience that may not be familiar with information theoretic statistics. We describe two key minimum message length inference approaches and demonstrate how the principle can be used to develop a new Bayesian alternative to the frequentist $t$-test as well as new…
▽ More
The aim of this manuscript is to introduce the Bayesian minimum message length principle of inductive inference to a general statistical audience that may not be familiar with information theoretic statistics. We describe two key minimum message length inference approaches and demonstrate how the principle can be used to develop a new Bayesian alternative to the frequentist $t$-test as well as new approaches to hypothesis testing for the correlation coefficient. Lastly, we compare the minimum message length approach to the closely related minimum description length principle and discuss similarities and differences between both approaches to inference.
△ Less
Submitted 29 September, 2022;
originally announced September 2022.
-
Maximum likelihood estimation of the Weibull distribution with reduced bias
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
In this short note, we derive a new bias adjusted maximum likelihood estimate for the shape parameter of the Weibull distribution with complete data and type I censored data. The proposed estimate of the shape parameter is significantly less biased and more efficient than the corresponding maximum likelihood estimate, while being simple to compute using existing maximum likelihood software procedu…
▽ More
In this short note, we derive a new bias adjusted maximum likelihood estimate for the shape parameter of the Weibull distribution with complete data and type I censored data. The proposed estimate of the shape parameter is significantly less biased and more efficient than the corresponding maximum likelihood estimate, while being simple to compute using existing maximum likelihood software procedures.
△ Less
Submitted 9 February, 2023; v1 submitted 29 September, 2022;
originally announced September 2022.
-
MML Probabilistic Principal Component Analysis
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
Principal component analysis (PCA) is perhaps the most widely method for data dimensionality reduction. A key question in PCA decomposition of data is deciding how many factors to retain. This manuscript describes a new approach to automatically selecting the number of principal components based on the Bayesian minimum message length method of inductive inference. We also derive a new estimate of…
▽ More
Principal component analysis (PCA) is perhaps the most widely method for data dimensionality reduction. A key question in PCA decomposition of data is deciding how many factors to retain. This manuscript describes a new approach to automatically selecting the number of principal components based on the Bayesian minimum message length method of inductive inference. We also derive a new estimate of the isotropic residual variance and demonstrate, via numerical experiments, that it improves on the usual maximum likelihood approach.
△ Less
Submitted 16 February, 2023; v1 submitted 29 September, 2022;
originally announced September 2022.
-
HYDRA: Competing convolutional kernels for fast and accurate time series classification
Authors:
Angus Dempster,
Daniel F. Schmidt,
Geoffrey I. Webb
Abstract:
We demonstrate a simple connection between dictionary methods for time series classification, which involve extracting and counting symbolic patterns in time series, and methods based on transforming input time series using convolutional kernels, namely ROCKET and its variants. We show that by adjusting a single hyperparameter it is possible to move by degrees between models resembling dictionary…
▽ More
We demonstrate a simple connection between dictionary methods for time series classification, which involve extracting and counting symbolic patterns in time series, and methods based on transforming input time series using convolutional kernels, namely ROCKET and its variants. We show that by adjusting a single hyperparameter it is possible to move by degrees between models resembling dictionary methods and models resembling ROCKET. We present HYDRA, a simple, fast, and accurate dictionary method for time series classification using competing convolutional kernels, combining key aspects of both ROCKET and conventional dictionary methods. HYDRA is faster and more accurate than the most accurate existing dictionary methods, and can be combined with ROCKET and its variants to further improve the accuracy of these methods.
△ Less
Submitted 25 March, 2022;
originally announced March 2022.
-
MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification
Authors:
Angus Dempster,
Daniel F. Schmidt,
Geoffrey I. Webb
Abstract:
Until recently, the most accurate methods for time series classification were limited by high computational complexity. ROCKET achieves state-of-the-art accuracy with a fraction of the computational expense of most existing methods by transforming input time series using random convolutional kernels, and using the transformed features to train a linear classifier. We reformulate ROCKET into a new…
▽ More
Until recently, the most accurate methods for time series classification were limited by high computational complexity. ROCKET achieves state-of-the-art accuracy with a fraction of the computational expense of most existing methods by transforming input time series using random convolutional kernels, and using the transformed features to train a linear classifier. We reformulate ROCKET into a new method, MINIROCKET, making it up to 75 times faster on larger datasets, and making it almost deterministic (and optionally, with additional computational expense, fully deterministic), while maintaining essentially the same accuracy. Using this method, it is possible to train and test a classifier on all of 109 datasets from the UCR archive to state-of-the-art accuracy in less than 10 minutes. MINIROCKET is significantly faster than any other method of comparable accuracy (including ROCKET), and significantly more accurate than any other method of even roughly-similar computational expense. As such, we suggest that MINIROCKET should now be considered and used as the default variant of ROCKET.
△ Less
Submitted 14 July, 2021; v1 submitted 16 December, 2020;
originally announced December 2020.
-
InceptionTime: Finding AlexNet for Time Series Classification
Authors:
Hassan Ismail Fawaz,
Benjamin Lucas,
Germain Forestier,
Charlotte Pelletier,
Daniel F. Schmidt,
Jonathan Weber,
Geoffrey I. Webb,
Lhassane Idoumghar,
Pierre-Alain Muller,
François Petitjean
Abstract:
This paper brings deep learning at the forefront of research into Time Series Classification (TSC). TSC is the area of machine learning tasked with the categorization (or labelling) of time series. The last few decades of work in this area have led to significant progress in the accuracy of classifiers, with the state of the art now represented by the HIVE-COTE algorithm. While extremely accurate,…
▽ More
This paper brings deep learning at the forefront of research into Time Series Classification (TSC). TSC is the area of machine learning tasked with the categorization (or labelling) of time series. The last few decades of work in this area have led to significant progress in the accuracy of classifiers, with the state of the art now represented by the HIVE-COTE algorithm. While extremely accurate, HIVE-COTE cannot be applied to many real-world datasets because of its high training time complexity in O(N2 * T4) for a dataset with N time series of length T. For example, it takes HIVE-COTE more than 8 days to learn from a small dataset with N = 1500 time series of short length T = 46. Meanwhile deep learning has received enormous attention because of its high accuracy and scalability. Recent approaches to deep learning for TSC have been scalable, but less accurate than HIVE-COTE. We introduce InceptionTime - an ensemble of deep Convolutional Neural Network (CNN) models, inspired by the Inception-v4 architecture. Our experiments show that InceptionTime is on par with HIVE-COTE in terms of accuracy while being much more scalable: not only can it learn from 1,500 time series in one hour but it can also learn from 8M time series in 13 hours, a quantity of data that is fully out of reach of HIVE-COTE.
△ Less
Submitted 5 December, 2020; v1 submitted 11 September, 2019;
originally announced September 2019.
-
An efficient algorithm for sampling from $\sin^k(x)$ for generating random correlation matrices
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
In this note, we develop a novel algorithm for generating random numbers from a distribution with a probability density function proportional to $\sin^k(x)$, $x \in (0,π)$ and $k \geq 1$. Our algorithm is highly efficient and is based on rejection sampling where the envelope distribution is an appropriately chosen beta distribution. An example application illustrating how the new algorithm can be…
▽ More
In this note, we develop a novel algorithm for generating random numbers from a distribution with a probability density function proportional to $\sin^k(x)$, $x \in (0,π)$ and $k \geq 1$. Our algorithm is highly efficient and is based on rejection sampling where the envelope distribution is an appropriately chosen beta distribution. An example application illustrating how the new algorithm can be used to generate random correlation matrices is discussed.
△ Less
Submitted 20 November, 2018; v1 submitted 13 September, 2018;
originally announced September 2018.
-
A Minimum Message Length Criterion for Robust Linear Regression
Authors:
Chi Kuen Wong,
Enes Makalic,
Daniel F. Schmidt
Abstract:
This paper applies the minimum message length principle to inference of linear regression models with Student-t errors. A new criterion for variable selection and parameter estimation in Student-t regression is proposed. By exploiting properties of the regression model, we derive a suitable non-informative proper uniform prior distribution for the regression coefficients that leads to a simple and…
▽ More
This paper applies the minimum message length principle to inference of linear regression models with Student-t errors. A new criterion for variable selection and parameter estimation in Student-t regression is proposed. By exploiting properties of the regression model, we derive a suitable non-informative proper uniform prior distribution for the regression coefficients that leads to a simple and easy-to-apply criterion. Our proposed criterion does not require specification of hyperparameters and is invariant under both full rank transformations of the design matrix and linear transformations of the outcomes. We compare the proposed criterion with several standard model selection criteria, such as the Akaike information criterion and the Bayesian information criterion, on simulations and real data with promising results.
△ Less
Submitted 19 February, 2018; v1 submitted 9 February, 2018;
originally announced February 2018.
-
Log-Scale Shrinkage Priors and Adaptive Bayesian Global-Local Shrinkage Estimation
Authors:
Daniel F. Schmidt,
Enes Makalic
Abstract:
Global-local shrinkage hierarchies are an important innovation in Bayesian estimation. We propose the use of log-scale distributions as a novel basis for generating familes of prior distributions for local shrinkage hyperparameters. By varying the scale parameter one may vary the degree to which the prior distribution promotes sparsity in the coefficient estimates. By examining the class of distri…
▽ More
Global-local shrinkage hierarchies are an important innovation in Bayesian estimation. We propose the use of log-scale distributions as a novel basis for generating familes of prior distributions for local shrinkage hyperparameters. By varying the scale parameter one may vary the degree to which the prior distribution promotes sparsity in the coefficient estimates. By examining the class of distributions over the logarithm of the local shrinkage parameter that have log-linear, or sub-log-linear tails, we show that many standard prior distributions for local shrinkage parameters can be unified in terms of the tail behaviour and concentration properties of their corresponding marginal distributions over the coefficients $β_j$. We derive upper bounds on the rate of concentration around $|β_j|=0$, and the tail decay as $|β_j| \to \infty$, achievable by this wide class of prior distributions.
We then propose a new type of ultra-heavy tailed prior, called the log-$t$ prior with the property that, irrespective of the choice of associated scale parameter, the marginal distribution always diverges at $β_j = 0$, and always possesses super-Cauchy tails. We develop results demonstrating when prior distributions with (sub)-log-linear tails attain Kullback--Leibler super-efficiency and prove that the log-$t$ prior distribution is always super-efficient. We show that the log-$t$ prior is less sensitive to misspecification of the global shrinkage parameter than the horseshoe or lasso priors. By incorporating the scale parameter of the log-scale prior distributions into the Bayesian hierarchy we derive novel adaptive shrinkage procedures. Simulations show that the adaptive log-$t$ procedure appears to always perform well, irrespective of the level of sparsity or signal-to-noise ratio of the underlying model.
△ Less
Submitted 30 January, 2020; v1 submitted 8 January, 2018;
originally announced January 2018.
-
Bayesian Sparse Global-Local Shrinkage Regression for Selection of Grouped Variables
Authors:
Zemei Xu,
Daniel F. Schmidt,
Enes Makalic,
Guoqi Qian,
John L. Hopper
Abstract:
Most estimates for penalised linear regression can be viewed as posterior modes for an appropriate choice of prior distribution. Bayesian shrinkage methods, particularly the horseshoe estimator, have recently attracted a great deal of attention in the problem of estimating sparse, high-dimensional linear models. This paper extends these ideas, and presents a Bayesian grouped model with continuous…
▽ More
Most estimates for penalised linear regression can be viewed as posterior modes for an appropriate choice of prior distribution. Bayesian shrinkage methods, particularly the horseshoe estimator, have recently attracted a great deal of attention in the problem of estimating sparse, high-dimensional linear models. This paper extends these ideas, and presents a Bayesian grouped model with continuous global-local shrinkage priors to handle complex group hierarchies that include overlap** and multilevel group structures. As the posterior mean is never a sparse estimate of the linear model coefficients, we extend the recently proposed decoupled shrinkage and selection (DSS) technique to the problem of selecting groups of variables from posterior samples. To choose a final, sparse model, we also adapt generalised information criteria approaches to the DSS framework. To ensure that sparse groups, in which only a few predictors are active, can be effectively identified, we provide an alternative degrees of freedom estimator for sparse Bayesian linear models that takes into account the effects of shrinkage on the model coefficients. Simulations and real data analysis using our proposed method show promising performance in terms of correct identification of active and inactive groups, and prediction, in comparison with a Bayesian grouped slab-and-spike approach.
△ Less
Submitted 3 November, 2017; v1 submitted 13 September, 2017;
originally announced September 2017.
-
Minimum message length inference of the Poisson and geometric models using heavy-tailed prior distributions
Authors:
Chi Kuen Wong,
Enes Makalic,
Daniel F. Schmidt
Abstract:
Minimum message length is a general Bayesian principle for model selection and parameter estimation that is based on information theory. This paper applies the minimum message length principle to a small-sample model selection problem involving Poisson and geometric data models. Since MML is a Bayesian principle, it requires prior distributions for all model parameters. We introduce three candidat…
▽ More
Minimum message length is a general Bayesian principle for model selection and parameter estimation that is based on information theory. This paper applies the minimum message length principle to a small-sample model selection problem involving Poisson and geometric data models. Since MML is a Bayesian principle, it requires prior distributions for all model parameters. We introduce three candidate prior distributions for the unknown model parameters with both light- and heavy-tails. The performance of the MML methods is compared with objective Bayesian inference and minimum description length techniques based on the normalized maximum likelihood code. Simulations show that our MML approach with a heavy-tail prior distribution provides an excellent performance in all tests.
△ Less
Submitted 11 February, 2018; v1 submitted 9 August, 2017;
originally announced August 2017.
-
High-Dimensional Bayesian Regularised Regression with the BayesReg Package
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
Bayesian penalized regression techniques, such as the Bayesian lasso and the Bayesian horseshoe estimator, have recently received a significant amount of attention in the statistics literature. However, software implementing state-of-the-art Bayesian penalized regression, outside of general purpose Markov chain Monte Carlo platforms such as STAN, is relatively rare. This paper introduces bayesreg,…
▽ More
Bayesian penalized regression techniques, such as the Bayesian lasso and the Bayesian horseshoe estimator, have recently received a significant amount of attention in the statistics literature. However, software implementing state-of-the-art Bayesian penalized regression, outside of general purpose Markov chain Monte Carlo platforms such as STAN, is relatively rare. This paper introduces bayesreg, a new toolbox for fitting Bayesian penalized regression models with continuous shrinkage prior densities. The toolbox features Bayesian linear regression with Gaussian or heavy-tailed error models and Bayesian logistic regression with ridge, lasso, horseshoe and horseshoe$+$ estimators. The toolbox is free, open-source and available for use with the MATLAB and R numerical platforms.
△ Less
Submitted 19 December, 2016; v1 submitted 21 November, 2016;
originally announced November 2016.
-
A simple sampler for the horseshoe estimator
Authors:
Enes Makalic,
Daniel F. Schmidt
Abstract:
In this note we derive a simple Bayesian sampler for linear regression with the horseshoe hierarchy. A new interpretation of the horseshoe model is presented, and extensions to logistic regression and alternative hierarchies, such as horseshoe$+$, are discussed. Due to the conjugacy of the proposed hierarchy, Chib's algorithm may be used to easily compute the marginal likelihood of the model.
In this note we derive a simple Bayesian sampler for linear regression with the horseshoe hierarchy. A new interpretation of the horseshoe model is presented, and extensions to logistic regression and alternative hierarchies, such as horseshoe$+$, are discussed. Due to the conjugacy of the proposed hierarchy, Chib's algorithm may be used to easily compute the marginal likelihood of the model.
△ Less
Submitted 28 September, 2015; v1 submitted 16 August, 2015;
originally announced August 2015.
-
Genome disorder and breast cancer susceptibility
Authors:
Conor Smyth,
Iva Špakulova,
Owen Cotton-Barratt,
Sajjad Rafiq,
William Tapper,
Rosanna Upstill-Goddard,
John L. Hopper,
Enes Makalic,
Daniel F. Schmidt,
Miroslav Kapuscinski,
Jörg Fliege,
Andrew Collins,
Jacek Brodzki,
Diana M. Eccles,
Ben D. MacArthur
Abstract:
Many common diseases have a complex genetic basis in which large numbers of genetic variations combine with environmental and lifestyle factors to determine risk. However, quantifying such polygenic effects and their relationship to disease risk has been challenging. In order to address these difficulties we developed a global measure of the information content of an individual's genome relative t…
▽ More
Many common diseases have a complex genetic basis in which large numbers of genetic variations combine with environmental and lifestyle factors to determine risk. However, quantifying such polygenic effects and their relationship to disease risk has been challenging. In order to address these difficulties we developed a global measure of the information content of an individual's genome relative to a reference population, which may be used to assess differences in global genome structure between cases and appropriate controls. Informally this measure, which we call relative genome information (RGI), quantifies the relative "disorder" of an individual's genome. In order to test its ability to predict disease risk we used RGI to compare single nucleotide polymorphism genotypes from two independent samples of women with early-onset breast cancer with three independent sets of controls. We found that RGI was significantly elevated in both sets of breast cancer cases in comparison with all three sets of controls, with disease risk rising sharply with RGI (odds ratio greater than 12 for the highest percentile RGI). Furthermore, we found that these differences are not due to associations with common variants at a small number of disease-associated loci, but rather are due to the combined associations of thousands of markers distributed throughout the genome. Our results indicate that the information content of an individual's genome may be used to measure the risk of a complex disease, and suggest that early-onset breast cancer has a strongly polygenic basis.
△ Less
Submitted 15 June, 2014;
originally announced June 2014.