-
Scalable expectation propagation for generalized linear models
Authors:
Niccolò Anceschi,
Augusto Fasano,
Beatrice Franzolini,
Giovanni Rebaudo
Abstract:
Generalized linear models (GLMs) arguably represent the standard approach for statistical regression beyond the Gaussian likelihood scenario. When Bayesian formulations are employed, the general absence of a tractable posterior distribution has motivated the development of deterministic approximations, which are generally more scalable than sampling techniques. Among them, expectation propagation…
▽ More
Generalized linear models (GLMs) arguably represent the standard approach for statistical regression beyond the Gaussian likelihood scenario. When Bayesian formulations are employed, the general absence of a tractable posterior distribution has motivated the development of deterministic approximations, which are generally more scalable than sampling techniques. Among them, expectation propagation (EP) showed extreme accuracy, usually higher than many variational Bayes solutions. However, the higher computational cost of EP posed concerns about its practical feasibility, especially in high-dimensional settings. We address these concerns by deriving a novel efficient formulation of EP for GLMs, whose cost scales linearly in the number of covariates p. This reduces the state-of-the-art O(p^2 n) per-iteration computational cost of the EP routine for GLMs to O(p n min{p,n}), with n being the sample size. We also show that, for binary models and log-linear GLMs approximate predictive means can be obtained at no additional cost. To preserve efficient moment matching for count data, we propose employing a combination of log-normal Laplace transform approximations, avoiding numerical integration. These novel results open the possibility of employing EP in settings that were believed to be practically impossible. Improvements over state-of-the-art approaches are illustrated both for simulated and real data. The efficient EP implementation is available at https://github.com/niccoloanceschi/EPglm.
△ Less
Submitted 2 July, 2024;
originally announced July 2024.
-
Expectation propagation for the smoothing distribution in dynamic probit
Authors:
Niccolò Anceschi,
Augusto Fasano,
Giovanni Rebaudo
Abstract:
The smoothing distribution of dynamic probit models with Gaussian state dynamics was recently proved to belong to the unified skew-normal family. Although this is computationally tractable in small-to-moderate settings, it may become computationally impractical in higher dimensions. In this work, adapting a recent more general class of expectation propagation (EP) algorithms, we derive an efficien…
▽ More
The smoothing distribution of dynamic probit models with Gaussian state dynamics was recently proved to belong to the unified skew-normal family. Although this is computationally tractable in small-to-moderate settings, it may become computationally impractical in higher dimensions. In this work, adapting a recent more general class of expectation propagation (EP) algorithms, we derive an efficient EP routine to perform inference for such a distribution. We show that the proposed approximation leads to accuracy gains over available approximate algorithms in a financial illustration.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Efficient computation of predictive probabilities in probit models via expectation propagation
Authors:
Augusto Fasano,
Niccolò Anceschi,
Beatrice Franzolini,
Giovanni Rebaudo
Abstract:
Binary regression models represent a popular model-based approach for binary classification. In the Bayesian framework, computational challenges in the form of the posterior distribution motivate still-ongoing fruitful research. Here, we focus on the computation of predictive probabilities in Bayesian probit models via expectation propagation (EP). Leveraging more general results in recent literat…
▽ More
Binary regression models represent a popular model-based approach for binary classification. In the Bayesian framework, computational challenges in the form of the posterior distribution motivate still-ongoing fruitful research. Here, we focus on the computation of predictive probabilities in Bayesian probit models via expectation propagation (EP). Leveraging more general results in recent literature, we show that such predictive probabilities admit a closed-form expression. Improvements over state-of-the-art approaches are shown in a simulation study.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Efficient expectation propagation for posterior approximation in high-dimensional probit models
Authors:
Augusto Fasano,
Niccolò Anceschi,
Beatrice Franzolini,
Giovanni Rebaudo
Abstract:
Bayesian binary regression is a prosperous area of research due to the computational challenges encountered by currently available methods either for high-dimensional settings or large datasets, or both. In the present work, we focus on the expectation propagation (EP) approximation of the posterior distribution in Bayesian probit regression under a multivariate Gaussian prior distribution. Adapti…
▽ More
Bayesian binary regression is a prosperous area of research due to the computational challenges encountered by currently available methods either for high-dimensional settings or large datasets, or both. In the present work, we focus on the expectation propagation (EP) approximation of the posterior distribution in Bayesian probit regression under a multivariate Gaussian prior distribution. Adapting more general derivations in Anceschi et al. (2023), we show how to leverage results on the extended multivariate skew-normal distribution to derive an efficient implementation of the EP routine having a per-iteration cost that scales linearly in the number of covariates. This makes EP computationally feasible also in challenging high-dimensional settings, as shown in a detailed simulation study.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Entropy regularization in probabilistic clustering
Authors:
Beatrice Franzolini,
Giovanni Rebaudo
Abstract:
Bayesian nonparametric mixture models are widely used to cluster observations. However, one major drawback of the approach is that the estimated partition often presents unbalanced clusters' frequencies with only a few dominating clusters and a large number of sparsely-populated ones. This feature translates into results that are often uninterpretable unless we accept to ignore a relevant number o…
▽ More
Bayesian nonparametric mixture models are widely used to cluster observations. However, one major drawback of the approach is that the estimated partition often presents unbalanced clusters' frequencies with only a few dominating clusters and a large number of sparsely-populated ones. This feature translates into results that are often uninterpretable unless we accept to ignore a relevant number of observations and clusters. Interpreting the posterior distribution as penalized likelihood, we show how the unbalance can be explained as a direct consequence of the cost functions involved in estimating the partition. In light of our findings, we propose a novel Bayesian estimator of the clustering configuration. The proposed estimator is equivalent to a post-processing procedure that reduces the number of sparsely-populated clusters and enhances interpretability. The procedure takes the form of entropy-regularization of the Bayesian estimate. While being computationally convenient with respect to alternative strategies, it is also theoretically justified as a correction to the Bayesian loss function used for point estimation and, as such, can be applied to any posterior distribution of clusters, regardless of the specific model used.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
Graph-Aligned Random Partition Model (GARP)
Authors:
Giovanni Rebaudo,
Peter Mueller
Abstract:
Bayesian nonparametric mixtures and random partition models are powerful tools for probabilistic clustering. However, standard independent mixture models can be restrictive in some applications such as inference on cell lineage due to the biological relations of the clusters. The increasing availability of large genomic data requires new statistical tools to perform model-based clustering and infe…
▽ More
Bayesian nonparametric mixtures and random partition models are powerful tools for probabilistic clustering. However, standard independent mixture models can be restrictive in some applications such as inference on cell lineage due to the biological relations of the clusters. The increasing availability of large genomic data requires new statistical tools to perform model-based clustering and infer the relationship between homogeneous subgroups of units. Motivated by single-cell RNA applications we develop a novel dependent mixture model to jointly perform cluster analysis and align the clusters on a graph. Our flexible graph-aligned random partition model (GARP) exploits Gibbs-type priors as building blocks, allowing us to derive analytical results on the graph-aligned random partition's probability mass function (pmf). We derive a generalization of the Chinese restaurant process from the pmf and a related efficient and neat MCMC algorithm to perform Bayesian inference. We perform posterior inference on real single-cell RNA data from mice stem cells. We further investigate the performance of our model in capturing the underlying clustering structure as well as the underlying graph by means of simulation studies.
△ Less
Submitted 17 May, 2024; v1 submitted 14 June, 2023;
originally announced June 2023.
-
Bayesian Mixed Multidimensional Scaling for Auditory Processing
Authors:
Giovanni Rebaudo,
Fernando Llanos,
Bharath Chandrasekaran,
Abhra Sarkar
Abstract:
The human brain distinguishes speech sound categories by representing acoustic signals in a latent multidimensional auditory-perceptual space. This space can be statistically constructed using multidimensional scaling, a technique that can compute lower-dimensional latent features representing the speech signals in such a way that their pairwise distances in the latent space closely resemble the c…
▽ More
The human brain distinguishes speech sound categories by representing acoustic signals in a latent multidimensional auditory-perceptual space. This space can be statistically constructed using multidimensional scaling, a technique that can compute lower-dimensional latent features representing the speech signals in such a way that their pairwise distances in the latent space closely resemble the corresponding distances in the observation space. The inter-individual and inter-population (e.g., native versus non-native listeners) heterogeneity in such representations is however not well understood. These questions have often been examined using joint analyses that ignore individual heterogeneity or using separate analyses that cannot characterize human similarities. Neither extreme, therefore, allows for principled comparisons between populations and individuals. The focus of the current literature has also often been on inference on latent distances between the categories and not on the latent features themselves, which are crucial for our applications, that make up these distances. Motivated by these problems, we develop a novel Bayesian mixed multidimensional scaling method, taking into account the heterogeneity across populations and subjects. We design a Markov chain Monte Carlo algorithm for posterior computation. We then recover the latent features using a post-processing scheme applied to the posterior samples. We evaluate the method's empirical performances through synthetic experiments. Applied to a motivating auditory neuroscience study, the method provides novel insights into how biologically interpretable lower-dimensional latent features reconstruct the observed distances between the stimuli and vary between individuals and their native language experiences.
△ Less
Submitted 1 December, 2023; v1 submitted 31 August, 2022;
originally announced September 2022.
-
Bayesian Inference for the Multinomial Probit Model under Gaussian Prior Distribution
Authors:
Augusto Fasano,
Giovanni Rebaudo,
Niccolò Anceschi
Abstract:
Multinomial probit (mnp) models are fundamental and widely-applied regression models for categorical data. Fasano and Durante (2022) proved that the class of unified skew-normal distributions is conjugate to several mnp sampling models. This allows to develop Monte Carlo samplers and accurate variational methods to perform Bayesian inference. In this paper, we adapt the abovementioned results for…
▽ More
Multinomial probit (mnp) models are fundamental and widely-applied regression models for categorical data. Fasano and Durante (2022) proved that the class of unified skew-normal distributions is conjugate to several mnp sampling models. This allows to develop Monte Carlo samplers and accurate variational methods to perform Bayesian inference. In this paper, we adapt the abovementioned results for a popular special case: the discrete-choice mnp model under zero mean and independent Gaussian priors. This allows to obtain simplified expressions for the parameters of the posterior distribution and an alternative derivation for the variational algorithm that gives a novel understanding of the fundamental results in Fasano and Durante (2022) as well as computational advantages in our special settings.
△ Less
Submitted 1 June, 2022;
originally announced June 2022.
-
Clustering consistency with Dirichlet process mixtures
Authors:
Filippo Ascolani,
Antonio Lijoi,
Giovanni Rebaudo,
Giacomo Zanella
Abstract:
Dirichlet process mixtures are flexible non-parametric models, particularly suited to density estimation and probabilistic clustering. In this work we study the posterior distribution induced by Dirichlet process mixtures as the sample size increases, and more specifically focus on consistency for the unknown number of clusters when the observed data are generated from a finite mixture. Crucially,…
▽ More
Dirichlet process mixtures are flexible non-parametric models, particularly suited to density estimation and probabilistic clustering. In this work we study the posterior distribution induced by Dirichlet process mixtures as the sample size increases, and more specifically focus on consistency for the unknown number of clusters when the observed data are generated from a finite mixture. Crucially, we consider the situation where a prior is placed on the concentration parameter of the underlying Dirichlet process. Previous findings in the literature suggest that Dirichlet process mixtures are typically not consistent for the number of clusters if the concentration parameter is held fixed and data come from a finite mixture. Here we show that consistency for the number of clusters can be achieved if the concentration parameter is adapted in a fully Bayesian way, as commonly done in practice. Our results are derived for data coming from a class of finite mixtures, with mild assumptions on the prior for the concentration parameter and for a variety of choices of likelihood kernels for the mixture.
△ Less
Submitted 25 May, 2022;
originally announced May 2022.
-
Flexible clustering via hidden hierarchical Dirichlet priors
Authors:
Antonio Lijoi,
Igor Prünster,
Giovanni Rebaudo
Abstract:
The Bayesian approach to inference stands out for naturally allowing borrowing information across heterogeneous populations, with different samples possibly sharing the same distribution. A popular Bayesian nonparametric model for clustering probability distributions is the nested Dirichlet process, which however has the drawback of grou** distributions in a single cluster when ties are observed…
▽ More
The Bayesian approach to inference stands out for naturally allowing borrowing information across heterogeneous populations, with different samples possibly sharing the same distribution. A popular Bayesian nonparametric model for clustering probability distributions is the nested Dirichlet process, which however has the drawback of grou** distributions in a single cluster when ties are observed across samples. With the goal of achieving a flexible and effective clustering method for both samples and observations, we investigate a nonparametric prior that arises as the composition of two different discrete random structures and derive a closed-form expression for the induced distribution of the random partition, the fundamental tool regulating the clustering behavior of the model. On the one hand, this allows to gain a deeper insight into the theoretical properties of the model and, on the other hand, it yields an MCMC algorithm for evaluating Bayesian inferences of interest. Moreover, we single out limitations of this algorithm when working with more than two populations and, consequently, devise an alternative more efficient sampling scheme, which as a by-product, allows testing homogeneity between different populations. Finally, we perform a comparison with the nested Dirichlet process and provide illustrative examples of both synthetic and real data.
△ Less
Submitted 18 January, 2022;
originally announced January 2022.
-
Separate Exchangeability as Modeling Principle in Bayesian Nonparametrics
Authors:
Giovanni Rebaudo,
Qiaohui Lin,
Peter Mueller
Abstract:
We argue for the use of separate exchangeability as a modeling principle in Bayesian nonparametric (BNP) inference. Separate exchangeability is \emph{de facto} widely applied in the Bayesian parametric case, e.g., it naturally arises in simple mixed models. However, while in some areas, such as random graphs, separate and (closely related) joint exchangeability are widely used, it is curiously und…
▽ More
We argue for the use of separate exchangeability as a modeling principle in Bayesian nonparametric (BNP) inference. Separate exchangeability is \emph{de facto} widely applied in the Bayesian parametric case, e.g., it naturally arises in simple mixed models. However, while in some areas, such as random graphs, separate and (closely related) joint exchangeability are widely used, it is curiously underused for several other applications in BNP. We briefly review the definition of separate exchangeability focusing on the implications of such a definition in Bayesian modeling. We then discuss two tractable classes of models that implement separate exchangeability that are the natural counterparts of familiar partially exchangeable BNP models.
The first is nested random partitions for a data matrix, defining a partition of columns and nested partitions of rows, nested within column clusters. Many recent models for nested partitions implement partially exchangeable models related to variations of the well-known nested Dirichlet process. We argue that inference under such models in some cases ignores important features of the experimental setup. We obtain the separately exchangeable counterpart of such partially exchangeable partition structures.
The second class is about setting up separately exchangeable priors for a nonparametric regression model when multiple sets of experimental units are involved. We highlight how a Dirichlet process mixture of linear models known as ANOVA DDP can naturally implement separate exchangeability in such regression problems. Finally, we illustrate how to perform inference under such models in two real data examples.
△ Less
Submitted 20 June, 2024; v1 submitted 14 December, 2021;
originally announced December 2021.
-
Variational Inference for the Smoothing Distribution in Dynamic Probit Models
Authors:
Augusto Fasano,
Giovanni Rebaudo
Abstract:
Recently, Fasano, Rebaudo, Durante and Petrone (2019) provided closed-form expressions for the filtering, predictive and smoothing distributions of multivariate dynamic probit models, leveraging on unified skew-normal distribution properties. This allows to develop algorithms to draw independent and identically distributed samples from such distributions, as well as sequential Monte Carlo procedur…
▽ More
Recently, Fasano, Rebaudo, Durante and Petrone (2019) provided closed-form expressions for the filtering, predictive and smoothing distributions of multivariate dynamic probit models, leveraging on unified skew-normal distribution properties. This allows to develop algorithms to draw independent and identically distributed samples from such distributions, as well as sequential Monte Carlo procedures for the filtering and predictive distributions, allowing to overcome computational bottlenecks that may arise for large sample sizes. In this paper, we briefly review the above-mentioned closed-form expressions, mainly focusing on the smoothing distribution of the univariate dynamic probit. We develop a variational Bayes approach, extending the partially factorized mean-field variational approximation introduced by Fasano, Durante and Zanella (2019) for the static binary probit model to the dynamic setting. Results are shown for a financial application.
△ Less
Submitted 18 April, 2021; v1 submitted 15 April, 2021;
originally announced April 2021.
-
A closed-form filter for binary time series
Authors:
Augusto Fasano,
Giovanni Rebaudo,
Daniele Durante,
Sonia Petrone
Abstract:
Non-Gaussian state-space models arise in several applications, and within this framework the binary time series setting provides a relevant example. However, unlike for Gaussian state-space models - where filtering, predictive and smoothing distributions are available in closed form - binary state-space models require approximations or sequential Monte Carlo strategies for inference and prediction…
▽ More
Non-Gaussian state-space models arise in several applications, and within this framework the binary time series setting provides a relevant example. However, unlike for Gaussian state-space models - where filtering, predictive and smoothing distributions are available in closed form - binary state-space models require approximations or sequential Monte Carlo strategies for inference and prediction. This is due to the apparent absence of conjugacy between the Gaussian states and the likelihood induced by the observation equation for the binary data. In this article we prove that the filtering, predictive and smoothing distributions in dynamic probit models with Gaussian state variables are, in fact, available and belong to a class of unified skew-normals (SUN) whose parameters can be updated recursively in time via analytical expressions. Also the key functionals of these distributions are, in principle, available, but their calculation requires the evaluation of multivariate Gaussian cumulative distribution functions. Leveraging SUN properties, we address this issue via novel Monte Carlo methods based on independent samples from the smoothing distribution, that can easily be adapted to the filtering and predictive case, thus improving state-of-the-art approximate and sequential Monte Carlo inference in small-to-moderate dimensional studies. Novel sequential Monte Carlo procedures that exploit the SUN properties are also developed to deal with online inference in high dimensions. Performance gains over competitors are outlined in a financial application.
△ Less
Submitted 18 May, 2021; v1 submitted 19 February, 2019;
originally announced February 2019.
