-
AutoEval Done Right: Using Synthetic Data for Model Evaluation
Authors:
Pierre Boyeau,
Anastasios N. Angelopoulos,
Nir Yosef,
Jitendra Malik,
Michael I. Jordan
Abstract:
The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of human annotations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this purpose that improve sample efficiency while remaining unbiased. These…
▽ More
The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of human annotations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this purpose that improve sample efficiency while remaining unbiased. These algorithms increase the effective human-labeled sample size by up to 50% on experiments with GPT-4.
△ Less
Submitted 28 May, 2024; v1 submitted 8 March, 2024;
originally announced March 2024.
-
Decision-Making with Auto-Encoding Variational Bayes
Authors:
Romain Lopez,
Pierre Boyeau,
Nir Yosef,
Michael I. Jordan,
Jeffrey Regier
Abstract:
To make decisions based on a model fit with auto-encoding variational Bayes (AEVB), practitioners often let the variational distribution serve as a surrogate for the posterior distribution. This approach yields biased estimates of the expected risk, and therefore leads to poor decisions for two reasons. First, the model fit with AEVB may not equal the underlying data distribution. Second, the vari…
▽ More
To make decisions based on a model fit with auto-encoding variational Bayes (AEVB), practitioners often let the variational distribution serve as a surrogate for the posterior distribution. This approach yields biased estimates of the expected risk, and therefore leads to poor decisions for two reasons. First, the model fit with AEVB may not equal the underlying data distribution. Second, the variational distribution may not equal the posterior distribution under the fitted model. We explore how fitting the variational distribution based on several objective functions other than the ELBO, while continuing to fit the generative model based on the ELBO, affects the quality of downstream decisions. For the probabilistic principal component analysis model, we investigate how importance sampling error, as well as the bias of the model parameter estimates, varies across several approximate posteriors when used as proposal distributions. Our theoretical results suggest that a posterior approximation distinct from the variational distribution should be used for making decisions. Motivated by these theoretical results, we propose learning several approximate proposals for the best model and combining them using multiple importance sampling for decision-making. In addition to toy examples, we present a full-fledged case study of single-cell RNA sequencing. In this challenging instance of multiple hypothesis testing, our proposed approach surpasses the current state of the art.
△ Less
Submitted 21 October, 2020; v1 submitted 17 February, 2020;
originally announced February 2020.
-
A joint model of unpaired data from scRNA-seq and spatial transcriptomics for imputing missing gene expression measurements
Authors:
Romain Lopez,
Achille Nazaret,
Maxime Langevin,
Jules Samaran,
Jeffrey Regier,
Michael I. Jordan,
Nir Yosef
Abstract:
Spatial studies of transcriptome provide biologists with gene expression maps of heterogeneous and complex tissues. However, most experimental protocols for spatial transcriptomics suffer from the need to select beforehand a small fraction of genes to be quantified over the entire transcriptome. Standard single-cell RNA sequencing (scRNA-seq) is more prevalent, easier to implement and can in princ…
▽ More
Spatial studies of transcriptome provide biologists with gene expression maps of heterogeneous and complex tissues. However, most experimental protocols for spatial transcriptomics suffer from the need to select beforehand a small fraction of genes to be quantified over the entire transcriptome. Standard single-cell RNA sequencing (scRNA-seq) is more prevalent, easier to implement and can in principle capture any gene but cannot recover the spatial location of the cells. In this manuscript, we focus on the problem of imputation of missing genes in spatial transcriptomic data based on (unpaired) standard scRNA-seq data from the same biological tissue. Building upon domain adaptation work, we propose gimVI, a deep generative model for the integration of spatial transcriptomic data and scRNA-seq data that can be used to impute missing genes. After describing our generative model and an inference procedure for it, we compare gimVI to alternative methods from computational biology or domain adaptation on real datasets and outperform Seurat Anchors, Liger and CORAL to impute held-out genes.
△ Less
Submitted 6 May, 2019;
originally announced May 2019.
-
A Deep Generative Model for Semi-Supervised Classification with Noisy Labels
Authors:
Maxime Langevin,
Edouard Mehlman,
Jeffrey Regier,
Romain Lopez,
Michael I. Jordan,
Nir Yosef
Abstract:
Class labels are often imperfectly observed, due to mistakes and to genuine ambiguity among classes. We propose a new semi-supervised deep generative model that explicitly models noisy labels, called the Mislabeled VAE (M-VAE). The M-VAE can perform better than existing deep generative models which do not account for label noise. Additionally, the derivation of M-VAE gives new theoretical insights…
▽ More
Class labels are often imperfectly observed, due to mistakes and to genuine ambiguity among classes. We propose a new semi-supervised deep generative model that explicitly models noisy labels, called the Mislabeled VAE (M-VAE). The M-VAE can perform better than existing deep generative models which do not account for label noise. Additionally, the derivation of M-VAE gives new theoretical insights into the popular M1+M2 semi-supervised model.
△ Less
Submitted 16 September, 2018;
originally announced September 2018.
-
Information Constraints on Auto-Encoding Variational Bayes
Authors:
Romain Lopez,
Jeffrey Regier,
Michael I. Jordan,
Nir Yosef
Abstract:
Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on Auto-Encoding Variational Bayes and w…
▽ More
Parameterizing the approximate posterior of a generative model with neural networks has become a common theme in recent machine learning research. While providing appealing flexibility, this approach makes it difficult to impose or assess structural constraints such as conditional independence. We propose a framework for learning representations that relies on Auto-Encoding Variational Bayes and whose search space is constrained via kernel-based measures of independence. In particular, our method employs the $d$-variable Hilbert-Schmidt Independence Criterion (dHSIC) to enforce independence between the latent representations and arbitrary nuisance factors. We show how to apply this method to a range of problems, including the problems of learning invariant representations and the learning of interpretable representations. We also present a full-fledged application to single-cell RNA sequencing (scRNA-seq). In this setting the biological signal is mixed in complex ways with sequencing errors and sampling effects. We show that our method out-performs the state-of-the-art in this domain.
△ Less
Submitted 28 November, 2018; v1 submitted 22 May, 2018;
originally announced May 2018.
-
A deep generative model for single-cell RNA sequencing with application to detecting differentially expressed genes
Authors:
Romain Lopez,
Jeffrey Regier,
Michael Cole,
Michael Jordan,
Nir Yosef
Abstract:
We propose a probabilistic model for interpreting gene expression levels that are observed through single-cell RNA sequencing. In the model, each cell has a low-dimensional latent representation. Additional latent variables account for technical effects that may erroneously set some observations of gene expression levels to zero. Conditional distributions are specified by neural networks, giving t…
▽ More
We propose a probabilistic model for interpreting gene expression levels that are observed through single-cell RNA sequencing. In the model, each cell has a low-dimensional latent representation. Additional latent variables account for technical effects that may erroneously set some observations of gene expression levels to zero. Conditional distributions are specified by neural networks, giving the proposed model enough flexibility to fit the data well. We use variational inference and stochastic optimization to approximate the posterior distribution. The inference procedure scales to over one million cells, whereas competing algorithms do not. Even for smaller datasets, for several tasks, the proposed procedure outperforms state-of-the-art methods like ZIFA and ZINB-WaVE. We also extend our framework to take into account batch effects and other confounding factors and propose a natural Bayesian hypothesis framework for differential expression that outperforms tradition DESeq2.
△ Less
Submitted 16 October, 2017; v1 submitted 13 October, 2017;
originally announced October 2017.
-
A deep generative model for gene expression profiles from single-cell RNA sequencing
Authors:
Romain Lopez,
Jeffrey Regier,
Michael Cole,
Michael Jordan,
Nir Yosef
Abstract:
We propose a probabilistic model for interpreting gene expression levels that are observed through single-cell RNA sequencing. In the model, each cell has a low-dimensional latent representation. Additional latent variables account for technical effects that may erroneously set some observations of gene expression levels to zero. Conditional distributions are specified by neural networks, giving t…
▽ More
We propose a probabilistic model for interpreting gene expression levels that are observed through single-cell RNA sequencing. In the model, each cell has a low-dimensional latent representation. Additional latent variables account for technical effects that may erroneously set some observations of gene expression levels to zero. Conditional distributions are specified by neural networks, giving the proposed model enough flexibility to fit the data well. We use variational inference and stochastic optimization to approximate the posterior distribution. The inference procedure scales to over one million cells, whereas competing algorithms do not. Even for smaller datasets, for several tasks, the proposed procedure outperforms state-of-the-art methods like ZIFA and ZINB-WaVE. We also extend our framework to account for batch effects and other confounding factors, and propose a Bayesian hypothesis test for differential expression that outperforms DESeq2.
△ Less
Submitted 16 January, 2018; v1 submitted 7 September, 2017;
originally announced September 2017.
-
Steiner Network Problems on Temporal Graphs
Authors:
Alex Khodaverdian,
Benjamin Weitz,
Jimmy Wu,
Nir Yosef
Abstract:
We introduce a temporal Steiner network problem in which a graph, as well as changes to its edges and/or vertices over a set of discrete times, are given as input; the goal is to find a minimal subgraph satisfying a set of $k$ time-sensitive connectivity demands. We show that this problem, $k$-Temporal Steiner Network ($k$-TSN), is NP-hard to approximate to a factor of $k - ε$, for every fixed…
▽ More
We introduce a temporal Steiner network problem in which a graph, as well as changes to its edges and/or vertices over a set of discrete times, are given as input; the goal is to find a minimal subgraph satisfying a set of $k$ time-sensitive connectivity demands. We show that this problem, $k$-Temporal Steiner Network ($k$-TSN), is NP-hard to approximate to a factor of $k - ε$, for every fixed $k \geq 2$ and $ε> 0$. This bound is tight, as certified by a trivial approximation algorithm. Conceptually this demonstrates, in contrast to known results for traditional Steiner problems, that a time dimension adds considerable complexity even when the problem is offline.
We also discuss special cases of $k$-TSN in which the graph changes satisfy a monotonicity property. We show approximation-preserving reductions from monotonic $k$-TSN to well-studied problems such as Priority Steiner Tree and Directed Steiner Tree, implying improved approximation algorithms.
Lastly, $k$-TSN and its variants arise naturally in computational biology; to facilitate such applications, we devise an integer linear program for $k$-TSN based on network flows.
△ Less
Submitted 31 August, 2017; v1 submitted 16 September, 2016;
originally announced September 2016.
