-
Conformal Load Prediction with Transductive Graph Autoencoders
Authors:
Rui Luo,
Nicolo Colombo
Abstract:
Predicting edge weights on graphs has various applications, from transportation systems to social networks. This paper describes a Graph Neural Network (GNN) approach for edge weight prediction with guaranteed coverage. We leverage conformal prediction to calibrate the GNN outputs and produce valid prediction intervals. We handle data heteroscedasticity through error reweighting and Conformalized…
▽ More
Predicting edge weights on graphs has various applications, from transportation systems to social networks. This paper describes a Graph Neural Network (GNN) approach for edge weight prediction with guaranteed coverage. We leverage conformal prediction to calibrate the GNN outputs and produce valid prediction intervals. We handle data heteroscedasticity through error reweighting and Conformalized Quantile Regression (CQR). We compare the performance of our method against baseline techniques on real-world transportation datasets. Our approach has better coverage and efficiency than all baselines and showcases robustness and adaptability.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Structured Learning of Compositional Sequential Interventions
Authors:
Jialin Yu,
Andreas Koukorinis,
Nicolò Colombo,
Yuchen Zhu,
Ricardo Silva
Abstract:
We consider sequential treatment regimes where each unit is exposed to combinations of interventions over time. When interventions are described by qualitative labels, such as ``close schools for a month due to a pandemic'' or ``promote this podcast to this user during this week'', it is unclear which appropriate structural assumptions allow us to generalize behavioral predictions to previously un…
▽ More
We consider sequential treatment regimes where each unit is exposed to combinations of interventions over time. When interventions are described by qualitative labels, such as ``close schools for a month due to a pandemic'' or ``promote this podcast to this user during this week'', it is unclear which appropriate structural assumptions allow us to generalize behavioral predictions to previously unseen combinatorial sequences. Standard black-box approaches map** sequences of categorical variables to outputs are applicable, but they rely on poorly understood assumptions on how reliable generalization can be obtained, and may underperform under sparse sequences, temporal variability, and large action spaces. To approach that, we pose an explicit model for \emph{composition}, that is, how the effect of sequential interventions can be isolated into modules, clarifying which data conditions allow for the identification of their combined effect at different units and time steps. We show the identification properties of our compositional model, inspired by advances in causal matrix factorization methods but focusing on predictive models for novel compositions of interventions instead of matrix completion tasks and causal effect estimation. We compare our approach to flexible but generic black-box models to illustrate how structure aids prediction in sparse data conditions.
△ Less
Submitted 9 June, 2024;
originally announced June 2024.
-
Normalizing Flows for Conformal Regression
Authors:
Nicolo Colombo
Abstract:
Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. The same calibration scheme usually applies to any model and data without modifications. The obtained prediction intervals are valid by construction but could be inefficient, i.e. unnecessarily big, if the prediction errors are not uniformly distributed over the input spa…
▽ More
Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. The same calibration scheme usually applies to any model and data without modifications. The obtained prediction intervals are valid by construction but could be inefficient, i.e. unnecessarily big, if the prediction errors are not uniformly distributed over the input space.
We present a general scheme to localize the intervals by training the calibration process. The standard prediction error is replaced by an optimized distance metric that depends explicitly on the object attributes. Learning the optimal metric is equivalent to training a Normalizing Flow that acts on the joint distribution of the errors and the inputs. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity. The approach is compatible with existing locally-adaptive CP strategies based on re-weighting the calibration samples and applies to any point-prediction model without retraining.
△ Less
Submitted 26 June, 2024; v1 submitted 5 June, 2024;
originally announced June 2024.
-
Calibrating gravitational-wave search algorithms with conformal prediction
Authors:
Gregory Ashton,
Nicolo Colombo,
Ian Harry,
Surabhi Sachdev
Abstract:
In astronomy, we frequently face the decision problem: does this data contain a signal? Typically, a statistical approach is used, which requires a threshold. The choice of threshold presents a common challenge in settings where signals and noise must be delineated, but their distributions overlap. Gravitational-wave astronomy, which has gone from the first discovery to catalogues of hundreds of e…
▽ More
In astronomy, we frequently face the decision problem: does this data contain a signal? Typically, a statistical approach is used, which requires a threshold. The choice of threshold presents a common challenge in settings where signals and noise must be delineated, but their distributions overlap. Gravitational-wave astronomy, which has gone from the first discovery to catalogues of hundreds of events in less than a decade, presents a fascinating case study. For signals from colliding compact objects, the field has evolved from a frequentist to a Bayesian methodology. However, the issue of choosing a threshold and validating noise contamination in a catalogue persists. Confusion and debate often arise due to the misapplication of statistical concepts, the complicated nature of the detection statistics, and the inclusion of astrophysical background models. We introduce Conformal Prediction (CP), a framework developed in Machine Learning to provide distribution-free uncertainty quantification to point predictors. We show that CP can be viewed as an extension of the traditional statistical frameworks whereby thresholds are calibrated such that the uncertainty intervals are statistically rigorous and the error rate can be validated. Moreover, we discuss how CP offers a framework to optimally build a meta-pipeline combining the outputs from multiple independent searches. We introduce CP with a toy cosmic-ray detector, which captures the salient features of most astrophysical search problems and allows us to demonstrate the features of CP in a simple context. We then apply the approach to a recent gravitational-wave Mock Data Challenge using multiple search algorithms for compact binary coalescence signals in interferometric gravitational-wave data. Finally, we conclude with a discussion on the future potential of the method for gravitational-wave astronomy.
△ Less
Submitted 6 June, 2024; v1 submitted 29 February, 2024;
originally announced February 2024.
-
On training locally adaptive CP
Authors:
Nicolo Colombo
Abstract:
We address the problem of making Conformal Prediction (CP) intervals locally adaptive. Most existing methods focus on approximating the object-conditional validity of the intervals by partitioning or re-weighting the calibration set. Our strategy is new and conceptually different. Instead of re-weighting the calibration data, we redefine the conformity measure through a trainable change of variabl…
▽ More
We address the problem of making Conformal Prediction (CP) intervals locally adaptive. Most existing methods focus on approximating the object-conditional validity of the intervals by partitioning or re-weighting the calibration set. Our strategy is new and conceptually different. Instead of re-weighting the calibration data, we redefine the conformity measure through a trainable change of variables, $A \to φ_X(A)$, that depends explicitly on the object attributes, $X$. Under certain conditions and if $φ_X$ is monotonic in $A$ for any $X$, the transformations produce prediction intervals that are guaranteed to be marginally valid and have $X$-dependent sizes. We describe how to parameterize and train $φ_X$ to maximize the interval efficiency. Contrary to other CP-aware training methods, the objective function is smooth and can be minimized through standard gradient methods without approximations.
△ Less
Submitted 5 June, 2023;
originally announced June 2023.
-
Differentiable Architecture Pruning for Transfer Learning
Authors:
Nicolo Colombo,
Yang Gao
Abstract:
We propose a new gradient-based approach for extracting sub-architectures from a given large model. Contrarily to existing pruning methods, which are unable to disentangle the network architecture and the corresponding weights, our architecture-pruning scheme produces transferable new structures that can be successfully retrained to solve different tasks. We focus on a transfer-learning setup wher…
▽ More
We propose a new gradient-based approach for extracting sub-architectures from a given large model. Contrarily to existing pruning methods, which are unable to disentangle the network architecture and the corresponding weights, our architecture-pruning scheme produces transferable new structures that can be successfully retrained to solve different tasks. We focus on a transfer-learning setup where architectures can be trained on a large data set but very few data points are available for fine-tuning them on new tasks. We define a new gradient-based algorithm that trains architectures of arbitrarily low complexity independently from the attached weights. Given a search space defined by an existing large neural model, we reformulate the architecture search task as a complexity-penalized subset-selection problem and solve it through a two-temperature relaxation scheme. We provide theoretical convergence guarantees and validate the proposed transfer-learning strategy on real data.
△ Less
Submitted 7 July, 2021;
originally announced July 2021.
-
Adapting by Pruning: A Case Study on BERT
Authors:
Yang Gao,
Nicolo Colombo,
Wei Wang
Abstract:
Adapting pre-trained neural models to downstream tasks has become the standard practice for obtaining high-quality models. In this work, we propose a novel model adaptation paradigm, adapting by pruning, which prunes neural connections in the pre-trained model to optimise the performance on the target task; all remaining connections have their weights intact. We formulate adapting-by-pruning as an…
▽ More
Adapting pre-trained neural models to downstream tasks has become the standard practice for obtaining high-quality models. In this work, we propose a novel model adaptation paradigm, adapting by pruning, which prunes neural connections in the pre-trained model to optimise the performance on the target task; all remaining connections have their weights intact. We formulate adapting-by-pruning as an optimisation problem with a differentiable loss and propose an efficient algorithm to prune the model. We prove that the algorithm is near-optimal under standard assumptions and apply the algorithm to adapt BERT to some GLUE tasks. Results suggest that our method can prune up to 50% weights in BERT while yielding similar performance compared to the fine-tuned full model. We also compare our method with other state-of-the-art pruning methods and study the topological differences of their obtained sub-networks.
△ Less
Submitted 7 May, 2021;
originally announced May 2021.
-
Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification
Authors:
Nicolo Colombo,
Yang Gao
Abstract:
The history of deep learning has shown that human-designed problem-specific networks can greatly improve the classification performance of general neural models. In most practical cases, however, choosing the optimal architecture for a given task remains a challenging problem. Recent architecture-search methods are able to automatically build neural models with strong performance but fail to fully…
▽ More
The history of deep learning has shown that human-designed problem-specific networks can greatly improve the classification performance of general neural models. In most practical cases, however, choosing the optimal architecture for a given task remains a challenging problem. Recent architecture-search methods are able to automatically build neural models with strong performance but fail to fully appreciate the interaction between neural architecture and weights. This work investigates the problem of disentangling the role of the neural structure and its edge weights, by showing that well-trained architectures may not need any link-specific fine-tuning of the weights. We compare the performance of such weight-free networks (in our case these are binary networks with {0, 1}-valued weights) with random, weight-agnostic, pruned and standard fully connected networks. To find the optimal weight-agnostic network, we use a novel and computationally efficient method that translates the hard architecture-search problem into a feasible optimization problem.More specifically, we look at the optimal task-specific architectures as the optimal configuration of binary networks with {0, 1}-valued weights, which can be found through an approximate gradient descent strategy. Theoretical convergence guarantees of the proposed algorithm are obtained by bounding the error in the gradient approximation and its practical performance is evaluated on two real-world data sets. For measuring the structural similarities between different architectures, we use a novel spectral approach that allows us to underline the intrinsic differences between real-valued networks and weight-free architectures.
△ Less
Submitted 11 September, 2020;
originally announced September 2020.
-
Training conformal predictors
Authors:
Nicolo Colombo,
Vladimir Vovk
Abstract:
Efficiency criteria for conformal prediction, such as \emph{observed fuzziness} (i.e., the sum of p-values associated with false labels), are commonly used to \emph{evaluate} the performance of given conformal predictors. Here, we investigate whether it is possible to exploit efficiency criteria to \emph{learn} classifiers, both conformal predictors and point classifiers, by using such criteria as…
▽ More
Efficiency criteria for conformal prediction, such as \emph{observed fuzziness} (i.e., the sum of p-values associated with false labels), are commonly used to \emph{evaluate} the performance of given conformal predictors. Here, we investigate whether it is possible to exploit efficiency criteria to \emph{learn} classifiers, both conformal predictors and point classifiers, by using such criteria as training objective functions. The proposed idea is implemented for the problem of binary classification of hand-written digits. By choosing a 1-dimensional model class (with one real-valued free parameter), we can solve the optimization problems through an (approximate) exhaustive search over (a discrete version of) the parameter space. Our empirical results suggest that conformal predictors trained by minimizing their observed fuzziness perform better than conformal predictors trained in the traditional way by minimizing the \emph{prediction error} of the corresponding point classifier. They also have a reasonable performance in terms of their prediction error on the test set.
△ Less
Submitted 14 May, 2020;
originally announced May 2020.
-
Multiple Metric Learning for Structured Data
Authors:
Nicolo Colombo
Abstract:
We address the problem of merging graph and feature-space information while learning a metric from structured data. Existing algorithms tackle the problem in an asymmetric way, by either extracting vectorized summaries of the graph structure or adding hard constraints to feature-space algorithms. Following a different path, we define a metric regression scheme where we train metric-constrained lin…
▽ More
We address the problem of merging graph and feature-space information while learning a metric from structured data. Existing algorithms tackle the problem in an asymmetric way, by either extracting vectorized summaries of the graph structure or adding hard constraints to feature-space algorithms. Following a different path, we define a metric regression scheme where we train metric-constrained linear combinations of dissimilarity matrices. The idea is that the input matrices can be pre-computed dissimilarity measures obtained from any kind of available data (e.g. node attributes or edge structure). As the model inputs are distance measures, we do not need to assume the existence of any underlying feature space. Main challenge is that metric constraints (especially positive-definiteness and sub-additivity), are not automatically respected if, for example, the coefficients of the linear combination are allowed to be negative. Both positive and sub-additive constraints are linear inequalities, but the computational complexity of imposing them scales as O(D3), where D is the size of the input matrices (i.e. the size of the data set). This becomes quickly prohibitive, even when D is relatively small. We propose a new graph-based technique for optimizing under such constraints and show that, in some cases, our approach may reduce the original computational complexity of the optimization process by one order of magnitude. Contrarily to existing methods, our scheme applies to any (possibly non-convex) metric-constrained objective function.
△ Less
Submitted 13 February, 2020;
originally announced February 2020.
-
Counterfactual Distribution Regression for Structured Inference
Authors:
Nicolo Colombo,
Ricardo Silva,
Soong M Kang,
Arthur Gretton
Abstract:
We consider problems in which a system receives external \emph{perturbations} from time to time. For instance, the system can be a train network in which particular lines are repeatedly disrupted without warning, having an effect on passenger behavior. The goal is to predict changes in the behavior of the system at particular points of interest, such as passenger traffic around stations at the aff…
▽ More
We consider problems in which a system receives external \emph{perturbations} from time to time. For instance, the system can be a train network in which particular lines are repeatedly disrupted without warning, having an effect on passenger behavior. The goal is to predict changes in the behavior of the system at particular points of interest, such as passenger traffic around stations at the affected rails. We assume that the data available provides records of the system functioning at its "natural regime" (e.g., the train network without disruptions) and data on cases where perturbations took place. The inference problem is how information concerning perturbations, with particular covariates such as location and time, can be generalized to predict the effect of novel perturbations. We approach this problem from the point of view of a map** from the counterfactual distribution of the system behavior without disruptions to the distribution of the disrupted system. A variant on \emph{distribution regression} is developed for this setup.
△ Less
Submitted 20 August, 2019;
originally announced August 2019.
-
Bayesian Semi-supervised Learning with Graph Gaussian Processes
Authors:
Yin Cheng Ng,
Nicolo Colombo,
Ricardo Silva
Abstract:
We propose a data-efficient Gaussian process-based Bayesian approach to the semi-supervised learning problem on graphs. The proposed model shows extremely competitive performance when compared to the state-of-the-art graph neural networks on semi-supervised learning benchmark experiments, and outperforms the neural networks in active learning experiments where labels are scarce. Furthermore, the m…
▽ More
We propose a data-efficient Gaussian process-based Bayesian approach to the semi-supervised learning problem on graphs. The proposed model shows extremely competitive performance when compared to the state-of-the-art graph neural networks on semi-supervised learning benchmark experiments, and outperforms the neural networks in active learning experiments where labels are scarce. Furthermore, the model does not require a validation data set for early stop** to control over-fitting. Our model can be viewed as an instance of empirical distribution regression weighted locally by network connectivity. We further motivate the intuitive construction of the model with a Bayesian linear model interpretation where the node features are filtered by an operator related to the graph Laplacian. The method can be easily implemented by adapting off-the-shelf scalable variational inference algorithms for Gaussian processes.
△ Less
Submitted 12 October, 2018; v1 submitted 12 September, 2018;
originally announced September 2018.
-
Experimental design trade-offs for gene regulatory network inference: an in silico study of the yeast Saccharomyces cerevisiae cell cycle
Authors:
Johan Markdahl,
Nicolo Colombo,
Johan Thunberg,
Jorge Goncalves
Abstract:
Time-series of high throughput gene sequencing data intended for gene regulatory network (GRN) inference are often short due to the high costs of sampling cell systems. Moreover, experimentalists lack a set of quantitative guidelines that prescribe the minimal number of samples required to infer a reliable GRN model. We study the temporal resolution of data vs quality of GRN inference in order to…
▽ More
Time-series of high throughput gene sequencing data intended for gene regulatory network (GRN) inference are often short due to the high costs of sampling cell systems. Moreover, experimentalists lack a set of quantitative guidelines that prescribe the minimal number of samples required to infer a reliable GRN model. We study the temporal resolution of data vs quality of GRN inference in order to ultimately overcome this deficit. The evolution of a Markovian jump process model for the Ras/cAMP/PKA pathway of proteins and metabolites in the G1 phase of the Saccharomyces cerevisiae cell cycle is sampled at a number of different rates. For each time-series we infer a linear regression model of the GRN using the LASSO method. The inferred network topology is evaluated in terms of the area under the precision-recall curve AUPR. By plotting the AUPR against the number of samples, we show that the trade-off has a, roughly speaking, sigmoid shape. An optimal number of samples corresponds to values on the ridge of the sigmoid.
△ Less
Submitted 14 December, 2017;
originally announced December 2017.
-
Approximate Joint Matrix Triangularization
Authors:
Nicolo Colombo,
Nikos Vlassis
Abstract:
We consider the problem of approximate joint triangularization of a set of noisy jointly diagonalizable real matrices. Approximate joint triangularizers are commonly used in the estimation of the joint eigenstructure of a set of matrices, with applications in signal processing, linear algebra, and tensor decomposition. By assuming the input matrices to be perturbations of noise-free, simultaneousl…
▽ More
We consider the problem of approximate joint triangularization of a set of noisy jointly diagonalizable real matrices. Approximate joint triangularizers are commonly used in the estimation of the joint eigenstructure of a set of matrices, with applications in signal processing, linear algebra, and tensor decomposition. By assuming the input matrices to be perturbations of noise-free, simultaneously diagonalizable ground-truth matrices, the approximate joint triangularizers are expected to be perturbations of the exact joint triangularizers of the ground-truth matrices. We provide a priori and a posteriori perturbation bounds on the `distance' between an approximate joint triangularizer and its exact counterpart. The a priori bounds are theoretical inequalities that involve functions of the ground-truth matrices and noise matrices, whereas the a posteriori bounds are given in terms of observable quantities that can be computed from the input matrices. From a practical perspective, the problem of finding the best approximate joint triangularizer of a set of noisy matrices amounts to solving a nonconvex optimization problem. We show that, under a condition on the noise level of the input matrices, it is possible to find a good initial triangularizer such that the solution obtained by any local descent-type algorithm has certain global guarantees. Finally, we discuss the application of approximate joint matrix triangularization to canonical tensor decomposition and we derive novel estimation error bounds.
△ Less
Submitted 2 July, 2016;
originally announced July 2016.
-
Spectral Sequence Motif Discovery
Authors:
Nicolò Colombo,
Nikos Vlassis
Abstract:
Sequence discovery tools play a central role in several fields of computational biology. In the framework of Transcription Factor binding studies, motif finding algorithms of increasingly high performance are required to process the big datasets produced by new high-throughput sequencing technologies. Most existing algorithms are computationally demanding and often cannot support the large size of…
▽ More
Sequence discovery tools play a central role in several fields of computational biology. In the framework of Transcription Factor binding studies, motif finding algorithms of increasingly high performance are required to process the big datasets produced by new high-throughput sequencing technologies. Most existing algorithms are computationally demanding and often cannot support the large size of new experimental data. We present a new motif discovery algorithm that is built on a recent machine learning technique, referred to as Method of Moments. Based on spectral decompositions, this method is robust under model misspecification and is not prone to locally optimal solutions. We obtain an algorithm that is extremely fast and designed for the analysis of big sequencing data. In a few minutes, we can process datasets of hundreds of thousand sequences and extract motif profiles that match those computed by various state-of-the-art algorithms.
△ Less
Submitted 26 August, 2014; v1 submitted 23 July, 2014;
originally announced July 2014.
-
Higher Spin Gravity Amplitudes From Zero-form Charges
Authors:
Nicolò Colombo,
Per Sundell
Abstract:
We examine zero-form charges in Vasiliev's four-dimensional bosonic higher spin gravities. These are classical observables given by integrals over noncommutative twistor space of adjoint combinations of the zero-form master fields, including insertions of delta functions in the deformed oscillators serving as gauge invariant regulators. The regularized charges admit perturbative expansions in term…
▽ More
We examine zero-form charges in Vasiliev's four-dimensional bosonic higher spin gravities. These are classical observables given by integrals over noncommutative twistor space of adjoint combinations of the zero-form master fields, including insertions of delta functions in the deformed oscillators serving as gauge invariant regulators. The regularized charges admit perturbative expansions in terms of multi-linear functionals in the Weyl zero-form, which are Bose symmetric and higher spin invariant by construction, and that can be interpreted as basic building blocks for higher spin gravity amplitudes. We compute two- and three-point functions by attaching external legs given by unfolded bulk-to-boundary propagators, and identify the result with the two- and three-current correlation functions in theories of free conformal scalars and fermions in three dimensions. Modulo assumptions on the structure of the sub-leading corrections, and relying on the generalized Hamiltonian off-shell formulation, we are thus led to propose an expression for the free energy as a sum of suitably normalized zero-form charges
△ Less
Submitted 11 February, 2013; v1 submitted 19 August, 2012;
originally announced August 2012.
-
A minimal BV action for Vasiliev's four-dimensional higher spin gravity
Authors:
Nicolas Boulanger,
Nicolo Colombo,
Per Sundell
Abstract:
The action principle for Vasiliev's four-dimensional higher-spin gravity proposed recently by two of the authors, is converted into a minimal BV master action using the AKSZ procedure, which amounts to replacing the classical differential forms by vectorial superfields of fixed total degree given by the sum of form degree and ghost number. The nilpotency of the BRST operator is achieved by imposin…
▽ More
The action principle for Vasiliev's four-dimensional higher-spin gravity proposed recently by two of the authors, is converted into a minimal BV master action using the AKSZ procedure, which amounts to replacing the classical differential forms by vectorial superfields of fixed total degree given by the sum of form degree and ghost number. The nilpotency of the BRST operator is achieved by imposing boundary conditions and choosing appropriate gauge transitions between charts leading to a globally-defined formulation based on a principal bundle.
△ Less
Submitted 14 February, 2013; v1 submitted 15 May, 2012;
originally announced May 2012.
-
Twistor space observables and quasi-amplitudes in 4D higher spin gravity
Authors:
Nicolo Colombo,
Per Sundell
Abstract:
Vasiliev equations facilitate globally defined formulations of higher-spin gravity in various correspondence spaces associated with different phases of the theory. In the four-dimensional case this induces a map from a generally covariant formulation in spacetime with higher-derivative interactions to a formulation in terms of a deformed symplectic structure on a noncommutative doubled twistor spa…
▽ More
Vasiliev equations facilitate globally defined formulations of higher-spin gravity in various correspondence spaces associated with different phases of the theory. In the four-dimensional case this induces a map from a generally covariant formulation in spacetime with higher-derivative interactions to a formulation in terms of a deformed symplectic structure on a noncommutative doubled twistor space, sending spacetime boundary conditions to various sectors of an associative star-product algebra. We look at observables given by integrals over twistor space defining composite zero-forms in spacetime that do not break any local symmetries and that are closed on shell. They can be evaluated locally in spacetime and interpreted as building blocks for dual amplitudes. To regularize potential twistor-space divergencies arising in their curvature expansion, we propose a closed-contour prescription that respects associativity and hence higher-spin gauge symmetry. As a sample calculation, we examine next-to-leading corrections to quasi-amplitudes for twistor-space plane waves, and find cancellations that we interpret using transgression properties in twistor space.
△ Less
Submitted 6 May, 2011; v1 submitted 3 December, 2010;
originally announced December 2010.
