-
A unified framework for learning with nonlinear model classes from arbitrary linear samples
Authors:
Ben Adcock,
Juan M. Cardenas,
Nick Dexter
Abstract:
This work considers the fundamental problem of learning an unknown object from training data using a given model class. We introduce a unified framework that allows for objects in arbitrary Hilbert spaces, general types of (random) linear measurements as training data and general types of nonlinear model classes. We establish a series of learning guarantees for this framework. These guarantees pro…
▽ More
This work considers the fundamental problem of learning an unknown object from training data using a given model class. We introduce a unified framework that allows for objects in arbitrary Hilbert spaces, general types of (random) linear measurements as training data and general types of nonlinear model classes. We establish a series of learning guarantees for this framework. These guarantees provide explicit relations between the amount of training data and properties of the model class to ensure near-best generalization bounds. In doing so, we also introduce and develop the key notion of the variation of a model class with respect to a distribution of sampling operators. To exhibit the versatility of this framework, we show that it can accommodate many different types of well-known problems of interest. We present examples such as matrix sketching by random sampling, compressed sensing with isotropic vectors, active learning in regression and compressed sensing with generative models. In all cases, we show how known results become straightforward corollaries of our general learning guarantees. For compressed sensing with generative models, we also present a number of generalizations and improvements of recent results. In summary, our work not only introduces a unified way to study learning unknown objects from general types of data, but also establishes a series of general theoretical guarantees which consolidate and improve various known results.
△ Less
Submitted 24 November, 2023;
originally announced November 2023.
-
CS4ML: A general framework for active learning with arbitrary data based on Christoffel functions
Authors:
Ben Adcock,
Juan M. Cardenas,
Nick Dexter
Abstract:
We introduce a general framework for active learning in regression problems. Our framework extends the standard setup by allowing for general types of data, rather than merely pointwise samples of the target function. This generalization covers many cases of practical interest, such as data acquired in transform domains (e.g., Fourier data), vector-valued data (e.g., gradient-augmented data), data…
▽ More
We introduce a general framework for active learning in regression problems. Our framework extends the standard setup by allowing for general types of data, rather than merely pointwise samples of the target function. This generalization covers many cases of practical interest, such as data acquired in transform domains (e.g., Fourier data), vector-valued data (e.g., gradient-augmented data), data acquired along continuous curves, and, multimodal data (i.e., combinations of different types of measurements). Our framework considers random sampling according to a finite number of sampling measures and arbitrary nonlinear approximation spaces (model classes). We introduce the concept of generalized Christoffel functions and show how these can be used to optimize the sampling measures. We prove that this leads to near-optimal sample complexity in various important cases. This paper focuses on applications in scientific computing, where active learning is often desirable, since it is usually expensive to generate data. We demonstrate the efficacy of our framework for gradient-augmented learning with polynomials, Magnetic Resonance Imaging (MRI) using generative models and adaptive sampling for solving PDEs using Physics-Informed Neural Networks (PINNs).
△ Less
Submitted 7 December, 2023; v1 submitted 1 June, 2023;
originally announced June 2023.
-
CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learning
Authors:
Ben Adcock,
Juan M. Cardenas,
Nick Dexter
Abstract:
The problem of approximating smooth, multivariate functions from sample points arises in many applications in scientific computing, e.g., in computational Uncertainty Quantification (UQ) for science and engineering. In these applications, the target function may represent a desired quantity of interest of a parameterized Partial Differential Equation (PDE). Due to the large cost of solving such pr…
▽ More
The problem of approximating smooth, multivariate functions from sample points arises in many applications in scientific computing, e.g., in computational Uncertainty Quantification (UQ) for science and engineering. In these applications, the target function may represent a desired quantity of interest of a parameterized Partial Differential Equation (PDE). Due to the large cost of solving such problems, where each sample is computed by solving a PDE, sample efficiency is a key concerning these applications. Recently, there has been increasing focus on the use of Deep Neural Networks (DNN) and Deep Learning (DL) for learning such functions from data. In this work, we propose an adaptive sampling strategy, CAS4DL (Christoffel Adaptive Sampling for Deep Learning) to increase the sample efficiency of DL for multivariate function approximation. Our novel approach is based on interpreting the second to last layer of a DNN as a dictionary of functions defined by the nodes on that layer. With this viewpoint, we then define an adaptive sampling strategy motivated by adaptive sampling schemes recently proposed for linear approximation schemes, wherein samples are drawn randomly with respect to the Christoffel function of the subspace spanned by this dictionary. We present numerical experiments comparing CAS4DL with standard Monte Carlo (MC) sampling. Our results demonstrate that CAS4DL often yields substantial savings in the number of samples required to achieve a given accuracy, particularly in the case of smooth activation functions, and it shows a better stability in comparison to MC. These results therefore are a promising step towards fully adapting DL towards scientific computing applications.
△ Less
Submitted 25 August, 2022;
originally announced August 2022.
-
Multi-frame Super-resolution from Noisy Data
Authors:
Kireeti Bodduna,
Joachim Weickert,
Marcelo Cárdenas
Abstract:
Obtaining high resolution images from low resolution data with clipped noise is algorithmically challenging due to the ill-posed nature of the problem. So far such problems have hardly been tackled, and the few existing approaches use simplistic regularisers. We show the usefulness of two adaptive regularisers based on anisotropic diffusion ideas: Apart from evaluating the classical edge-enhancing…
▽ More
Obtaining high resolution images from low resolution data with clipped noise is algorithmically challenging due to the ill-posed nature of the problem. So far such problems have hardly been tackled, and the few existing approaches use simplistic regularisers. We show the usefulness of two adaptive regularisers based on anisotropic diffusion ideas: Apart from evaluating the classical edge-enhancing anisotropic diffusion regulariser, we introduce a novel non-local one with one-sided differences and superior performance. It is termed sector diffusion. We combine it with all six variants of the classical super-resolution observational model that arise from permutations of its three operators for war**, blurring, and downsampling. Surprisingly, the evaluation in a practically relevant noisy scenario produces a different ranking than the one in the noise-free setting in our previous work (SSVM 2017).
△ Less
Submitted 25 March, 2021;
originally announced March 2021.
-
Stable Backward Diffusion Models that Minimise Convex Energies
Authors:
Leif Bergerhoff,
Marcelo Cárdenas,
Joachim Weickert,
Martin Welk
Abstract:
The inverse problem of backward diffusion is known to be ill-posed and highly unstable. Backward diffusion processes appear naturally in image enhancement and deblurring applications. It is therefore greatly desirable to establish a backward diffusion model which implements a smart stabilisation approach that can be used in combination with an easy to handle numerical scheme. So far, existing stab…
▽ More
The inverse problem of backward diffusion is known to be ill-posed and highly unstable. Backward diffusion processes appear naturally in image enhancement and deblurring applications. It is therefore greatly desirable to establish a backward diffusion model which implements a smart stabilisation approach that can be used in combination with an easy to handle numerical scheme. So far, existing stabilisation strategies in literature require sophisticated numerics to solve the underlying initial value problem. We derive a class of space-discrete one-dimensional backward diffusion as gradient descent of energies where we gain stability by imposing range constraints. Interestingly, these energies are even convex. Furthermore, we establish a comprehensive theory for the time-continuous evolution and we show that stability carries over to a simple explicit time discretisation of our model. Finally, we confirm the stability and usefulness of our technique in experiments in which we enhance the contrast of digital greyscale and colour images.
△ Less
Submitted 17 June, 2020; v1 submitted 8 March, 2019;
originally announced March 2019.
-
Increasing GP Computing Power via Volunteer Computing
Authors:
Daniel Lombrana Gonzalez,
Francisco Fernandez de Vega,
L. Trujillo,
G. Olague,
F. Chavez de la O,
M. Cardenas,
L. Araujo,
P. Castillo,
K. Sharman
Abstract:
This paper describes how it is possible to increase GP Computing Power via Volunteer Computing (VC) using the BOINC framework. Two experiments using well-known GP tools -Lil-gp & ECJ- are performed in order to demonstrate the benefit of using VC in terms of computing power and speed up. Finally we present an extension of the model where any GP tool or framework can be used inside BOINC regardles…
▽ More
This paper describes how it is possible to increase GP Computing Power via Volunteer Computing (VC) using the BOINC framework. Two experiments using well-known GP tools -Lil-gp & ECJ- are performed in order to demonstrate the benefit of using VC in terms of computing power and speed up. Finally we present an extension of the model where any GP tool or framework can be used inside BOINC regardless of its programming language, complexity or required operating system.
△ Less
Submitted 8 January, 2008;
originally announced January 2008.
