-
Optimal deep learning of holomorphic operators between Banach spaces
Authors:
Ben Adcock,
Nick Dexter,
Sebastian Moraga
Abstract:
Operator learning problems arise in many key areas of scientific computing where Partial Differential Equations (PDEs) are used to model physical systems. In such scenarios, the operators map between Banach or Hilbert spaces. In this work, we tackle the problem of learning operators between Banach spaces, in contrast to the vast majority of past works considering only Hilbert spaces. We focus on l…
▽ More
Operator learning problems arise in many key areas of scientific computing where Partial Differential Equations (PDEs) are used to model physical systems. In such scenarios, the operators map between Banach or Hilbert spaces. In this work, we tackle the problem of learning operators between Banach spaces, in contrast to the vast majority of past works considering only Hilbert spaces. We focus on learning holomorphic operators - an important class of problems with many applications. We combine arbitrary approximate encoders and decoders with standard feedforward Deep Neural Network (DNN) architectures - specifically, those with constant width exceeding the depth - under standard $\ell^2$-loss minimization. We first identify a family of DNNs such that the resulting Deep Learning (DL) procedure achieves optimal generalization bounds for such operators. For standard fully-connected architectures, we then show that there are uncountably many minimizers of the training problem that yield equivalent optimal performance. The DNN architectures we consider are `problem agnostic', with width and depth only depending on the amount of training data $m$ and not on regularity assumptions of the target operator. Next, we show that DL is optimal for this problem: no recovery procedure can surpass these generalization bounds up to log terms. Finally, we present numerical results demonstrating the practical performance on challenging problems including the parametric diffusion, Navier-Stokes-Brinkman and Boussinesq PDEs.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks
Authors:
Ben Adcock,
Simone Brugiapaglia,
Nick Dexter,
Sebastian Moraga
Abstract:
Learning approximations to smooth target functions of many variables from finite sets of pointwise samples is an important task in scientific computing and its many applications in computational science and engineering. Despite well over half a century of research on high-dimensional approximation, this remains a challenging problem. Yet, significant advances have been made in the last decade towa…
▽ More
Learning approximations to smooth target functions of many variables from finite sets of pointwise samples is an important task in scientific computing and its many applications in computational science and engineering. Despite well over half a century of research on high-dimensional approximation, this remains a challenging problem. Yet, significant advances have been made in the last decade towards efficient methods for doing this, commencing with so-called sparse polynomial approximation methods and continuing most recently with methods based on Deep Neural Networks (DNNs). In tandem, there have been substantial advances in the relevant approximation theory and analysis of these techniques. In this work, we survey this recent progress. We describe the contemporary motivations for this problem, which stem from parametric models and computational uncertainty quantification; the relevant function classes, namely, classes of infinite-dimensional, Banach-valued, holomorphic functions; fundamental limits of learnability from finite data for these classes; and finally, sparse polynomial and DNN methods for efficiently learning such functions from finite data. For the latter, there is currently a significant gap between the approximation theory of DNNs and the practical performance of deep learning. Aiming to narrow this gap, we develop the topic of practical existence theory, which asserts the existence of dimension-independent DNN architectures and training strategies that achieve provably near-optimal generalization errors in terms of the amount of training data.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
On efficient algorithms for computing near-best polynomial approximations to high-dimensional, Hilbert-valued functions from limited samples
Authors:
Ben Adcock,
Simone Brugiapaglia,
Nick Dexter,
Sebastian Moraga
Abstract:
Sparse polynomial approximation has become indispensable for approximating smooth, high- or infinite-dimensional functions from limited samples. This is a key task in computational science and engineering, e.g., surrogate modelling in uncertainty quantification where the function is the solution map of a parametric or stochastic differential equation (DE). Yet, sparse polynomial approximation lack…
▽ More
Sparse polynomial approximation has become indispensable for approximating smooth, high- or infinite-dimensional functions from limited samples. This is a key task in computational science and engineering, e.g., surrogate modelling in uncertainty quantification where the function is the solution map of a parametric or stochastic differential equation (DE). Yet, sparse polynomial approximation lacks a complete theory. On the one hand, there is a well-developed theory of best $s$-term polynomial approximation, which asserts exponential or algebraic rates of convergence for holomorphic functions. On the other, there are increasingly mature methods such as (weighted) $\ell^1$-minimization for computing such approximations. While the sample complexity of these methods has been analyzed with compressed sensing, whether they achieve best $s$-term approximation rates is not fully understood. Furthermore, these methods are not algorithms per se, as they involve exact minimizers of nonlinear optimization problems.
This paper closes these gaps. Specifically, we consider the following question: are there robust, efficient algorithms for computing approximations to finite- or infinite-dimensional, holomorphic and Hilbert-valued functions from limited samples that achieve best $s$-term rates? We answer this affirmatively by introducing algorithms and theoretical guarantees that assert exponential or algebraic rates of convergence, along with robustness to sampling, algorithmic, and physical discretization errors. We tackle both scalar- and Hilbert-valued functions, this being key to parametric or stochastic DEs. Our results involve significant developments of existing techniques, including a novel restarted primal-dual iteration for solving weighted $\ell^1$-minimization problems in Hilbert spaces. Our theory is supplemented by numerical experiments demonstrating the efficacy of these algorithms.
△ Less
Submitted 6 November, 2023; v1 submitted 25 March, 2022;
originally announced March 2022.
-
Canvas Adoption Assessment and Acceptance of the Learning Management System on a Web-Based Platform
Authors:
Julius G. Garcia,
Mark Gil T. Gangan,
Marita N. Tolentino,
Marc Ligas,
Shirley D. Moraga,
Amelia A. Pasilan
Abstract:
The acquisition of non-proprietary and proprietary learning management system has provided a richer learning experience to users and raised interest among education providers. This study aims to assess student adoption of Canvas as a new learning management system and its potential as a web-based platform in the e-learning programme of the University of the East. This study also assessed student r…
▽ More
The acquisition of non-proprietary and proprietary learning management system has provided a richer learning experience to users and raised interest among education providers. This study aims to assess student adoption of Canvas as a new learning management system and its potential as a web-based platform in the e-learning programme of the University of the East. This study also assessed student readiness in using Canvas. A survey was administered to 214 students of the University of the East through snowball sampling. An Exploratory Factor Analysis was conducted to examine the validity of the model. A Confirmatory Factory Analysis was used to validate the Exploratory Factor Analysis results and analyse the correlation of the constructs. A Structural Equation Modelling was conducted to analyse the relationships between the constructs, which were evaluated using fit indices. Adopted from the Technology Acceptance Model, the constructs perceived ease of use, perceived usefulness, and attitude were studied. The study reveals that students perceived usefulness and attitude towards using Canvas in a web-based platform have direct and significant effects on their intention to use Canvas. The students perceived ease of use has a significant effect on their perceived usefulness but has no significant effects on their attitude towards the use of Canvas. The students technological maturity and prior experience in using a learning management system influenced their beliefs on the adaptation of similar technology. Exploring the potential benefits of Canvas and factors affecting the students adoption amplifies access to quality education to fulfil educational directives. Furthermore, educational institutions should explore technological migration related to teaching and learning processes.
△ Less
Submitted 26 May, 2021; v1 submitted 28 January, 2021;
originally announced January 2021.
-
Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data
Authors:
Ben Adcock,
Simone Brugiapaglia,
Nick Dexter,
Sebastian Moraga
Abstract:
Accurate approximation of scalar-valued functions from sample points is a key task in computational science. Recently, machine learning with Deep Neural Networks (DNNs) has emerged as a promising tool for scientific computing, with impressive results achieved on problems where the dimension of the data or problem domain is large. This work broadens this perspective, focusing on approximating funct…
▽ More
Accurate approximation of scalar-valued functions from sample points is a key task in computational science. Recently, machine learning with Deep Neural Networks (DNNs) has emerged as a promising tool for scientific computing, with impressive results achieved on problems where the dimension of the data or problem domain is large. This work broadens this perspective, focusing on approximating functions that are Hilbert-valued, i.e. take values in a separable, but typically infinite-dimensional, Hilbert space. This arises in science and engineering problems, in particular those involving solution of parametric Partial Differential Equations (PDEs). Such problems are challenging: 1) pointwise samples are expensive to acquire, 2) the function domain is high dimensional, and 3) the range lies in a Hilbert space. Our contributions are twofold. First, we present a novel result on DNN training for holomorphic functions with so-called hidden anisotropy. This result introduces a DNN training procedure and full theoretical analysis with explicit guarantees on error and sample complexity. The error bound is explicit in three key errors occurring in the approximation procedure: the best approximation, measurement, and physical discretization errors. Our result shows that there exists a procedure (albeit non-standard) for learning Hilbert-valued functions via DNNs that performs as well as, but no better than current best-in-class schemes. It gives a benchmark lower bound for how well DNNs can perform on such problems. Second, we examine whether better performance can be achieved in practice through different types of architectures and training. We provide preliminary numerical results illustrating practical performance of DNNs on parametric PDEs. We consider different parameters, modifying the DNN architecture to achieve better and competitive results, comparing these to current best-in-class schemes.
△ Less
Submitted 4 March, 2021; v1 submitted 10 December, 2020;
originally announced December 2020.
