-
Standalone Neural ODEs with Sensitivity Analysis
Authors:
Rym Jaroudi,
Lukáš Malý,
Gabriel Eilertsen,
B. Tomas Johansson,
Jonas Unger,
George Baravdish
Abstract:
This paper presents the Standalone Neural ODE (sNODE), a continuous-depth neural ODE model capable of describing a full deep neural network. This uses a novel nonlinear conjugate gradient (NCG) descent optimization scheme for training, where the Sobolev gradient can be incorporated to improve smoothness of model weights. We also present a general formulation of the neural sensitivity problem and s…
▽ More
This paper presents the Standalone Neural ODE (sNODE), a continuous-depth neural ODE model capable of describing a full deep neural network. This uses a novel nonlinear conjugate gradient (NCG) descent optimization scheme for training, where the Sobolev gradient can be incorporated to improve smoothness of model weights. We also present a general formulation of the neural sensitivity problem and show how it is used in the NCG training. The sensitivity analysis provides a reliable measure of uncertainty propagation throughout a network, and can be used to study model robustness and to generate adversarial attacks. Our evaluations demonstrate that our novel formulations lead to increased robustness and performance as compared to ResNet models, and that it opens up for new opportunities for designing and develo** machine learning with improved explainability.
△ Less
Submitted 8 June, 2022; v1 submitted 27 May, 2022;
originally announced May 2022.
-
Learning via nonlinear conjugate gradients and depth-varying neural ODEs
Authors:
George Baravdish,
Gabriel Eilertsen,
Rym Jaroudi,
B. Tomas Johansson,
Lukáš Malý,
Jonas Unger
Abstract:
The inverse problem of supervised reconstruction of depth-variable (time-dependent) parameters in a neural ordinary differential equation (NODE) is considered, that means finding the weights of a residual network with time continuous layers. The NODE is treated as an isolated entity describing the full network as opposed to earlier research, which embedded it between pre- and post-appended layers…
▽ More
The inverse problem of supervised reconstruction of depth-variable (time-dependent) parameters in a neural ordinary differential equation (NODE) is considered, that means finding the weights of a residual network with time continuous layers. The NODE is treated as an isolated entity describing the full network as opposed to earlier research, which embedded it between pre- and post-appended layers trained by conventional methods. The proposed parameter reconstruction is done for a general first order differential equation by minimizing a cost functional covering a variety of loss functions and penalty terms. A nonlinear conjugate gradient method (NCG) is derived for the minimization. Mathematical properties are stated for the differential equation and the cost functional. The adjoint problem needed is derived together with a sensitivity problem. The sensitivity problem can estimate changes in the network output under perturbation of the trained parameters. To preserve smoothness during the iterations the Sobolev gradient is calculated and incorporated. As a proof-of-concept, numerical results are included for a NODE and two synthetic datasets, and compared with standard gradient approaches (not based on NODEs). The results show that the proposed method works well for deep learning with infinite numbers of layers, and has built-in stability and smoothness.
△ Less
Submitted 11 February, 2022;
originally announced February 2022.
-
Iterative TV minimization on the graph
Authors:
Japhet Niyobuhungiro,
Eric Setterqvist,
Freddie Åström,
George Baravdish
Abstract:
We define the space of functions of bounded variation ($BV$) on the graph. Using the notion of divergence of flows on graphs, we show that the unit ball of the dual space to $BV$ in the graph setting can be described as the image of the unit ball of the space $\ell^{\infty}$ by the divergence operator. Based on this result, we propose a new iterative algorithm to find the exact minimizer for the t…
▽ More
We define the space of functions of bounded variation ($BV$) on the graph. Using the notion of divergence of flows on graphs, we show that the unit ball of the dual space to $BV$ in the graph setting can be described as the image of the unit ball of the space $\ell^{\infty}$ by the divergence operator. Based on this result, we propose a new iterative algorithm to find the exact minimizer for the total variation (TV) denoising problem on the graph. The proposed algorithm is provable convergent and its performance on image denoising examples is compared with the Split Bregman and Primal-Dual algorithms as benchmarks for iterative methods and with BM3D as a benchmark for other state-of-the-art denoising methods. The experimental results show highly competitive empirical convergence rate and visual quality for the proposed algorithm.
△ Less
Submitted 26 November, 2019;
originally announced November 2019.
-
Damped second order flow applied to image denoising
Authors:
George Baravdish,
Olof Svensson,
Mårten Gulliksson,
Ye Zhang
Abstract:
In this paper, we introduce a new image denoising model: the damped flow (DF), which is a second order nonlinear evolution equation associated with a class of energy functionals of image. The existence, uniqueness and regularization property of DF are proven. For the numerical implementation, based on the Störmer-Verlet method, a discrete damped flow, SV-DDF, is developed. The convergence of SV-DD…
▽ More
In this paper, we introduce a new image denoising model: the damped flow (DF), which is a second order nonlinear evolution equation associated with a class of energy functionals of image. The existence, uniqueness and regularization property of DF are proven. For the numerical implementation, based on the Störmer-Verlet method, a discrete damped flow, SV-DDF, is developed. The convergence of SV-DDF is studied as well. Several numerical experiments, as well as a comparison with other methods, are provided to demonstrate the feasibility and effectiveness of the SV-DDF.
△ Less
Submitted 27 September, 2019; v1 submitted 18 June, 2018;
originally announced June 2018.
