-
On the complexity of nonsmooth automatic differentiation
Authors:
Jérôme Bolte,
Ryan Boustany,
Edouard Pauwels,
Béatrice Pesquet-Popescu
Abstract:
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs. The overhead complexity of the backward mode turns out to be independent of the dimension when using programs with locally Lipschitz semi-algebraic or definable elementary functions. This co…
▽ More
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs. The overhead complexity of the backward mode turns out to be independent of the dimension when using programs with locally Lipschitz semi-algebraic or definable elementary functions. This considerably extends Baur-Strassen's smooth cheap gradient principle. We illustrate our results by establishing fast backpropagation results of conservative gradients through feedforward neural networks with standard activation and loss functions. Nonsmooth backpropagation's cheapness contrasts with concurrent forward approaches, which have, to this day, dimensional-dependent worst-case overhead estimates. We provide further results suggesting the superiority of backward propagation of conservative gradients. Indeed, we relate the complexity of computing a large number of directional derivatives to that of matrix multiplication, and we show that finding two subgradients in the Clarke subdifferential of a function is an NP-hard problem.
△ Less
Submitted 6 February, 2023; v1 submitted 1 June, 2022;
originally announced June 2022.
-
A Proximal Approach for Sparse Multiclass SVM
Authors:
G. Chierchia,
Nelly Pustelnik,
Jean-Christophe Pesquet,
B. Pesquet-Popescu
Abstract:
Sparsity-inducing penalties are useful tools to design multiclass support vector machines (SVMs). In this paper, we propose a convex optimization approach for efficiently and exactly solving the multiclass SVM learning problem involving a sparse regularization and the multiclass hinge loss formulated by Crammer and Singer. We provide two algorithms: the first one dealing with the hinge loss as a p…
▽ More
Sparsity-inducing penalties are useful tools to design multiclass support vector machines (SVMs). In this paper, we propose a convex optimization approach for efficiently and exactly solving the multiclass SVM learning problem involving a sparse regularization and the multiclass hinge loss formulated by Crammer and Singer. We provide two algorithms: the first one dealing with the hinge loss as a penalty term, and the other one addressing the case when the hinge loss is enforced through a constraint. The related convex optimization problems can be efficiently solved thanks to the flexibility offered by recent primal-dual proximal algorithms and epigraphical splitting techniques. Experiments carried out on several datasets demonstrate the interest of considering the exact expression of the hinge loss rather than a smooth approximation. The efficiency of the proposed algorithms w.r.t. several state-of-the-art methods is also assessed through comparisons of execution times.
△ Less
Submitted 14 December, 2015; v1 submitted 15 January, 2015;
originally announced January 2015.
-
A Non-Local Structure Tensor Based Approach for Multicomponent Image Recovery Problems
Authors:
Giovanni Chierchia,
Nelly Pustelnik,
Beatrice Pesquet-Popescu,
Jean-Christophe Pesquet
Abstract:
Non-Local Total Variation (NLTV) has emerged as a useful tool in variational methods for image recovery problems. In this paper, we extend the NLTV-based regularization to multicomponent images by taking advantage of the Structure Tensor (ST) resulting from the gradient of a multicomponent image. The proposed approach allows us to penalize the non-local variations, jointly for the different compon…
▽ More
Non-Local Total Variation (NLTV) has emerged as a useful tool in variational methods for image recovery problems. In this paper, we extend the NLTV-based regularization to multicomponent images by taking advantage of the Structure Tensor (ST) resulting from the gradient of a multicomponent image. The proposed approach allows us to penalize the non-local variations, jointly for the different components, through various $\ell_{1,p}$ matrix norms with $p \ge 1$. To facilitate the choice of the hyper-parameters, we adopt a constrained convex optimization approach in which we minimize the data fidelity term subject to a constraint involving the ST-NLTV regularization. The resulting convex optimization problem is solved with a novel epigraphical projection method. This formulation can be efficiently implemented thanks to the flexibility offered by recent primal-dual proximal algorithms. Experiments are carried out for multispectral and hyperspectral images. The results demonstrate the interest of introducing a non-local structure tensor regularization and show that the proposed approach leads to significant improvements in terms of convergence speed over current state-of-the-art methods.
△ Less
Submitted 14 October, 2014; v1 submitted 21 March, 2014;
originally announced March 2014.
-
Consistent Reconstruction of the Input of an Oversampled Filter Bank From Noisy Subbands
Authors:
Manel Abid,
Michel Kieffer,
Beatrice Pesquet-Popescu
Abstract:
This paper introduces a reconstruction approach for the input signal of an oversampled filter bank (OFB) when the sub-bands generated at its output are quantized and transmitted over a noisy channel. This approach exploits the redundancy introduced by the OFB and the fact that the quantization noise is bounded. A maximum-likelihood estimate of the input signal is evaluated, which only considers th…
▽ More
This paper introduces a reconstruction approach for the input signal of an oversampled filter bank (OFB) when the sub-bands generated at its output are quantized and transmitted over a noisy channel. This approach exploits the redundancy introduced by the OFB and the fact that the quantization noise is bounded. A maximum-likelihood estimate of the input signal is evaluated, which only considers the vectors of quantization indexes corresponding to subband signals that could have been generated by the OFB and that are compliant with the quantization errors. When considering an OFB with an oversampling ratio of 3/2 and a transmission of quantized subbands on an AWGN channel, compared to a classical decoder, the performance gains are up to 9 dB in terms of SNR for the reconstructed signal, and 3 dB in terms of channel SNR.
△ Less
Submitted 29 August, 2011;
originally announced August 2011.
