Showing 1–29 of 29 results for author: Chouzenoux, E

A divergence-based condition to ensure quantile improvement in black-box global optimization

Authors: Thomas Guilmeau, Emilie Chouzenoux, Víctor Elvira

Abstract: Black-box global optimization aims at minimizing an objective function whose analytical form is not known. To do so, many state-of-the-art methods rely on sampling-based strategies, where sampling distributions are built in an iterative fashion, so that their mass concentrate where the objective function is low. Despite empirical success, the theoretical study of these methods remains difficult. I… ▽ More Black-box global optimization aims at minimizing an objective function whose analytical form is not known. To do so, many state-of-the-art methods rely on sampling-based strategies, where sampling distributions are built in an iterative fashion, so that their mass concentrate where the objective function is low. Despite empirical success, the theoretical study of these methods remains difficult. In this work, we introduce a new framework, based on divergence-decrease conditions, to study and design black-box global optimization algorithms. Our approach allows to establish and quantify the improvement of proposals at each iteration, in terms of expected value or quantile of the objective. We show that the information-geometric optimization approach fits within our framework, yielding a new approach for its analysis. We also establish proposal improvement results for two novel algorithms, one related with the cross-entropy approach with mixture models, and another one using heavy-tailed sampling proposal distributions. △ Less

Submitted 19 June, 2024; v1 submitted 2 February, 2024; originally announced February 2024.

Comments: 22 pages, 1 figure

MSC Class: 90C26 (Primary) 90C59; 65K05 (Secondary)

A Novel Variational Approach for Multiphoton Microscopy Image Restoration: from PSF Estimation to 3D Deconvolution

Authors: Julien Ajdenbaum, Emilie Chouzenoux, Claire Lefort, Ségolène Martin, Jean-Christophe Pesquet

Abstract: In multi-photon microscopy (MPM), a recent in-vivo fluorescence microscopy system, the task of image restoration can be decomposed into two interlinked inverse problems: firstly, the characterization of the Point Spread Function (PSF) and subsequently, the deconvolution (i.e., deblurring) to remove the PSF effect, and reduce noise. The acquired MPM image quality is critically affected by PSF blurr… ▽ More In multi-photon microscopy (MPM), a recent in-vivo fluorescence microscopy system, the task of image restoration can be decomposed into two interlinked inverse problems: firstly, the characterization of the Point Spread Function (PSF) and subsequently, the deconvolution (i.e., deblurring) to remove the PSF effect, and reduce noise. The acquired MPM image quality is critically affected by PSF blurring and intense noise. The PSF in MPM is highly spread in 3D and is not well characterized, presenting high variability with respect to the observed objects. This makes the restoration of MPM images challenging. Common PSF estimation methods in fluorescence microscopy, including MPM, involve capturing images of sub-resolution beads, followed by quantifying the resulting ellipsoidal 3D spot. In this work, we revisit this approach, co** with its inherent limitations in terms of accuracy and practicality. We estimate the PSF from the observation of relatively large beads (approximately 1$μ$m in diameter). This goes through the formulation and resolution of an original non-convex minimization problem, for which we propose a proximal alternating method along with convergence guarantees. Following the PSF estimation step, we then introduce an innovative strategy to deal with the high level multiplicative noise degrading the acquisitions. We rely on a heteroscedastic noise model for which we estimate the parameters. We then solve a constrained optimization problem to restore the image, accounting for the estimated PSF and noise, while allowing a minimal hyper-parameter tuning. Theoretical guarantees are given for the restoration algorithm. These algorithmic contributions lead to an end-to-end pipeline for 3D image restoration in MPM, that we share as a publicly available Python software. We demonstrate its effectiveness through several experiments on both simulated and real data. △ Less

Submitted 30 November, 2023; originally announced November 2023.

Solution of Mismatched Monotone+Lipschitz Inclusion Problems

Authors: Emilie Chouzenoux, Jean-Christophe Pesquet, Fernando Roldán

Abstract: In this article, we study the convergence of algorithms for solving monotone inclusions in the presence of adjoint mismatch. The adjoint mismatch arises when the adjoint of a linear operator is replaced by an approximation, due to computational or physical issues. This occurs in inverse problems, particularly in computed tomography. In real Hilbert spaces, monotone inclusion problems involving a m… ▽ More In this article, we study the convergence of algorithms for solving monotone inclusions in the presence of adjoint mismatch. The adjoint mismatch arises when the adjoint of a linear operator is replaced by an approximation, due to computational or physical issues. This occurs in inverse problems, particularly in computed tomography. In real Hilbert spaces, monotone inclusion problems involving a maximally $ρ$-monotone operator, a cocoercive operator, and a Lipschitzian operator can be solved by the Forward-Backward-Half-Forward and the Forward-Douglas-Rachford-Forward methods. We investigate the case of a mismatched Lipschitzian operator. We propose variants of the two aforementioned methods to cope with the mismatch, and establish conditions under which the weak convergence to a solution is guaranteed for these variants. The proposed algorithms hence enable each iteration to be implemented with a possibly iteration-dependent approximation to the mismatch operator, thus allowing this operator to be modified at each iteration. Finally, we present numerical experiments on a computed tomography example in material science, showing the applicability of our theoretical findings. △ Less

Submitted 9 November, 2023; v1 submitted 10 October, 2023; originally announced October 2023.

MSC Class: 47H05; 47H10; 65K05; 90C25

On variational inference and maximum likelihood estimation with the λ-exponential family

Authors: Thomas Guilmeau, Emilie Chouzenoux, Víctor Elvira

Abstract: The λ-exponential family has recently been proposed to generalize the exponential family. While the exponential family is well-understood and widely used, this it not the case of the λ-exponential family. However, many applications require models that are more general than the exponential family. In this work, we propose a theoretical and algorithmic framework to solve variational inference and ma… ▽ More The λ-exponential family has recently been proposed to generalize the exponential family. While the exponential family is well-understood and widely used, this it not the case of the λ-exponential family. However, many applications require models that are more general than the exponential family. In this work, we propose a theoretical and algorithmic framework to solve variational inference and maximum likelihood estimation problems over the λ-exponential family. We give new sufficient optimality conditions for variational inference problems. Our conditions take the form of generalized moment-matching conditions and generalize existing similar results for the exponential family. We exhibit novel characterizations of the solutions of maximum likelihood estimation problems, that recover optimality conditions in the case of the exponential family. For the resolution of both problems, we propose novel proximal-like algorithms that exploit the geometry underlying the λ-exponential family. These new theoretical and methodological insights are tested on numerical examples, showcasing their usefulness and interest, especially on heavy-tailed target distributions. △ Less

Submitted 19 June, 2024; v1 submitted 6 October, 2023; originally announced October 2023.

MSC Class: 62F99; 62B11; 49K10; 90C26

Majorization-Minimization for sparse SVMs

Authors: Alessandro Benfenati, Emilie Chouzenoux, Giorgia Franchini, Salla Latva-Aijo, Dominik Narnhofer, Jean-Christophe Pesquet, Sebastian J. Scott, Mahsa Yousefi

Abstract: Several decades ago, Support Vector Machines (SVMs) were introduced for performing binary classification tasks, under a supervised framework. Nowadays, they often outperform other supervised methods and remain one of the most popular approaches in the machine learning arena. In this work, we investigate the training of SVMs through a smooth sparse-promoting-regularized squared hinge loss minimizat… ▽ More Several decades ago, Support Vector Machines (SVMs) were introduced for performing binary classification tasks, under a supervised framework. Nowadays, they often outperform other supervised methods and remain one of the most popular approaches in the machine learning arena. In this work, we investigate the training of SVMs through a smooth sparse-promoting-regularized squared hinge loss minimization. This choice paves the way to the application of quick training methods built on majorization-minimization approaches, benefiting from the Lipschitz differentiabililty of the loss function. Moreover, the proposed approach allows us to handle sparsity-preserving regularizers promoting the selection of the most significant features, so enhancing the performance. Numerical tests and comparisons conducted on three different datasets demonstrate the good performance of the proposed methodology in terms of qualitative metrics (accuracy, precision, recall, and F 1 score) as well as computational cost. △ Less

Submitted 31 August, 2023; originally announced August 2023.

A new non-convex framework to improve asymptotical knowledge on generic stochastic gradient descent

Authors: Jean-Baptiste Fest, Audrey Repetti, Emilie Chouzenoux

Abstract: Stochastic gradient optimization methods are broadly used to minimize non-convex smooth objective functions, for instance when training deep neural networks. However, theoretical guarantees on the asymptotic behaviour of these methods remain scarce. Especially, ensuring almost-sure convergence of the iterates to a stationary point is quite challenging. In this work, we introduce a new Kurdyka-Loja… ▽ More Stochastic gradient optimization methods are broadly used to minimize non-convex smooth objective functions, for instance when training deep neural networks. However, theoretical guarantees on the asymptotic behaviour of these methods remain scarce. Especially, ensuring almost-sure convergence of the iterates to a stationary point is quite challenging. In this work, we introduce a new Kurdyka-Lojasiewicz theoretical framework to analyze asymptotic behavior of stochastic gradient descent (SGD) schemes when minimizing non-convex smooth objectives. In particular, our framework provides new almost-sure convergence results, on iterates generated by any SGD method satisfying mild conditional descent conditions. We illustrate the proposed framework by means of several toy simulation examples. We illustrate the role of the considered theoretical assumptions, and investigate how SGD iterates are impacted whether these assumptions are either fully or partially satisfied. △ Less

Submitted 13 July, 2023; originally announced July 2023.

Sparse Graphical Linear Dynamical Systems

Authors: Emilie Chouzenoux, Victor Elvira

Abstract: Time-series datasets are central in machine learning with applications in numerous fields of science and engineering, such as biomedicine, Earth observation, and network analysis. Extensive research exists on state-space models (SSMs), which are powerful mathematical tools that allow for probabilistic and interpretable learning on time series. Learning the model parameters in SSMs is arguably one… ▽ More Time-series datasets are central in machine learning with applications in numerous fields of science and engineering, such as biomedicine, Earth observation, and network analysis. Extensive research exists on state-space models (SSMs), which are powerful mathematical tools that allow for probabilistic and interpretable learning on time series. Learning the model parameters in SSMs is arguably one of the most complicated tasks, and the inclusion of prior knowledge is known to both ease the interpretation but also to complicate the inferential tasks. Very recent works have attempted to incorporate a graphical perspective on some of those model parameters, but they present notable limitations that this work addresses. More generally, existing graphical modeling tools are designed to incorporate either static information, focusing on statistical dependencies among independent random variables (e.g., graphical Lasso approach), or dynamic information, emphasizing causal relationships among time series samples (e.g., graphical Granger approaches). However, there are no joint approaches combining static and dynamic graphical modeling within the context of SSMs. This work proposes a novel approach to fill this gap by introducing a joint graphical modeling framework that bridges the graphical Lasso model and a causal-based graphical approach for the linear-Gaussian SSM. We present DGLASSO (Dynamic Graphical Lasso), a new inference method within this framework that implements an efficient block alternating majorization-minimization algorithm. The algorithm's convergence is established by departing from modern tools from nonlinear analysis. Experimental validation on various synthetic data showcases the effectiveness of the proposed model and inference algorithm. △ Less

Submitted 14 June, 2024; v1 submitted 6 July, 2023; originally announced July 2023.

A Kurdyka-Lojasiewicz property for stochastic optimization algorithms in a non-convex setting

Authors: Emilie Chouzenoux, Jean-Baptiste Fest, Audrey Repetti

Abstract: Stochastic differentiable approximation schemes are widely used for solving high dimensional problems. Most of existing methods satisfy some desirable properties, including conditional descent inequalities, and almost sure (a.s.) convergence guarantees on the objective function, or on the involved gradient. However, for non-convex objective functions, a.s. convergence of the iterates, i.e., the st… ▽ More Stochastic differentiable approximation schemes are widely used for solving high dimensional problems. Most of existing methods satisfy some desirable properties, including conditional descent inequalities, and almost sure (a.s.) convergence guarantees on the objective function, or on the involved gradient. However, for non-convex objective functions, a.s. convergence of the iterates, i.e., the stochastic process, to a critical point is usually not guaranteed, and remains an important challenge. In this article, we develop a framework to bridge the gap between descent-type inequalities and a.s. convergence of the associated stochastic process. Leveraging a novel Kurdyka-Lojasiewicz property, we show convergence guarantees of stochastic processes under mild assumptions on the objective function. We also provide examples of stochastic algorithms benefiting from the proposed framework and derive a.s. convergence guarantees on the iterates. △ Less

Submitted 23 March, 2023; v1 submitted 13 February, 2023; originally announced February 2023.

MSC Class: 65K05; 90C26; 90C15; 90C53

PENDANTSS: PEnalized Norm-ratios Disentangling Additive Noise, Trend and Sparse Spikes

Authors: Paul Zheng, Emilie Chouzenoux, Laurent Duval

Abstract: Denoising, detrending, deconvolution: usual restoration tasks, traditionally decoupled. Coupled formulations entail complex ill-posed inverse problems. We propose PENDANTSS for joint trend removal and blind deconvolution of sparse peak-like signals. It blends a parsimonious prior with the hypothesis that smooth trend and noise can somewhat be separated by low-pass filtering. We combine the general… ▽ More Denoising, detrending, deconvolution: usual restoration tasks, traditionally decoupled. Coupled formulations entail complex ill-posed inverse problems. We propose PENDANTSS for joint trend removal and blind deconvolution of sparse peak-like signals. It blends a parsimonious prior with the hypothesis that smooth trend and noise can somewhat be separated by low-pass filtering. We combine the generalized quasi-norm ratio SOOT/SPOQ sparse penalties $\ell_p/\ell_q$ with the BEADS ternary assisted source separation algorithm. This results in a both convergent and efficient tool, with a novel Trust-Region block alternating variable metric forward-backward approach. It outperforms comparable methods, when applied to typically peaked analytical chemistry signals. Reproducible code is provided. △ Less

Submitted 16 February, 2023; v1 submitted 4 January, 2023; originally announced January 2023.

Regularized Rényi divergence minimization through Bregman proximal gradient algorithms

Authors: Thomas Guilmeau, Emilie Chouzenoux, Víctor Elvira

Abstract: We study the variational inference problem of minimizing a regularized Rényi divergence over an exponential family, and propose a relaxed moment-matching algorithm, which includes a proximal-like step. Using the information-geometric link between Bregman divergences and the Kullback-Leibler divergence, this algorithm is shown to be equivalent to a Bregman proximal gradient algorithm. This novel pe… ▽ More We study the variational inference problem of minimizing a regularized Rényi divergence over an exponential family, and propose a relaxed moment-matching algorithm, which includes a proximal-like step. Using the information-geometric link between Bregman divergences and the Kullback-Leibler divergence, this algorithm is shown to be equivalent to a Bregman proximal gradient algorithm. This novel perspective allows us to exploit the geometry of our approximate model while using stochastic black-box updates. We use this point of view to prove strong convergence guarantees including monotonic decrease of the objective, convergence to a stationary point or to the minimizer, and geometric convergence rates. These new theoretical insights lead to a versatile, robust, and competitive method, as illustrated by numerical experiments. △ Less

Submitted 19 June, 2024; v1 submitted 9 November, 2022; originally announced November 2022.

MSC Class: 62F15; 62F30; 62B11; 90C26; 90C30

Towards Practical Few-Shot Query Sets: Transductive Minimum Description Length Inference

Authors: Ségolène Martin, Malik Boudiaf, Emilie Chouzenoux, Jean-Christophe Pesquet, Ismail Ben Ayed

Abstract: Standard few-shot benchmarks are often built upon simplifying assumptions on the query sets, which may not always hold in practice. In particular, for each task at testing time, the classes effectively present in the unlabeled query set are known a priori, and correspond exactly to the set of classes represented in the labeled support set. We relax these assumptions and extend current benchmarks,… ▽ More Standard few-shot benchmarks are often built upon simplifying assumptions on the query sets, which may not always hold in practice. In particular, for each task at testing time, the classes effectively present in the unlabeled query set are known a priori, and correspond exactly to the set of classes represented in the labeled support set. We relax these assumptions and extend current benchmarks, so that the query-set classes of a given task are unknown, but just belong to a much larger set of possible classes. Our setting could be viewed as an instance of the challenging yet practical problem of extremely imbalanced K-way classification, K being much larger than the values typically used in standard benchmarks, and with potentially irrelevant supervision from the support set. Expectedly, our setting incurs drops in the performances of state-of-the-art methods. Motivated by these observations, we introduce a PrimAl Dual Minimum Description LEngth (PADDLE) formulation, which balances data-fitting accuracy and model complexity for a given few-shot task, under supervision constraints from the support set. Our constrained MDL-like objective promotes competition among a large set of possible classes, preserving only effective classes that befit better the data of a few-shot task. It is hyperparameter free, and could be applied on top of any base-class training. Furthermore, we derive a fast block coordinate descent algorithm for optimizing our objective, with convergence guarantee, and a linear computational complexity at each iteration. Comprehensive experiments over the standard few-shot datasets and the more realistic and challenging i-Nat dataset show highly competitive performances of our method, more so when the numbers of possible classes in the tasks increase. Our code is publicly available at https://github.com/SegoleneMartin/PADDLE. △ Less

Submitted 26 October, 2022; originally announced October 2022.

Gradient-based Adaptive Importance Samplers

Authors: Víctor Elvira, Emilie Chouzenoux, Ömer Deniz Akyildiz, Luca Martino

Abstract: Importance sampling (IS) is a powerful Monte Carlo methodology for the approximation of intractable integrals, very often involving a target probability density function. The performance of IS heavily depends on the appropriate selection of the proposal distributions where the samples are simulated from. In this paper, we propose an adaptive importance sampler, called GRAMIS, that iteratively impr… ▽ More Importance sampling (IS) is a powerful Monte Carlo methodology for the approximation of intractable integrals, very often involving a target probability density function. The performance of IS heavily depends on the appropriate selection of the proposal distributions where the samples are simulated from. In this paper, we propose an adaptive importance sampler, called GRAMIS, that iteratively improves the set of proposals. The algorithm exploits geometric information of the target to adapt the location and scale parameters of those proposals. Moreover, in order to allow for a cooperative adaptation, a repulsion term is introduced that favors a coordinated exploration of the state space. This translates into a more diverse exploration and a better approximation of the target via the mixture of proposals. Moreover, we provide a theoretical justification of the repulsion term. We show the good performance of GRAMIS in two problems where the target has a challenging shape and cannot be easily approximated by a standard uni-modal proposal. △ Less

Submitted 21 June, 2023; v1 submitted 19 October, 2022; originally announced October 2022.

Deep Unfolding of the DBFB Algorithm with Application to ROI CT Imaging with Limited Angular Density

Authors: Marion Savanier, Emilie Chouzenoux, Jean-Christophe Pesquet, Cyril Riddell

Abstract: This paper presents a new method for reconstructing regions of interest (ROI) from a limited number of computed tomography (CT) measurements. Classical model-based iterative reconstruction methods lead to images with predictable features. Still, they often suffer from tedious parameterization and slow convergence. On the contrary, deep learning methods are fast, and they can reach high reconstruct… ▽ More This paper presents a new method for reconstructing regions of interest (ROI) from a limited number of computed tomography (CT) measurements. Classical model-based iterative reconstruction methods lead to images with predictable features. Still, they often suffer from tedious parameterization and slow convergence. On the contrary, deep learning methods are fast, and they can reach high reconstruction quality by leveraging information from large datasets, but they lack interpretability. At the crossroads of both methods, deep unfolding networks have been recently proposed. Their design includes the physics of the imaging system and the steps of an iterative optimization algorithm. Motivated by the success of these networks for various applications, we introduce an unfolding neural network called U-RDBFB designed for ROI CT reconstruction from limited data. Few-view truncated data are effectively handled thanks to a robust non-convex data fidelity term combined with a sparsity-inducing regularization function. We unfold the Dual Block coordinate Forward-Backward (DBFB) algorithm, embedded in an iterative reweighted scheme, allowing the learning of key parameters in a supervised manner. Our experiments show an improvement over several state-of-the-art methods, including a model-based iterative scheme, a multi-scale deep learning architecture, and other deep unfolding methods. △ Less

Submitted 17 May, 2023; v1 submitted 27 September, 2022; originally announced September 2022.

A CNC approach for Directional Total Variation

Authors: Gabriele Scrivanti, Emilie Chouzenoux, Jean-Christophe Pesquet

Abstract: The core of many approaches for the resolution of variational inverse problems arising in signal and image processing consists of promoting the sought solution to have a sparse representation in a well-suited space. A crucial task in this context is the choice of a good sparsity prior that can ensure a good trade-off between the quality of the solution and the resulting computational cost. The rec… ▽ More The core of many approaches for the resolution of variational inverse problems arising in signal and image processing consists of promoting the sought solution to have a sparse representation in a well-suited space. A crucial task in this context is the choice of a good sparsity prior that can ensure a good trade-off between the quality of the solution and the resulting computational cost. The recently introduced Convex-Non-Convex (CNC) strategy appears as a great compromise, as it combines the high qualitative performance of non-convex sparsity-promoting functions with the convenience of dealing with convex optimization problems. This work proposes a new variational formulation to implement CNC approach in the context of image denoising. By suitably exploiting duality properties, our formulation allows to encompass sophisticated directional total variation (DTV) priors. We additionally propose an efficient optimisation strategy for the resulting convex minimisation problem. We illustrate on numerical examples the good performance of the resulting CNC-DTV method, when compared to the standard convex total variation denoiser. △ Less

Submitted 3 September, 2022; originally announced September 2022.

Comments: Accepted for EUSIPCO 2022 - 30th European Signal Processing Conference, Aug 2022, Belgrade, Serbia

A Variational Approach for Joint Image Recovery and Feature Extraction Based on Spatially-Varying Generalised Gaussian Models

Authors: Emilie Chouzenoux, Marie-Caroline Corbineau, Jean-Christophe Pesquet, Gabriele Scrivanti

Abstract: The joint problem of reconstruction / feature extraction is a challenging task in image processing. It consists in performing, in a joint manner, the restoration of an image and the extraction of its features. In this work, we firstly propose a novel nonsmooth and non-convex variational formulation of the problem. For this purpose, we introduce a versatile generalised Gaussian prior whose paramete… ▽ More The joint problem of reconstruction / feature extraction is a challenging task in image processing. It consists in performing, in a joint manner, the restoration of an image and the extraction of its features. In this work, we firstly propose a novel nonsmooth and non-convex variational formulation of the problem. For this purpose, we introduce a versatile generalised Gaussian prior whose parameters, including its exponent, are space-variant. Secondly, we design an alternating proximal-based optimisation algorithm that efficiently exploits the structure of the proposed non-convex objective function. We also analyse the convergence of this algorithm. As shown in numerical experiments conducted on joint deblurring/segmentation tasks, the proposed method provides high-quality results. △ Less

Submitted 5 March, 2024; v1 submitted 3 September, 2022; originally announced September 2022.

Unrolled Variational Bayesian Algorithm for Image Blind Deconvolution

Authors: Yunshi Huang, Emilie Chouzenoux, Jean-Christophe Pesquet

Abstract: In this paper, we introduce a variational Bayesian algorithm (VBA) for image blind deconvolution. Our generic framework incorporates smoothness priors on the unknown blur/image and possible affine constraints (e.g., sum to one) on the blur kernel. One of our main contributions is the integration of VBA within a neural network paradigm, following an unrolling methodology. The proposed architecture… ▽ More In this paper, we introduce a variational Bayesian algorithm (VBA) for image blind deconvolution. Our generic framework incorporates smoothness priors on the unknown blur/image and possible affine constraints (e.g., sum to one) on the blur kernel. One of our main contributions is the integration of VBA within a neural network paradigm, following an unrolling methodology. The proposed architecture is trained in a supervised fashion, which allows us to optimally set two key hyperparameters of the VBA model and lead to further improvements in terms of resulting visual quality. Various experiments involving grayscale/color images and diverse kernel shapes, are performed. The numerical examples illustrate the high performance of our approach when compared to state-of-the-art techniques based on optimization, Bayesian estimation, or deep learning. △ Less

Submitted 14 October, 2021; originally announced October 2021.

Comments: 13 pages

Inversion of Integral Models: a Neural Network Approach

Authors: Emilie Chouzenoux, Cecile Della Valle, Jean-Christophe Pesquet

Abstract: We introduce a neural network architecture to solve inverse problems linked to a one-dimensional integral operator. This architecture is built by unfolding a forward-backward algorithm derived from the minimization of an objective function which consists of the sum of a data-fidelity function and a Tikhonov-type regularization function. The robustness of this inversion method with respect to a per… ▽ More We introduce a neural network architecture to solve inverse problems linked to a one-dimensional integral operator. This architecture is built by unfolding a forward-backward algorithm derived from the minimization of an objective function which consists of the sum of a data-fidelity function and a Tikhonov-type regularization function. The robustness of this inversion method with respect to a perturbation of the input is theoretically analyzed. Ensuring robustness is consistent with inverse problem theory since it guarantees both the continuity of the inversion method and its insensitivity to small noise. The latter is a critical property as deep neural networks have been shown to be vulnerable to adversarial perturbations. One of the main novelties of our work is to show that the proposed network is also robust to perturbations of its bias. In our architecture, the bias accounts for the observed data in the inverse problem. We apply our method to the inversion of Abel integral operators, which define a fractional integration involved in wide range of physical processes. The neural network is numerically implemented and tested to illustrate the efficiency of the method. Lipschitz constants after training are computed to measure the robustness of the neural networks. △ Less

Submitted 31 May, 2021; originally announced May 2021.

Comments: 35 pages, 10 figures

Block Distributed Majorize-Minimize Memory Gradient Algorithm and its application to 3D image restoration

Authors: Mathieu Chalvidal, Emilie Chouzenoux

Abstract: Modern 3D image recovery problems require powerful optimization frameworks to handle high dimensionality while providing reliable numerical solutions in a reasonable time. In this perspective, asynchronous parallel optimization algorithms have received an increasing attention by overcoming memory limitation issues and communication bottlenecks. In this work, we propose a block distributed Majorize… ▽ More Modern 3D image recovery problems require powerful optimization frameworks to handle high dimensionality while providing reliable numerical solutions in a reasonable time. In this perspective, asynchronous parallel optimization algorithms have received an increasing attention by overcoming memory limitation issues and communication bottlenecks. In this work, we propose a block distributed Majorize-Minorize Memory Gradient (BD3MG) optimization algorithm for solving large scale non-convex differentiable optimization problems. Assuming a distributed memory environment, the algorithm casts the efficient 3MG scheme into smaller dimension subproblems where blocks of variables are addressed in an asynchronous manner. Convergence of the sequence built by the proposed BD3MG method is established under mild assumptions. Application to the restoration of 3D images degraded by a depth-variant blur shows that our method yields significant computational time reduction compared to several synchronous and asynchronous competitors, while exhibiting great scalability potential. △ Less

Submitted 25 June, 2020; v1 submitted 6 February, 2020; originally announced February 2020.

Comments: Accepted at ICIP 2020

SPOQ $\ell_p$-Over-$\ell_q$ Regularization for Sparse Signal Recovery applied to Mass Spectrometry

Authors: Afef Cherni, Emilie Chouzenoux, Laurent Duval, Jean-Christophe Pesquet

Abstract: Underdetermined or ill-posed inverse problems require additional information for \ldd{d} sound solutions with tractable optimization algorithms. Sparsity yields consequent heuristics to that matter, with numerous applications in signal restoration, image recovery, or machine learning. Since the $\ell_0$ count measure is barely tractable, many statistical or learning approaches have invested in com… ▽ More Underdetermined or ill-posed inverse problems require additional information for \ldd{d} sound solutions with tractable optimization algorithms. Sparsity yields consequent heuristics to that matter, with numerous applications in signal restoration, image recovery, or machine learning. Since the $\ell_0$ count measure is barely tractable, many statistical or learning approaches have invested in computable proxies, such as the $\ell_1$ norm. However, the latter does not exhibit the desirable property of scale invariance for sparse data. Extending the SOOT Euclidean/Taxicab $\ell_1$-over-$\ell_2$ norm-ratio initially introduced for blind deconvolution, we propose SPOQ, a family of smoothed (approximately) scale-invariant penalty functions. It consists of a Lipschitz-differentiable surrogate for $\ell_p$-over-$\ell_q$ quasi-norm/norm ratios with $p\in\,]0,2[$ and $q\ge 2$. This surrogate is embedded into a novel majorize-minimize trust-region approach, generalizing the variable metric forward-backward algorithm. For naturally sparse mass-spectrometry signals, we show that SPOQ significantly outperforms $\ell_0$, $\ell_1$, Cauchy, Welsch, SCAD and Celo penalties on several performance measures. Guidelines on SPOQ hyperparameters tuning are also provided, suggesting simple data-driven choices. △ Less

Submitted 22 September, 2020; v1 submitted 23 January, 2020; originally announced January 2020.

Journal ref: IEEE Transactions on Signal Processing, 2020, Volume 68, pages 6070--6084

GraphEM: EM algorithm for blind Kalman filtering under graphical sparsity constraints

Authors: Émilie Chouzenoux, Víctor Elvira

Abstract: Modeling and inference with multivariate sequences is central in a number of signal processing applications such as acoustics, social network analysis, biomedical, and finance, to name a few. The linear-Gaussian state-space model is a common way to describe a time series through the evolution of a hidden state, with the advantage of presenting a simple inference procedure due to the celebrated Kal… ▽ More Modeling and inference with multivariate sequences is central in a number of signal processing applications such as acoustics, social network analysis, biomedical, and finance, to name a few. The linear-Gaussian state-space model is a common way to describe a time series through the evolution of a hidden state, with the advantage of presenting a simple inference procedure due to the celebrated Kalman filter. A fundamental question when analyzing multivariate sequences is the search for relationships between their entries (or the modeled hidden states), especially when the inherent structure is a non-fully connected graph. In such context, graphical modeling combined with parsimony constraints allows to limit the proliferation of parameters and enables a compact data representation which is easier to interpret by the experts. In this work, we propose a novel expectation-minimization algorithm for estimating the linear matrix operator in the state equation of a linear-Gaussian state-space model. Lasso regularization is included in the M-step, that we solved using a proximal splitting Douglas-Rachford algorithm. Numerical experiments illustrate the benefits of the proposed model and inference technique, named GraphEM, over competitors relying on Granger causality. △ Less

Submitted 9 January, 2020; originally announced January 2020.

General risk measures for robust machine learning

Authors: Emilie Chouzenoux, Henri Gérard, Jean-Christophe Pesquet

Abstract: A wide array of machine learning problems are formulated as the minimization of the expectation of a convex loss function on some parameter space. Since the probability distribution of the data of interest is usually unknown, it is is often estimated from training sets, which may lead to poor out-of-sample performance. In this work, we bring new insights in this problem by using the framework whic… ▽ More A wide array of machine learning problems are formulated as the minimization of the expectation of a convex loss function on some parameter space. Since the probability distribution of the data of interest is usually unknown, it is is often estimated from training sets, which may lead to poor out-of-sample performance. In this work, we bring new insights in this problem by using the framework which has been developed in quantitative finance for risk measures. We show that the original min-max problem can be recast as a convex minimization problem under suitable assumptions. We discuss several important examples of robust formulations, in particular by defining ambiguity sets based on $\varphi$-divergences and the Wasserstein metric.We also propose an efficient algorithm for solving the corresponding convex optimization problems involving complex convex constraints. Through simulation examples, we demonstrate that this algorithm scales well on real data sets. △ Less

Submitted 24 May, 2019; v1 submitted 26 April, 2019; originally announced April 2019.

Deep Unfolding of a Proximal Interior Point Method for Image Restoration

Authors: Carla Bertocchi, Emilie Chouzenoux, Marie-Caroline Corbineau, Jean-Christophe Pesquet, Marco Prato

Abstract: Variational methods are widely applied to ill-posed inverse problems for they have the ability to embed prior knowledge about the solution. However, the level of performance of these methods significantly depends on a set of parameters, which can be estimated through computationally expensive and time-consuming methods. In contrast, deep learning offers very generic and efficient architectures, at… ▽ More Variational methods are widely applied to ill-posed inverse problems for they have the ability to embed prior knowledge about the solution. However, the level of performance of these methods significantly depends on a set of parameters, which can be estimated through computationally expensive and time-consuming methods. In contrast, deep learning offers very generic and efficient architectures, at the expense of explainability, since it is often used as a black-box, without any fine control over its output. Deep unfolding provides a convenient approach to combine variational-based and deep learning approaches. Starting from a variational formulation for image restoration, we develop iRestNet, a neural network architecture obtained by unfolding a proximal interior point algorithm. Hard constraints, encoding desirable properties for the restored image, are incorporated into the network thanks to a logarithmic barrier, while the barrier parameter, the stepsize, and the penalization weight are learned by the network. We derive explicit expressions for the gradient of the proximity operator for various choices of constraints, which allows training iRestNet with gradient descent and backpropagation. In addition, we provide theoretical results regarding the stability of the network for a common inverse problem example. Numerical experiments on image deblurring problems show that the proposed approach compares favorably with both state-of-the-art variational and machine learning methods in terms of image quality. △ Less

Submitted 21 January, 2020; v1 submitted 11 December, 2018; originally announced December 2018.

Journal ref: Inverse Problems (2019)

A probabilistic incremental proximal gradient method

Authors: Ömer Deniz Akyildiz, Émilie Chouzenoux, Víctor Elvira, Joaquín Míguez

Abstract: In this paper, we propose a probabilistic optimization method, named probabilistic incremental proximal gradient (PIPG) method, by develo** a probabilistic interpretation of the incremental proximal gradient algorithm. We explicitly model the update rules of the incremental proximal gradient method and develop a systematic approach to propagate the uncertainty of the solution estimate over itera… ▽ More In this paper, we propose a probabilistic optimization method, named probabilistic incremental proximal gradient (PIPG) method, by develo** a probabilistic interpretation of the incremental proximal gradient algorithm. We explicitly model the update rules of the incremental proximal gradient method and develop a systematic approach to propagate the uncertainty of the solution estimate over iterations. The PIPG algorithm takes the form of Bayesian filtering updates for a state-space model constructed by using the cost function. Our framework makes it possible to utilize well-known exact or approximate Bayesian filters, such as Kalman or extended Kalman filters, to solve large-scale regularized optimization problems. △ Less

Submitted 19 June, 2019; v1 submitted 4 December, 2018; originally announced December 2018.

Comments: 5 pages, includes an extra numerical experiment

A Proximal Approach for a Class of Matrix Optimization Problems

Authors: A. Benfenati, E. Chouzenoux, J. -C. Pesquet

Abstract: In recent years, there has been a growing interest in mathematical models leading to the minimization, in a symmetric matrix space, of a Bregman divergence coupled with a regularization term. We address problems of this type within a general framework where the regularization term is split in two parts, one being a spectral function while the other is arbitrary. A Douglas-Rachford approach is prop… ▽ More In recent years, there has been a growing interest in mathematical models leading to the minimization, in a symmetric matrix space, of a Bregman divergence coupled with a regularization term. We address problems of this type within a general framework where the regularization term is split in two parts, one being a spectral function while the other is arbitrary. A Douglas-Rachford approach is proposed to address such problems and a list of proximity operators is provided allowing us to consider various choices for the fit-to-data functional and for the regularization term. Numerical experiments show the validity of this approach for solving convex optimization problems encountered in the context of sparse covariance matrix estimation. Based on our theoretical results, an algorithm is also proposed for noisy graphical lasso where a precision matrix has to be estimated in the presence of noise. The nonconvexity of the resulting objective function is dealt with a majorization-minimization approach, i.e. by building a sequence of convex surrogates and solving the inner optimization subproblems via the aforementioned Douglas-Rachford procedure. We establish conditions for the convergence of this iterative scheme and we illustrate its good numerical performance with respect to state-of-the-art approaches. △ Less

Submitted 23 January, 2018; originally announced January 2018.

MSC Class: 15A18 15B48 62J10 65K10 90C06 90C25 90C26 90C35

A Random Block-Coordinate Douglas-Rachford Splitting Method with Low Computational Complexity for Binary Logistic Regression

Authors: Luis M. Briceno-Arias, Giovanni Chierchia, Emilie Chouzenoux, Jean-Christophe Pesquet

Abstract: In this paper, we propose a new optimization algorithm for sparse logistic regression based on a stochastic version of the Douglas-Rachford splitting method. Our algorithm sweeps the training set by randomly selecting a mini-batch of data at each iteration, and it allows us to update the variables in a block coordinate manner. Our approach leverages the proximity operator of the logistic loss, whi… ▽ More In this paper, we propose a new optimization algorithm for sparse logistic regression based on a stochastic version of the Douglas-Rachford splitting method. Our algorithm sweeps the training set by randomly selecting a mini-batch of data at each iteration, and it allows us to update the variables in a block coordinate manner. Our approach leverages the proximity operator of the logistic loss, which is expressed with the generalized Lambert W function. Experiments carried out on standard datasets demonstrate the efficiency of our approach w.r.t. stochastic gradient-like methods. △ Less

Submitted 25 December, 2017; originally announced December 2017.

A Variational Bayesian Approach for Image Restoration. Application to Image Deblurring with Poisson-Gaussian Noise

Authors: Yosra Marnissi, Yuling Zheng, Emilie Chouzenoux, Jean-Christophe Pesquet

Abstract: In this paper, a methodology is investigated for signal recovery in the presence of non-Gaussian noise. In contrast with regularized minimization approaches often adopted in the literature, in our algorithm the regularization parameter is reliably estimated from the observations. As the posterior density of the unknown parameters is analytically intractable, the estimation problem is derived in a… ▽ More In this paper, a methodology is investigated for signal recovery in the presence of non-Gaussian noise. In contrast with regularized minimization approaches often adopted in the literature, in our algorithm the regularization parameter is reliably estimated from the observations. As the posterior density of the unknown parameters is analytically intractable, the estimation problem is derived in a variational Bayesian framework where the goal is to provide a good approximation to the posterior distribution in order to compute posterior mean estimates. Moreover, a majorization technique is employed to circumvent the difficulties raised by the intricate forms of the non-Gaussian likelihood and of the prior density. We demonstrate the potential of the proposed approach through comparisons with state-of-the-art techniques that are specifically tailored to signal recovery in the presence of mixed Poisson-Gaussian noise. Results show that the proposed approach is efficient and achieves performance comparable with other methods where the regularization parameter is manually tuned from the ground truth. △ Less

Submitted 20 January, 2017; v1 submitted 24 October, 2016; originally announced October 2016.

Convergence Rate Analysis of the Majorize-Minimize Subspace Algorithm -- Extended Version

Authors: Emilie Chouzenoux, Jean-Christophe Pesquet

Abstract: State-of-the-art methods for solving smooth optimization problems are nonlinear conjugate gradient, low memory BFGS, and Majorize-Minimize (MM) subspace algorithms. The MM subspace algorithm which has been introduced more recently has shown good practical performance when compared with other methods on various optimization problems arising in signal and image processing. However, to the best of ou… ▽ More State-of-the-art methods for solving smooth optimization problems are nonlinear conjugate gradient, low memory BFGS, and Majorize-Minimize (MM) subspace algorithms. The MM subspace algorithm which has been introduced more recently has shown good practical performance when compared with other methods on various optimization problems arising in signal and image processing. However, to the best of our knowledge, no general result exists concerning the theoretical convergence rate of the MM subspace algorithm. This paper aims at deriving such convergence rates both for batch and online versions of the algorithm and, in particular, discusses the influence of the choice of the subspace. △ Less

Submitted 19 July, 2016; v1 submitted 23 March, 2016; originally announced March 2016.

Comments: 14 pages

Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed l1/l2 Regularization

Authors: Audrey Repetti, Mai Quyen Pham, Laurent Duval, Emilie Chouzenoux, Jean-Christophe Pesquet

Abstract: The l1/l2 ratio regularization function has shown good performance for retrieving sparse signals in a number of recent works, in the context of blind deconvolution. Indeed, it benefits from a scale invariance property much desirable in the blind context. However, the l1/l2 function raises some difficulties when solving the nonconvex and nonsmooth minimization problems resulting from the use of suc… ▽ More The l1/l2 ratio regularization function has shown good performance for retrieving sparse signals in a number of recent works, in the context of blind deconvolution. Indeed, it benefits from a scale invariance property much desirable in the blind context. However, the l1/l2 function raises some difficulties when solving the nonconvex and nonsmooth minimization problems resulting from the use of such a penalty term in current restoration methods. In this paper, we propose a new penalty based on a smooth approximation to the l1/l2 function. In addition, we develop a proximal-based algorithm to solve variational problems involving this function and we derive theoretical convergence results. We demonstrate the effectiveness of our method through a comparison with a recent alternating optimization strategy dealing with the exact l1/l2 term, on an application to seismic data blind deconvolution. △ Less

Submitted 8 November, 2014; v1 submitted 21 July, 2014; originally announced July 2014.

Comments: 5 pages

Journal ref: IEEE Signal Processing Letters, May 2015, Volume 22, Number 5, pages 539-543

A Majorize-Minimize subspace approach for l2-l0 image regularization

Authors: Emilie Chouzenoux, Anna Jezierska, Jean-Christophe Pesquet, Hugues Talbot

Abstract: In this work, we consider a class of differentiable criteria for sparse image computing problems, where a nonconvex regularization is applied to an arbitrary linear transform of the target image. As special cases, it includes edge-preserving measures or frame-analysis potentials commonly used in image processing. As shown by our asymptotic results, the l2-l0 penalties we consider may be employed t… ▽ More In this work, we consider a class of differentiable criteria for sparse image computing problems, where a nonconvex regularization is applied to an arbitrary linear transform of the target image. As special cases, it includes edge-preserving measures or frame-analysis potentials commonly used in image processing. As shown by our asymptotic results, the l2-l0 penalties we consider may be employed to provide approximate solutions to l0-penalized optimization problems. One of the advantages of the proposed approach is that it allows us to derive an efficient Majorize-Minimize subspace algorithm. The convergence of the algorithm is investigated by using recent results in nonconvex optimization. The fast convergence properties of the proposed optimization method are illustrated through image processing examples. In particular, its effectiveness is demonstrated on several data recovery problems. △ Less

Submitted 26 August, 2013; v1 submitted 29 December, 2011; originally announced December 2011.

MSC Class: 65K10; 90C26; 68U10; 94A08; 47A52; 65F22

Journal ref: SIAM Journal on Imaging Science, Vol. 6, No. 1, pages 563-591, 2013