Showing 1–34 of 34 results for author: Herzet, C

A New Branch-and-Bound Pruning Framework for $\ell_0$-Regularized Problems

Authors: Theo Guyard, Cédric Herzet, Clément Elvira, Ayşe-Nur Arslan

Abstract: We consider the resolution of learning problems involving $\ell_0$-regularization via Branch-and-Bound (BnB) algorithms. These methods explore regions of the feasible space of the problem and check whether they do not contain solutions through "pruning tests". In standard implementations, evaluating a pruning test requires to solve a convex optimization problem, which may result in computational b… ▽ More We consider the resolution of learning problems involving $\ell_0$-regularization via Branch-and-Bound (BnB) algorithms. These methods explore regions of the feasible space of the problem and check whether they do not contain solutions through "pruning tests". In standard implementations, evaluating a pruning test requires to solve a convex optimization problem, which may result in computational bottlenecks. In this paper, we present an alternative to implement pruning tests for some generic family of $\ell_0$-regularized problems. Our proposed procedure allows the simultaneous assessment of several regions and can be embedded in standard BnB implementations with a negligible computational overhead. We show through numerical simulations that our pruning strategy can improve the solving time of BnB procedures by several orders of magnitude for typical problems encountered in machine-learning applications. △ Less

Submitted 3 June, 2024; originally announced June 2024.

One to beat them all: "RYU'' -- a unifying framework for the construction of safe balls

Authors: Thu-Le Tran, Clément Elvira, Hong-Phuong Dang, Cédric Herzet

Abstract: In this paper, we put forth a novel framework (named ``RYU'') for the construction of ``safe'' balls, i.e. regions that provably contain the dual solution of a target optimization problem. We concentrate on the standard setup where the cost function is the sum of two terms: a closed, proper, convex Lipschitz-smooth function and a closed, proper, convex function. The RYU framework is shown to gener… ▽ More In this paper, we put forth a novel framework (named ``RYU'') for the construction of ``safe'' balls, i.e. regions that provably contain the dual solution of a target optimization problem. We concentrate on the standard setup where the cost function is the sum of two terms: a closed, proper, convex Lipschitz-smooth function and a closed, proper, convex function. The RYU framework is shown to generalize or improve upon all the results proposed in the last decade for the considered family of optimization problems. △ Less

Submitted 1 December, 2023; originally announced December 2023.

Comments: 19 pages, 1 table

Safe Peeling for L0-Regularized Least-Squares with supplementary material

Authors: Théo Guyard, Gilles Monnoyer, Clément Elvira, Cédric Herzet

Abstract: We introduce a new methodology dubbed ``safe peeling'' to accelerate the resolution of L0-regularized least-squares problems via a Branch-and-Bound (BnB) algorithm. Our procedure enables to tighten the convex relaxation considered at each node of the BnB decision tree and therefore potentially allows for more aggressive pruning. Numerical simulations show that our proposed methodology leads to sig… ▽ More We introduce a new methodology dubbed ``safe peeling'' to accelerate the resolution of L0-regularized least-squares problems via a Branch-and-Bound (BnB) algorithm. Our procedure enables to tighten the convex relaxation considered at each node of the BnB decision tree and therefore potentially allows for more aggressive pruning. Numerical simulations show that our proposed methodology leads to significant gains in terms of number of nodes explored and overall solving time.s show that our proposed methodology leads to significant gains in terms of number of nodes explored and overall solving time. △ Less

Submitted 6 June, 2023; v1 submitted 28 February, 2023; originally announced February 2023.

Beyond GAP screening for Lasso by exploiting new dual cutting half-spaces with supplementary material

Authors: Thu-Le Tran, Clément Elvira, Hong-Phuong Dang, Cédric Herzet

Abstract: In this paper, we propose a novel safe screening test for Lasso. Our procedure is based on a safe region with a dome geometry and exploits a canonical representation of the set of half-spaces (referred to as "dual cutting half-spaces" in this paper) containing the dual feasible set. The proposed safe region is shown to be always included in the state-of-the-art "GAP Sphere" and "GAP Dome" proposed… ▽ More In this paper, we propose a novel safe screening test for Lasso. Our procedure is based on a safe region with a dome geometry and exploits a canonical representation of the set of half-spaces (referred to as "dual cutting half-spaces" in this paper) containing the dual feasible set. The proposed safe region is shown to be always included in the state-of-the-art "GAP Sphere" and "GAP Dome" proposed by Fercoq et al. (and strictly so under very mild conditions) while involving the same computational burden. Numerical experiments confirm that our new dome enables to devise more powerful screening tests than GAP regions and lead to significant acceleration to solve Lasso. △ Less

Submitted 2 March, 2022; originally announced March 2022.

Comments: 6 pages, 2 figures

Safe rules for the identification of zeros in the solutions of the SLOPE problem

Authors: Clément Elvira, Cédric Herzet

Abstract: In this paper we propose a methodology to accelerate the resolution of the so-called "Sorted L-One Penalized Estimation" (SLOPE) problem. Our method leverages the concept of "safe screening", well-studied in the literature for \textit{group-separable} sparsity-inducing norms, and aims at identifying the zeros in the solution of SLOPE. More specifically, we derive a set of \(\tfrac{n(n+1)}{2}\) ine… ▽ More In this paper we propose a methodology to accelerate the resolution of the so-called "Sorted L-One Penalized Estimation" (SLOPE) problem. Our method leverages the concept of "safe screening", well-studied in the literature for \textit{group-separable} sparsity-inducing norms, and aims at identifying the zeros in the solution of SLOPE. More specifically, we derive a set of \(\tfrac{n(n+1)}{2}\) inequalities for each element of the \(n\)-dimensional primal vector and prove that the latter can be safely screened if some subsets of these inequalities are verified. We propose moreover an efficient algorithm to jointly apply the proposed procedure to all the primal variables. Our procedure has a complexity \(\mathcal{O}(n\log n + LT)\) where \(T\leq n\) is a problem-dependent constant and \(L\) is the number of zeros identified by the tests. Numerical experiments confirm that, for a prescribed computational budget, the proposed methodology leads to significant improvements of the solving precision. △ Less

Submitted 4 October, 2022; v1 submitted 22 October, 2021; originally announced October 2021.

Comments: 26 pages, 3 figures

Node-screening tests for L0-penalized least-squares problem with supplementary material

Authors: Théo Guyard, Cédric Herzet, Clément Elvira

Abstract: We present a novel screening methodology to safely discard irrelevant nodes within a generic branch-and-bound (BnB) algorithm solving the l0-penalized least-squares problem. Our contribution is a set of two simple tests to detect sets of feasible vectors that cannot yield optimal solutions. This allows to prune nodes of the BnB search tree, thus reducing the overall optimization time. One cornerst… ▽ More We present a novel screening methodology to safely discard irrelevant nodes within a generic branch-and-bound (BnB) algorithm solving the l0-penalized least-squares problem. Our contribution is a set of two simple tests to detect sets of feasible vectors that cannot yield optimal solutions. This allows to prune nodes of the BnB search tree, thus reducing the overall optimization time. One cornerstone of our contribution is a nesting property between tests at different nodes that allows to implement them with a low computational cost. Our work leverages the concept of safe screening, well known for sparsity-inducing convex problems, and some recent advances in this field for l0-penalized regression problems. △ Less

Submitted 3 February, 2022; v1 submitted 14 October, 2021; originally announced October 2021.

Screen & Relax: Accelerating the resolution of Elastic-net by safe identification of the solution support

Authors: Théo Guyard, Cédric Herzet, Clément Elvira

Abstract: In this paper, we propose a procedure to accelerate the resolution of the well-known "Elastic-Net" problem. Our procedure is based on the (partial) identification of the solution support and the reformulation of the original problem into a problem of reduced dimension. The identification of the support leverages the novel concept of "safe relaxing" where one aims to identify non-zero coefficients… ▽ More In this paper, we propose a procedure to accelerate the resolution of the well-known "Elastic-Net" problem. Our procedure is based on the (partial) identification of the solution support and the reformulation of the original problem into a problem of reduced dimension. The identification of the support leverages the novel concept of "safe relaxing" where one aims to identify non-zero coefficients of the solution. It can be viewed as a dual approach to "safe screening" introduced in the last decade and allowing to reduce the problem dimension using the identification of zero coefficients of the solution. We show numerically that combining both methodologies in a "Screen & Relax" strategy enables to significantly improve the tradeoff between complexity and accuracy achievable by standard resolution techniques. △ Less

Submitted 3 February, 2022; v1 submitted 14 October, 2021; originally announced October 2021.

State-Of-The-Art Algorithms For Low-Rank Dynamic Mode Decomposition

Authors: Patrick Heas, Cedric Herzet

Abstract: This technical note reviews sate-of-the-art algorithms for linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition (DMD). While repeating several parts of our article "low-rank dynamic mode decomposition: an exact and tractable solution", this work provides additional details useful for building a comprehensive picture of state-of-the-art methods. This technical note reviews sate-of-the-art algorithms for linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition (DMD). While repeating several parts of our article "low-rank dynamic mode decomposition: an exact and tractable solution", this work provides additional details useful for building a comprehensive picture of state-of-the-art methods. △ Less

Submitted 20 August, 2021; originally announced August 2021.

Comments: arXiv admin note: substantial text overlap with arXiv:1610.02962

Translation-invariant interpolation of parametric dictionaries

Authors: Frédéric Champagnat, Cédric Herzet

Abstract: In this communication, we address the problem of approximating the atoms of a parametric dictionary, commonly encountered in the context of sparse representations in "continuous" dictionaries. We focus on the case of translation-invariant dictionaries, where the inner product between atoms only depends on the difference between parameters. We investigate the following general question: is there so… ▽ More In this communication, we address the problem of approximating the atoms of a parametric dictionary, commonly encountered in the context of sparse representations in "continuous" dictionaries. We focus on the case of translation-invariant dictionaries, where the inner product between atoms only depends on the difference between parameters. We investigate the following general question: is there some low-rank approximation of the dictionary $ which interpolates a subset of atoms while preserving the translation-invariant nature of the original dictionary? We derive necessary and sufficient conditions characterizing the existence of such an "interpolating" and "translation-invariant" low-rank approximation. Moreover, we provide closed-form expressions of such a dictionary when it exists. We illustrate the applicability of our results in the case of a two-dimensional isotropic Gaussian dictionary. We show that, in this particular setup, the proposed approximation framework outperforms standard Taylor approximation. △ Less

Submitted 29 November, 2020; originally announced November 2020.

Comments: in Proceedings of iTWIST'20, Paper-ID: 49, Nantes, France, December, 2-4, 2020

Continuous dictionaries meet low-rank tensor approximations

Authors: Clement Elvira, Jeremy E. Cohen, Cedric Herzet, Remi Gribonval

Abstract: In this short paper we bridge two seemingly unrelated sparse approximation topics: continuous sparse coding and low-rank approximations. We show that for a specific choice of continuous dictionary, linear systems with nuclear-norm regularization have the same solutions as a BLasso problem. Although this fact was already partially understood in the matrix case, we further show that for tensor data,… ▽ More In this short paper we bridge two seemingly unrelated sparse approximation topics: continuous sparse coding and low-rank approximations. We show that for a specific choice of continuous dictionary, linear systems with nuclear-norm regularization have the same solutions as a BLasso problem. Although this fact was already partially understood in the matrix case, we further show that for tensor data, using BLasso solvers for the low-rank approximation problem leads to a new branch of optimization methods yet vastly unexplored. In particular, the proposed Frank-Wolfe algorithm is showcased on an automatic tensor rank selection problem. △ Less

Submitted 14 September, 2020; originally announced September 2020.

Comments: in Proceedings of iTWIST'20, Paper-ID: 28, Nantes, France, December, 2-4, 2020

Generalized Kernel-Based Dynamic Mode Decomposition

Authors: Patrick Heas, Cedric Herzet, Benoit Combes

Abstract: Reduced modeling in high-dimensional reproducing kernel Hilbert spaces offers the opportunity to approximate efficiently non-linear dynamics. In this work, we devise an algorithm based on low rank constraint optimization and kernel-based computation that generalizes a recent approach called "kernel-based dynamic mode decomposition". This new algorithm is characterized by a gain in approximation ac… ▽ More Reduced modeling in high-dimensional reproducing kernel Hilbert spaces offers the opportunity to approximate efficiently non-linear dynamics. In this work, we devise an algorithm based on low rank constraint optimization and kernel-based computation that generalizes a recent approach called "kernel-based dynamic mode decomposition". This new algorithm is characterized by a gain in approximation accuracy, as evidenced by numerical simulations, and in computational complexity. △ Less

Submitted 11 February, 2020; originally announced February 2020.

Comments: 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020). arXiv admin note: substantial text overlap with arXiv:1710.10919

Safe squeezing for antisparse coding

Authors: Clément Elvira, Cédric Herzet

Abstract: Spreading the information over all coefficients of a representation is a desirable property in many applications such as digital communication or machine learning. This so-called antisparse representation can be obtained by solving a convex program involving an $\ell_\infty$-norm penalty combined with a quadratic discrepancy. In this paper, we propose a new methodology, dubbed safe squeezing, to a… ▽ More Spreading the information over all coefficients of a representation is a desirable property in many applications such as digital communication or machine learning. This so-called antisparse representation can be obtained by solving a convex program involving an $\ell_\infty$-norm penalty combined with a quadratic discrepancy. In this paper, we propose a new methodology, dubbed safe squeezing, to accelerate the computation of antisparse representation. We describe a test that allows to detect saturated entries in the solution of the optimization problem. The contribution of these entries is compacted into a single vector, thus operating a form of dimensionality reduction. We propose two algorithms to solve the resulting lower dimensional problem. Numerical experiments show the effectiveness of the proposed method to detect the saturated components of the solution and illustrates the induced computational gains in the resolution of the antisparse problem. △ Less

Submitted 28 March, 2020; v1 submitted 18 November, 2019; originally announced November 2019.

When does OMP achieve exact recovery with continuous dictionaries?

Authors: Clément Elvira, Rémi Gribonval, Charles Soussen, Cédric Herzet

Abstract: This paper presents new theoretical results on sparse recovery guarantees for a greedy algorithm, Orthogonal Matching Pursuit (OMP), in the context of continuous parametric dictionaries. Here, the continuous setting means that the dictionary is made up of an infinite uncountable number of atoms. In this work, we rely on the Hilbert structure of the observation space to express our recovery results… ▽ More This paper presents new theoretical results on sparse recovery guarantees for a greedy algorithm, Orthogonal Matching Pursuit (OMP), in the context of continuous parametric dictionaries. Here, the continuous setting means that the dictionary is made up of an infinite uncountable number of atoms. In this work, we rely on the Hilbert structure of the observation space to express our recovery results as a property of the kernel defined by the inner product between two atoms. Using a continuous extension of Tropp's Exact Recovery Condition, we identify key assumptions allowing to analyze OMP in the continuous setting. Under these assumptions, OMP unambiguously identifies in exactly $k$ steps the atom parameters from any observed linear combination of $k$ atoms. These parameters play the role of the so-called support of a sparse representation in traditional sparse recovery. In our paper, any kernel and set of parameters that satisfy these conditions are said to be admissible. In the one-dimensional setting, we exhibit a family of kernels relying on completely monotone functions for which admissibility holds for any set of atom parameters. For higher dimensional parameter spaces, the analysis turns out to be more subtle. An additional assumption, so-called axis admissibility, is imposed to ensure a form of delayed recovery (in at most $k^D$ steps, where $D$ is the dimension of the parameter space). Furthermore, guarantees for recovery in exactly $k$ steps are derived under an additional algebraic condition involving a finite subset of atoms (built as an extension of the set of atoms to be recovered). We show that the latter technical conditions simplify in the case of Laplacian kernels, allowing us to derive simple conditions for $k$-step exact recovery, and to carry out a coherence-based analysis in terms of a minimum separation assumption between the atoms to be recovered. △ Less

Submitted 22 June, 2020; v1 submitted 12 April, 2019; originally announced April 2019.

Comments: 52 pages, 2 figures

Low-rank Approximation of Linear Maps

Authors: Patrick Heas, Cedric Herzet

Abstract: This work provides closed-form solutions and minimum achievable errors for a large class of low-rank approximation problems in Hilbert spaces. The proposed theorem generalizes to the case of bounded linear operators the previous results obtained in the finite dimensional case for the Frobenius norm. The theorem provides the basis for the design of tractable algorithms for kernel or continuous DMD. This work provides closed-form solutions and minimum achievable errors for a large class of low-rank approximation problems in Hilbert spaces. The proposed theorem generalizes to the case of bounded linear operators the previous results obtained in the finite dimensional case for the Frobenius norm. The theorem provides the basis for the design of tractable algorithms for kernel or continuous DMD. △ Less

Submitted 6 January, 2023; v1 submitted 21 December, 2018; originally announced December 2018.

Efficient atom selection strategy for iterative sparse approximations

Authors: Clément Dorffer, Angélique Drémeau, Cedric Herzet

Abstract: We propose a low-computational strategy for the efficient implementation of the "atom selection step" in sparse representation algorithms. The proposed procedure is based on simple tests enabling to identify subsets of atoms which cannot be selected. Our procedure applies on both discrete or continuous dictionaries. Experiments performed on DOA and Gaussian deconvolution problems show the computat… ▽ More We propose a low-computational strategy for the efficient implementation of the "atom selection step" in sparse representation algorithms. The proposed procedure is based on simple tests enabling to identify subsets of atoms which cannot be selected. Our procedure applies on both discrete or continuous dictionaries. Experiments performed on DOA and Gaussian deconvolution problems show the computational gain induced by the proposed approach. △ Less

Submitted 5 December, 2018; originally announced December 2018.

Comments: in Proceedings of iTWIST'18, Paper-ID: 14, Marseille, France, November, 21-23, 2018

Sea surface temperature prediction and reconstruction using patch-level neural network representations

Authors: Said Ouala, Cedric Herzet, Ronan Fablet

Abstract: The forecasting and reconstruction of ocean and atmosphere dynamics from satellite observation time series are key challenges. While model-driven representations remain the classic approaches, data-driven representations become more and more appealing to benefit from available large-scale observation and simulation datasets. In this work we investigate the relevance of recently introduced bilinear… ▽ More The forecasting and reconstruction of ocean and atmosphere dynamics from satellite observation time series are key challenges. While model-driven representations remain the classic approaches, data-driven representations become more and more appealing to benefit from available large-scale observation and simulation datasets. In this work we investigate the relevance of recently introduced bilinear residual neural network representations, which mimic numerical integration schemes such as Runge-Kutta, for the forecasting and assimilation of geophysical fields from satellite-derived remote sensing data. As a case-study, we consider satellite-derived Sea Surface Temperature time series off South Africa, which involves intense and complex upper ocean dynamics. Our numerical experiments demonstrate that the proposed patch-level neural-network-based representations outperform other data-driven models, including analog schemes, both in terms of forecasting and missing data interpolation performance with a relative gain up to 50\% for highly dynamic areas. △ Less

Submitted 31 May, 2018; originally announced June 2018.

A Mathematical Characterization of the Performance of the "Multi-Slice" Projector

Authors: C. Herzet, M. Diallo, P. Héas

Abstract: We consider an enhanced version of the well-kwown "Petrov-Galerkin" projection in Hilbert spaces. The proposed procedure, dubbed "multi-slice" projector, exploits the fact that the sought solution belongs to the intersection of several high-dimensional slices. This setup is for example of interest in model-order reduction where this type of prior may be computed off-line. In this note, we provide… ▽ More We consider an enhanced version of the well-kwown "Petrov-Galerkin" projection in Hilbert spaces. The proposed procedure, dubbed "multi-slice" projector, exploits the fact that the sought solution belongs to the intersection of several high-dimensional slices. This setup is for example of interest in model-order reduction where this type of prior may be computed off-line. In this note, we provide a mathematical characterization of the performance achievable by the multi-slice projector and compare the latter with the results holding in the Petrov-Galerkin setup. In particular, we illustrate the superiority of the multi-slice approach in certain situations. △ Less

Submitted 10 May, 2018; originally announced May 2018.

Bilinear residual Neural Network for the identification and forecasting of dynamical systems

Authors: Ronan Fablet, Said Ouala, Cedric Herzet

Abstract: Due to the increasing availability of large-scale observation and simulation datasets, data-driven representations arise as efficient and relevant computation representations of dynamical systems for a wide range of applications, where model-driven models based on ordinary differential equation remain the state-of-the-art approaches. In this work, we investigate neural networks (NN) as physically-… ▽ More Due to the increasing availability of large-scale observation and simulation datasets, data-driven representations arise as efficient and relevant computation representations of dynamical systems for a wide range of applications, where model-driven models based on ordinary differential equation remain the state-of-the-art approaches. In this work, we investigate neural networks (NN) as physically-sound data-driven representations of such systems. Reinterpreting Runge-Kutta methods as graphical models, we consider a residual NN architecture and introduce bilinear layers to embed non-linearities which are intrinsic features of dynamical systems. From numerical experiments for classic dynamical systems, we demonstrate the relevance of the proposed NN-based architecture both in terms of forecasting performance and model identification. △ Less

Submitted 19 December, 2017; originally announced December 2017.

Comments: Submitted

Non-Linear Reduced Modeling by Generalized Kernel-Based Dynamic Mode Decomposition

Authors: Patrick Héas, Cédric Herzet, Benoit Combès

Abstract: Reduced modeling of a computationally demanding dynamical system aims at approximating its trajectories, while optimizing the trade-off between accuracy and computational complexity. In this work, we propose to achieve such an approximation by first embedding the trajectories in a reproducing kernel Hilbert space (RKHS), which exhibits appealing approximation and computational capabilities, and th… ▽ More Reduced modeling of a computationally demanding dynamical system aims at approximating its trajectories, while optimizing the trade-off between accuracy and computational complexity. In this work, we propose to achieve such an approximation by first embedding the trajectories in a reproducing kernel Hilbert space (RKHS), which exhibits appealing approximation and computational capabilities, and then solving the associated reduced model problem. More specifically, we propose a new efficient algorithm for data-driven reduced modeling of non-linear dynamics based on linear approximations in a RKHS. This algorithm takes advantage of the closed-form solution of a low-rank constraint optimization problem while exploiting advantageously kernel-based computations. Reduced modeling with this algorithm reveals a gain in approximation accuracy, as shown by numerical simulations, and in complexity with respect to existing approaches. △ Less

Submitted 21 February, 2020; v1 submitted 30 October, 2017; originally announced October 2017.

Joint Screening Tests for LASSO

Authors: C. Herzet, A. Drémeau

Abstract: This paper focusses on "safe" screening techniques for the LASSO problem. Motivated by the need for low-complexity algorithms, we propose a new approach, dubbed "joint" screening test, allowing to screen a set of atoms by carrying out one single test. The approach is particularized to two different sets of atoms, respectively expressed as sphere and dome regions. After presenting the mathematical… ▽ More This paper focusses on "safe" screening techniques for the LASSO problem. Motivated by the need for low-complexity algorithms, we propose a new approach, dubbed "joint" screening test, allowing to screen a set of atoms by carrying out one single test. The approach is particularized to two different sets of atoms, respectively expressed as sphere and dome regions. After presenting the mathematical derivations of the tests, we elaborate on their relative effectiveness and discuss the practical use of such procedures. △ Less

Submitted 9 November, 2017; v1 submitted 26 October, 2017; originally announced October 2017.

Reduced Modeling of Unknown Trajectories

Authors: Patrick Héas, Cédric Herzet

Abstract: This paper deals with model order reduction of parametrical dynamical systems. We consider the specific setup where the distribution of the system's trajectories is unknown but the following two sources of information are available: \textit{(i)} some "rough" prior knowledge on the system's realisations; \textit{(ii)} a set of "incomplete" observations of the system's trajectories. We propose a Bay… ▽ More This paper deals with model order reduction of parametrical dynamical systems. We consider the specific setup where the distribution of the system's trajectories is unknown but the following two sources of information are available: \textit{(i)} some "rough" prior knowledge on the system's realisations; \textit{(ii)} a set of "incomplete" observations of the system's trajectories. We propose a Bayesian methodological framework to build reduced-order models (ROMs) by exploiting these two sources of information. We emphasise that complementing the prior knowledge with the collected data provably enhances the knowledge of the distribution of the system's trajectories. We then propose an implementation of the proposed methodology based on Monte-Carlo methods. In this context, we show that standard ROM learning techniques, such e.g. Proper Orthogonal Decomposition or Dynamic Mode Decomposition, can be revisited and recast within the probabilistic framework considered in this paper.~We illustrate the performance of the proposed approach by numerical results obtained for a standard geophysical model. △ Less

Submitted 9 May, 2017; v1 submitted 28 February, 2017; originally announced February 2017.

Optimal Low-Rank Dynamic Mode Decomposition

Authors: Patrick Héas, Cédric Herzet

Abstract: Dynamic Mode Decomposition (DMD) has emerged as a powerful tool for analyzing the dynamics of non-linear systems from experimental datasets. Recently, several attempts have extended DMD to the context of low-rank approximations. This extension is of particular interest for reduced-order modeling in various applicative domains, e.g. for climate prediction, to study molecular dynamics or micro-elect… ▽ More Dynamic Mode Decomposition (DMD) has emerged as a powerful tool for analyzing the dynamics of non-linear systems from experimental datasets. Recently, several attempts have extended DMD to the context of low-rank approximations. This extension is of particular interest for reduced-order modeling in various applicative domains, e.g. for climate prediction, to study molecular dynamics or micro-electromechanical devices. This low-rank extension takes the form of a non-convex optimization problem. To the best of our knowledge, only sub-optimal algorithms have been proposed in the literature to compute the solution of this problem. In this paper, we prove that there exists a closed-form optimal solution to this problem and design an effective algorithm to compute it based on Singular Value Decomposition (SVD). A toy-example illustrates the gain in performance of the proposed algorithm compared to state-of-the-art techniques. △ Less

Submitted 17 May, 2018; v1 submitted 4 January, 2017; originally announced January 2017.

Comments: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASPP), New Orleans, USA, 2017

Low-Rank Dynamic Mode Decomposition: An Exact and Tractable Solution

Authors: Patrick Héas, Cédric Herzet

Abstract: This work studies the linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition (DMD). Searching this approximation in a data-driven approach is formalised as attempting to solve a low-rank constrained optimisation problem. This problem is non-convex and state-of-the-art algorithms are all sub-optimal. This paper shows that there exists a closed-form solu… ▽ More This work studies the linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition (DMD). Searching this approximation in a data-driven approach is formalised as attempting to solve a low-rank constrained optimisation problem. This problem is non-convex and state-of-the-art algorithms are all sub-optimal. This paper shows that there exists a closed-form solution, which is computed in polynomial time, and characterises the l2-norm of the optimal approximation error. The paper also proposes low-complexity algorithms building reduced models from this optimal solution, based on singular value decomposition or eigen value decomposition. The algorithms are evaluated by numerical simulations using synthetic and physical data benchmarks. △ Less

Submitted 20 August, 2021; v1 submitted 10 October, 2016; originally announced October 2016.

Journal ref: Journal of Nonlinear Science, 2021

Model Reduction from Partial Observations

Authors: C. Herzet, P. Héas, A. Drémeau

Abstract: This paper deals with model-order reduction of parametric partial differential equations (PPDE). More specifically, we consider the problem of finding a good approximation subspace of the solution manifold of the PPDE when only partial information on the latter is available. We assume that two sources of information are available: i) a "rough" prior knowledge, taking the form of a manifold contain… ▽ More This paper deals with model-order reduction of parametric partial differential equations (PPDE). More specifically, we consider the problem of finding a good approximation subspace of the solution manifold of the PPDE when only partial information on the latter is available. We assume that two sources of information are available: i) a "rough" prior knowledge, taking the form of a manifold containing the target solution manifold, ii) partial linear measurements of the solutions of the PPDE (the term partial refers to the fact that observation operator cannot be inverted). We provide and study several tools to derive good approximation subspaces from these two sources of information. We first identify the best worst-case performance achievable in this setup and propose simple procedures to approximate the corresponding optimal approximation subspace. We then provide, in a simplified setup, a theoretical analysis relating the achievable reduction performance to the choice of the observation operator and the prior knowledge available on the solution manifold. △ Less

Submitted 3 July, 2017; v1 submitted 28 September, 2016; originally announced September 2016.

DOA estimation in structured phase-noisy environments: technical report

Authors: Angélique Drémeau, Cédric Herzet

Abstract: In this paper we focus on the problem of estimating the directions of arrival (DOA) of a set of incident plane waves. Unlike many previous works, which assume that the received observations are only affected by additive noise, we consider the setup where some phase noise also corrupts the data (as for example observed in atmospheric sound propagation or underwater acoustics). We propose a new meth… ▽ More In this paper we focus on the problem of estimating the directions of arrival (DOA) of a set of incident plane waves. Unlike many previous works, which assume that the received observations are only affected by additive noise, we consider the setup where some phase noise also corrupts the data (as for example observed in atmospheric sound propagation or underwater acoustics). We propose a new methodology to solve this problem in a Bayesian framework by resorting to a variational mean-field approximation. Our simulation results illustrate the benefits of carefully accounting for the phase noise in the DOA estimation process. △ Less

Submitted 9 January, 2017; v1 submitted 12 September, 2016; originally announced September 2016.

Reduced-Order Modeling Of Hidden Dynamics

Authors: Patrick Héas, Cédric Herzet

Abstract: The objective of this paper is to investigate how noisy and incomplete observations can be integrated in the process of building a reduced-order model. This problematic arises in many scientific domains where there exists a need for accurate low-order descriptions of highly-complex phenomena, which can not be directly and/or deterministically observed. Within this context, the paper proposes a p… ▽ More The objective of this paper is to investigate how noisy and incomplete observations can be integrated in the process of building a reduced-order model. This problematic arises in many scientific domains where there exists a need for accurate low-order descriptions of highly-complex phenomena, which can not be directly and/or deterministically observed. Within this context, the paper proposes a probabilistic framework for the construction of "POD-Galerkin" reduced-order models. Assuming a hidden Markov chain, the inference integrates the uncertainty of the hidden states relying on their posterior distribution. Simulations show the benefits obtained by exploiting the proposed framework. △ Less

Submitted 17 May, 2018; v1 submitted 8 October, 2015; originally announced October 2015.

Comments: 5 pages, 2 figures

An Efficient Algorithm for Video Super-Resolution Based On a Sequential Model

Authors: Patrick Héas, Angélique Drémeau, Cédric Herzet

Abstract: In this work, we propose a novel procedure for video super-resolution, that is the recovery of a sequence of high-resolution images from its low-resolution counterpart. Our approach is based on a "sequential" model (i.e., each high-resolution frame is supposed to be a displaced version of the preceding one) and considers the use of sparsity-enforcing priors. Both the recovery of the high-resolutio… ▽ More In this work, we propose a novel procedure for video super-resolution, that is the recovery of a sequence of high-resolution images from its low-resolution counterpart. Our approach is based on a "sequential" model (i.e., each high-resolution frame is supposed to be a displaced version of the preceding one) and considers the use of sparsity-enforcing priors. Both the recovery of the high-resolution images and the motion fields relating them is tackled. This leads to a large-dimensional, non-convex and non-smooth problem. We propose an algorithmic framework to address the latter. Our approach relies on fast gradient evaluation methods and modern optimization techniques for non-differentiable/non-convex problems. Unlike some other previous works, we show that there exists a provably-convergent method with a complexity linear in the problem dimensions. We assess the proposed optimization method on {several video benchmarks and emphasize its good performance with respect to the state of the art.} △ Less

Submitted 15 February, 2016; v1 submitted 1 June, 2015; originally announced June 2015.

Comments: 37 pages, SIAM Journal on Imaging Sciences, 2016

Proceedings of the second "international Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST'14)

Authors: L. Jacques, C. De Vleeschouwer, Y. Boursier, P. Sudhakar, C. De Mol, A. Pizurica, S. Anthoine, P. Vandergheynst, P. Frossard, C. Bilen, S. Kitic, N. Bertin, R. Gribonval, N. Boumal, B. Mishra, P. -A. Absil, R. Sepulchre, S. Bundervoet, C. Schretter, A. Dooms, P. Schelkens, O. Chabiron, F. Malgouyres, J. -Y. Tourneret, N. Dobigeon , et al. (42 additional authors not shown)

Abstract: The implicit objective of the biennial "international - Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST) is to foster collaboration between international scientific teams by disseminating ideas through both specific oral/poster presentations and free discussions. For its second edition, the iTWIST workshop took place in the medieval and picturesque town of Namur in… ▽ More The implicit objective of the biennial "international - Traveling Workshop on Interactions between Sparse models and Technology" (iTWIST) is to foster collaboration between international scientific teams by disseminating ideas through both specific oral/poster presentations and free discussions. For its second edition, the iTWIST workshop took place in the medieval and picturesque town of Namur in Belgium, from Wednesday August 27th till Friday August 29th, 2014. The workshop was conveniently located in "The Arsenal" building within walking distance of both hotels and town center. iTWIST'14 has gathered about 70 international participants and has featured 9 invited talks, 10 oral presentations, and 14 posters on the following themes, all related to the theory, application and generalization of the "sparsity paradigm": Sparsity-driven data sensing and processing; Union of low dimensional subspaces; Beyond linear and convex inverse problem; Matrix/manifold/graph sensing/processing; Blind inverse problems and dictionary learning; Sparsity and computational neuroscience; Information theory, geometry and randomness; Complexity/accuracy tradeoffs in numerical methods; Sparsity? What's next?; Sparse machine learning and inference. △ Less

Submitted 9 October, 2014; v1 submitted 2 October, 2014; originally announced October 2014.

Comments: 69 pages, 24 extended abstracts, iTWIST'14 website: http://sites.google.com/site/itwist14

Bayesian Pursuit Algorithms

Authors: Cédric Herzet, Angélique Drémeau

Abstract: This paper addresses the sparse representation (SR) problem within a general Bayesian framework. We show that the Lagrangian formulation of the standard SR problem, i.e., $\mathbf{x}^\star=\arg\min_\mathbf{x} \lbrace \| \mathbf{y}-\mathbf{D}\mathbf{x} \|_2^2+λ\| \mathbf{x}\|_0 \rbrace$, can be regarded as a limit case of a general maximum a posteriori (MAP) problem involving Bernoulli-Gaussian var… ▽ More This paper addresses the sparse representation (SR) problem within a general Bayesian framework. We show that the Lagrangian formulation of the standard SR problem, i.e., $\mathbf{x}^\star=\arg\min_\mathbf{x} \lbrace \| \mathbf{y}-\mathbf{D}\mathbf{x} \|_2^2+λ\| \mathbf{x}\|_0 \rbrace$, can be regarded as a limit case of a general maximum a posteriori (MAP) problem involving Bernoulli-Gaussian variables. We then propose different tractable implementations of this MAP problem that we refer to as "Bayesian pursuit algorithms". The Bayesian algorithms are shown to have strong connections with several well-known pursuit algorithms of the literature (e.g., MP, OMP, StOMP, CoSaMP, SP) and generalize them in several respects. In particular, i) they allow for atom deselection; ii) they can include any prior information about the probability of occurrence of each atom within the selection process; iii) they can encompass the estimation of unkown model parameters into their recursions. △ Less

Submitted 29 January, 2014; originally announced January 2014.

Relaxed Recovery Conditions for OMP/OLS by Exploiting both Coherence and Decay

Authors: Cédric Herzet, Angélique Drémeau, Charles Soussen

Abstract: We propose extended coherence-based conditions for exact sparse support recovery using orthogonal matching pursuit (OMP) and orthogonal least squares (OLS). Unlike standard uniform guarantees, we embed some information about the decay of the sparse vector coefficients in our conditions. As a result, the standard condition $μ<1/(2k-1)$ (where $μ$ denotes the mutual coherence and $k$ the sparsity le… ▽ More We propose extended coherence-based conditions for exact sparse support recovery using orthogonal matching pursuit (OMP) and orthogonal least squares (OLS). Unlike standard uniform guarantees, we embed some information about the decay of the sparse vector coefficients in our conditions. As a result, the standard condition $μ<1/(2k-1)$ (where $μ$ denotes the mutual coherence and $k$ the sparsity level) can be weakened as soon as the non-zero coefficients obey some decay, both in the noiseless and the bounded-noise scenarios. Furthermore, the resulting condition is approaching $μ<1/k$ for strongly decaying sparse signals. Finally, in the noiseless setting, we prove that the proposed conditions, in particular the bound $μ<1/k$, are the tightest achievable guarantees based on mutual coherence. △ Less

Submitted 8 October, 2015; v1 submitted 29 January, 2014; originally announced January 2014.

Enhanced blind decoding of Tardos codes with new map-based functions

Authors: Mathieu Desoubeaux, Cédric Herzet, William Puech, Gaëtan Le Guelvouit

Abstract: This paper presents a new decoder for probabilistic binary traitor tracing codes under the marking assumption. It is based on a binary hypothesis testing rule which integrates a collusion channel relaxation so as to obtain numerical and simple accusation functions. This decoder is blind as no estimation of the collusion channel prior to the accusation is required. Experimentations show that using… ▽ More This paper presents a new decoder for probabilistic binary traitor tracing codes under the marking assumption. It is based on a binary hypothesis testing rule which integrates a collusion channel relaxation so as to obtain numerical and simple accusation functions. This decoder is blind as no estimation of the collusion channel prior to the accusation is required. Experimentations show that using the proposed decoder gives better performance than the well-known symmetric version of the Tardos decoder for common attack channels. △ Less

Submitted 30 May, 2013; originally announced May 2013.

Exact Recovery Conditions for Sparse Representations with Partial Support Information

Authors: C. Herzet, C. Soussen, J. Idier, R. Gribonval

Abstract: We address the exact recovery of a k-sparse vector in the noiseless setting when some partial information on the support is available. This partial information takes the form of either a subset of the true support or an approximate subset including wrong atoms as well. We derive a new sufficient and worst-case necessary (in some sense) condition for the success of some procedures based on lp-relax… ▽ More We address the exact recovery of a k-sparse vector in the noiseless setting when some partial information on the support is available. This partial information takes the form of either a subset of the true support or an approximate subset including wrong atoms as well. We derive a new sufficient and worst-case necessary (in some sense) condition for the success of some procedures based on lp-relaxation, Orthogonal Matching Pursuit (OMP) and Orthogonal Least Squares (OLS). Our result is based on the coherence "mu" of the dictionary and relaxes the well-known condition mu<1/(2k-1) ensuring the recovery of any k-sparse vector in the non-informed setup. It reads mu<1/(2k-g+b-1) when the informed support is composed of g good atoms and b wrong atoms. We emphasize that our condition is complementary to some restricted-isometry based conditions by showing that none of them implies the other. Because this mutual coherence condition is common to all procedures, we carry out a finer analysis based on the Null Space Property (NSP) and the Exact Recovery Condition (ERC). Connections are established regarding the characterization of lp-relaxation procedures and OMP in the informed setup. First, we emphasize that the truncated NSP enjoys an ordering property when p is decreased. Second, the partial ERC for OMP (ERC-OMP) implies in turn the truncated NSP for the informed l1 problem, and the truncated NSP for p<1. △ Less

Submitted 17 May, 2013; originally announced May 2013.

Comments: arXiv admin note: substantial text overlap with arXiv:1211.7283

Coherence-based Partial Exact Recovery Condition for OMP/OLS

Authors: Cedric Herzet, Charles Soussen, Jerome Idier, Remi Gribonval

Abstract: We address the exact recovery of the support of a k-sparse vector with Orthogonal Matching Pursuit (OMP) and Orthogonal Least Squares (OLS) in a noiseless setting. We consider the scenario where OMP/OLS have selected good atoms during the first l iterations (l<k) and derive a new sufficient and worst-case necessary condition for their success in k steps. Our result is based on the coherence μof th… ▽ More We address the exact recovery of the support of a k-sparse vector with Orthogonal Matching Pursuit (OMP) and Orthogonal Least Squares (OLS) in a noiseless setting. We consider the scenario where OMP/OLS have selected good atoms during the first l iterations (l<k) and derive a new sufficient and worst-case necessary condition for their success in k steps. Our result is based on the coherence μof the dictionary and relaxes Tropp's well-known condition μ<1/(2k-1) to the case where OMP/OLS have a partial knowledge of the support. △ Less

Submitted 30 November, 2012; originally announced November 2012.

Joint k-step analysis of Orthogonal Matching Pursuit and Orthogonal Least Squares

Authors: Charles Soussen, Rémi Gribonval, Jérôme Idier, Cédric Herzet

Abstract: Tropp's analysis of Orthogonal Matching Pursuit (OMP) using the Exact Recovery Condition (ERC) is extended to a first exact recovery analysis of Orthogonal Least Squares (OLS). We show that when the ERC is met, OLS is guaranteed to exactly recover the unknown support in at most k iterations. Moreover, we provide a closer look at the analysis of both OMP and OLS when the ERC is not fulfilled. The e… ▽ More Tropp's analysis of Orthogonal Matching Pursuit (OMP) using the Exact Recovery Condition (ERC) is extended to a first exact recovery analysis of Orthogonal Least Squares (OLS). We show that when the ERC is met, OLS is guaranteed to exactly recover the unknown support in at most k iterations. Moreover, we provide a closer look at the analysis of both OMP and OLS when the ERC is not fulfilled. The existence of dictionaries for which some subsets are never recovered by OMP is proved. This phenomenon also appears with basis pursuit where support recovery depends on the sign patterns, but it does not occur for OLS. Finally, numerical experiments show that none of the considered algorithms is uniformly better than the other but for correlated dictionaries, guaranteed exact recovery may be obtained after fewer iterations for OLS than for OMP. △ Less

Submitted 7 December, 2012; v1 submitted 2 November, 2011; originally announced November 2011.

Comments: 39 pages

Journal ref: IEEE Transactions on Information Theory, vol. 59, no. 5, May 2013, pp. 3158-3174