Showing 1–29 of 29 results for author: Obozinski, G

From STEM-EDXS data to phase separation and quantification using physics-guided NMF

Authors: Adrien Teurtrie, Nathanaël Perraudin, Thomas Holvoet, Hui Chen, Duncan T. L. Alexander, Guillaume Obozinski, Cécile Hébert

Abstract: We present the development of a new algorithm which combines state-of-the-art energy-dispersive X-ray (EDX) spectroscopy theory and a suitable machine learning formulation for the hyperspectral unmixing of scanning transmission electron microscope EDX spectrum images. The algorithm is based on non-negative matrix factorization (NMF) incorporating a physics-guided factorization model. It optimizes… ▽ More We present the development of a new algorithm which combines state-of-the-art energy-dispersive X-ray (EDX) spectroscopy theory and a suitable machine learning formulation for the hyperspectral unmixing of scanning transmission electron microscope EDX spectrum images. The algorithm is based on non-negative matrix factorization (NMF) incorporating a physics-guided factorization model. It optimizes a Poisson likelihood, under additional simplex constraint together with user-chosen sparsity-inducing and smoothing regularizations, and is based on iterative multiplicative updates. The fluorescence of X-rays is fully modeled thanks to state-of-the-art theoretical work. It is shown that the output of the algorithm can be used for a direct chemical quantification. With this approach, it is straightforward to include a priori knowledge on the specimen such as the presence or absence of certain chemical elements in some of its phases. This work is implemented within two open-source Python packages, espm and emtables, which are used here for data simulation, data analysis and quantification. Using simulated data, we demonstrate that incorporating physical modeling in the decomposition helps retrieve meaningful components from spatially and spectrally mixed phases, even when the data are very noisy. For synthetic data with a higher signal, the regularizations yield a tenfold increase in the quality of the reconstructed abundance maps compared to standard NMF. Our approach is further validated on experimental data with a known ground truth, where state-of-the art results are achieved by using prior knowledge about the sample. Our model can be generalized to any other scanning spectroscopy techniques where underlying physical modeling can be linearized. △ Less

Submitted 3 May, 2024; v1 submitted 26 April, 2024; originally announced April 2024.

Comments: 30 pages, 4 figures

Efficient algorithms for regularized Poisson Non-negative Matrix Factorization

Authors: Nathanaël Perraudin, Adrien Teutrie, Cécile Hébert, Guillaume Obozinski

Abstract: We consider the problem of regularized Poisson Non-negative Matrix Factorization (NMF) problem, encompassing various regularization terms such as Lipschitz and relatively smooth functions, alongside linear constraints. This problem holds significant relevance in numerous Machine Learning applications, particularly within the domain of physical linear unmixing problems. A notable challenge arises f… ▽ More We consider the problem of regularized Poisson Non-negative Matrix Factorization (NMF) problem, encompassing various regularization terms such as Lipschitz and relatively smooth functions, alongside linear constraints. This problem holds significant relevance in numerous Machine Learning applications, particularly within the domain of physical linear unmixing problems. A notable challenge arises from the main loss term in the Poisson NMF problem being a KL divergence, which is non-Lipschitz, rendering traditional gradient descent-based approaches inefficient. In this contribution, we explore the utilization of Block Successive Upper Minimization (BSUM) to overcome this challenge. We build approriate majorizing function for Lipschitz and relatively smooth functions, and show how to introduce linear constraints into the problem. This results in the development of two novel algorithms for regularized Poisson NMF. We conduct numerical simulations to showcase the effectiveness of our approach. △ Less

Submitted 25 April, 2024; originally announced April 2024.

Laue Indexing with Optimal Transport

Authors: Tomasz Kacprzak, Stavros Samothrakitis, Camilla Buhl Larsen, Jaromír Kopeček, Markus Strobl, Efthymios Polatidis, Guillaume Obozinski

Abstract: Laue tomography experiments retrieve the positions and orientations of crystal grains in a polycrystalline samples from diffraction patterns recorded at multiple viewing angles. The use of a broad wavelength spectrum beam can greatly reduce the experimental time, but poses a difficult challenge for the indexing of diffraction peaks in polycrystalline samples; the information about the wavelength o… ▽ More Laue tomography experiments retrieve the positions and orientations of crystal grains in a polycrystalline samples from diffraction patterns recorded at multiple viewing angles. The use of a broad wavelength spectrum beam can greatly reduce the experimental time, but poses a difficult challenge for the indexing of diffraction peaks in polycrystalline samples; the information about the wavelength of these Bragg peaks is absent and the diffraction patterns from multiple grains are superimposed. To date, no algorithms exist capable of indexing samples with more than about 500 grains efficiently. To address this need we present a novel method: Laue indexing with Optimal Transport (LaueOT). We create a probabilistic description of the multi-grain indexing problem and propose a solution based on Sinkhorn Expectation-Maximization method, which allows to efficiently find the maximum of the likelihood thanks to the assignments being calculated using Optimal Transport. This is a non-convex optimization problem, where the orientations and positions of grains are optimized simultaneously with grain-to-spot assignments, while robustly handling the outliers. The selection of initial prototype grains to consider in the optimization problem are also calculated within the Optimal Transport framework. LaueOT can rapidly and effectively index up to 1000 grains on a single large memory GPU within less than 30 minutes. We demonstrate the performance of LaueOT on simulations with variable numbers of grains, spot position measurement noise levels, and outlier fractions. The algorithm recovers the correct number of grains even for high noise levels and up to 70% outliers in our experiments. We compare the results of indexing with LaueOT to existing algorithms both on synthetic and real neutron diffraction data from well-characterized samples. △ Less

Submitted 9 April, 2024; originally announced April 2024.

Comments: Data and code available at https://github.com/LaueOT/laueotx. 19 Pages, 8 figures

An evaluation of deep learning models for predicting water depth evolution in urban floods

Authors: Stefania Russo, Nathanaël Perraudin, Steven Stalder, Fernando Perez-Cruz, Joao Paulo Leitao, Guillaume Obozinski, Jan Dirk Wegner

Abstract: In this technical report we compare different deep learning models for prediction of water depth rasters at high spatial resolution. Efficient, accurate, and fast methods for water depth prediction are nowadays important as urban floods are increasing due to higher rainfall intensity caused by climate change, expansion of cities and changes in land use. While hydrodynamic models models can provide… ▽ More In this technical report we compare different deep learning models for prediction of water depth rasters at high spatial resolution. Efficient, accurate, and fast methods for water depth prediction are nowadays important as urban floods are increasing due to higher rainfall intensity caused by climate change, expansion of cities and changes in land use. While hydrodynamic models models can provide reliable forecasts by simulating water depth at every location of a catchment, they also have a high computational burden which jeopardizes their application to real-time prediction in large urban areas at high spatial resolution. Here, we propose to address this issue by using data-driven techniques. Specifically, we evaluate deep learning models which are trained to reproduce the data simulated by the CADDIES cellular-automata flood model, providing flood forecasts that can occur at different future time horizons. The advantage of using such models is that they can learn the underlying physical phenomena a priori, preventing manual parameter setting and computational burden. We perform experiments on a dataset consisting of two catchments areas within Switzerland with 18 simpler, short rainfall patterns and 4 long, more complex ones. Our results show that the deep learning models present in general lower errors compared to the other methods, especially for water depths $>0.5m$. However, when testing on more complex rainfall events or unseen catchment areas, the deep models do not show benefits over the simpler ones. △ Less

Submitted 20 February, 2023; originally announced February 2023.

Optirank: classification for RNA-Seq data with optimal ranking reference genes

Authors: Paola Malsot, Filipe Martins, Didier Trono, Guillaume Obozinski

Abstract: Classification algorithms using RNA-Sequencing (RNA-Seq) data as input are used in a variety of biological applications. By nature, RNA-Seq data is subject to uncontrolled fluctuations both within and especially across datasets, which presents a major difficulty for a trained classifier to generalize to an external dataset. Replacing raw gene counts with the rank of gene counts inside an observati… ▽ More Classification algorithms using RNA-Sequencing (RNA-Seq) data as input are used in a variety of biological applications. By nature, RNA-Seq data is subject to uncontrolled fluctuations both within and especially across datasets, which presents a major difficulty for a trained classifier to generalize to an external dataset. Replacing raw gene counts with the rank of gene counts inside an observation has proven effective to mitigate this problem. However, the rank of a feature is by definition relative to all other features, including highly variable features that introduce noise in the ranking. To address this problem and obtain more robust ranks, we propose a logistic regression model, optirank, which learns simultaneously the parameters of the model and the genes to use as a reference set in the ranking. We show the effectiveness of this method on simulated data. We also consider real classification tasks, which present different kinds of distribution shifts between train and test data. Those tasks concern a variety of applications, such as cancer of unknown primary classification, identification of specific gene signatures, and determination of cell type in single-cell RNA-Seq datasets. On those real tasks, optirank performs at least as well as the vanilla logistic regression on classical ranks, while producing sparser solutions. In addition, to increase the robustness against dataset shifts, we propose a multi-source learning scheme and demonstrate its effectiveness when used in combination with rank-based classifiers. △ Less

Submitted 11 January, 2023; originally announced January 2023.

Robust detection and attribution of climate change under interventions

Authors: Enikő Székely, Sebastian Sippel, Nicolai Meinshausen, Guillaume Obozinski, Reto Knutti

Abstract: Fingerprints are key tools in climate change detection and attribution (D&A) that are used to determine whether changes in observations are different from internal climate variability (detection), and whether observed changes can be assigned to specific external drivers (attribution). We propose a direct D&A approach based on supervised learning to extract fingerprints that lead to robust predicti… ▽ More Fingerprints are key tools in climate change detection and attribution (D&A) that are used to determine whether changes in observations are different from internal climate variability (detection), and whether observed changes can be assigned to specific external drivers (attribution). We propose a direct D&A approach based on supervised learning to extract fingerprints that lead to robust predictions under relevant interventions on exogenous variables, i.e., climate drivers other than the target. We employ anchor regression, a distributionally-robust statistical learning method inspired by causal inference that extrapolates well to perturbed data under the interventions considered. The residuals from the prediction achieve either uncorrelatedness or mean independence with the exogenous variables, thus guaranteeing robustness. We define D&A as a unified hypothesis testing framework that relies on the same statistical model but uses different targets and test statistics. In the experiments, we first show that the CO2 forcing can be robustly predicted from temperature spatial patterns under strong interventions on the solar forcing. Second, we illustrate attribution to the greenhouse gases and aerosols while protecting against interventions on the aerosols and CO2 forcing, respectively. Our study shows that incorporating robustness constraints against relevant interventions may significantly benefit detection and attribution of climate change. △ Less

Submitted 9 December, 2022; originally announced December 2022.

Data-driven modeling of beam loss in the LHC

Authors: Ekaterina Krymova, Guillaume Obozinski, Michael Schenk, Loic Coyle, Tatiana Pieloni

Abstract: In the Large Hadron Collider, the beam losses are continuously measured for machine protection. By design, most of the particle losses occur in the collimation system, where the particles with high oscillation amplitudes or large momentum error are scraped from the beams. The level of particle losses typically is optimized manually by changing multiple control parameters, among which are, for exam… ▽ More In the Large Hadron Collider, the beam losses are continuously measured for machine protection. By design, most of the particle losses occur in the collimation system, where the particles with high oscillation amplitudes or large momentum error are scraped from the beams. The level of particle losses typically is optimized manually by changing multiple control parameters, among which are, for example, currents in the focusing and defocusing magnets along the collider. It is generally challenging to model and predict losses based on the control parameters due to various (non-linear) effects in the system, such as electron clouds, resonance effects, etc, and multiple sources of uncertainty. At the same time understanding the influence of control parameters on the losses is extremely important in order to improve the operation and performance, and future design of accelerators. Existing results showed that it is hard to generalize the models, which assume the regression model of losses depending on control parameters, from fills carried out throughout one year to the data of another year. To circumvent this, we propose to use an autoregressive modeling approach, where we take into account not only the observed control parameters but also previous loss values. We use an equivalent Kalman Filter (KF) formulation in order to efficiently estimate models with different lags. △ Less

Submitted 18 August, 2022; originally announced August 2022.

MSC Class: 62P35; 37M10

Trend estimation and short-term forecasting of COVID-19 cases and deaths worldwide

Authors: Ekaterina Krymova, Benjamín Béjar, Dorina Thanou, Tao Sun, Elisa Manetti, Gavin Lee, Kristen Namigai, Christine Choirat, Antoine Flahault, Guillaume Obozinski

Abstract: Since the beginning of the COVID-19 pandemic, many dashboards have emerged as useful tools to monitor the evolution of the pandemic, inform the public, and assist governments in decision making. Our goal is to develop a globally applicable method, integrated in a twice daily updated dashboard that provides an estimate of the trend in the evolution of the number of cases and deaths from reported da… ▽ More Since the beginning of the COVID-19 pandemic, many dashboards have emerged as useful tools to monitor the evolution of the pandemic, inform the public, and assist governments in decision making. Our goal is to develop a globally applicable method, integrated in a twice daily updated dashboard that provides an estimate of the trend in the evolution of the number of cases and deaths from reported data of more than 200 countries and territories, as well as a seven-day forecast. One of the significant difficulties to manage a quickly propagating epidemic is that the details of the dynamic needed to forecast its evolution are obscured by the delays in the identification of cases and deaths and by irregular reporting. Our forecasting methodology substantially relies on estimating the underlying trend in the observed time series using robust seasonal trend decomposition techniques. This allows us to obtain forecasts with simple, yet effective extrapolation methods in linear or log scale. We present the results of an assessment of our forecasting methodology and discuss its application to the production of global and regional risk maps. △ Less

Submitted 24 March, 2022; v1 submitted 18 June, 2021; originally announced June 2021.

MSC Class: 92D30; 00-01; 60G25

Empirical Bayes Transductive Meta-Learning with Synthetic Gradients

Authors: Shell Xu Hu, Pablo G. Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D. Lawrence, Andreas Damianou

Abstract: We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging the unlabeled query set in addition to the support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a su… ▽ More We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging the unlabeled query set in addition to the support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior on the query set of each task. We derive a novel amortized variational inference that couples all the variational posteriors via a meta-model, which consists of a synthetic gradient network and an initialization network. Each variational posterior is derived from synthetic gradient descent to approximate the true posterior on the query set, although where we do not have access to the true gradient. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods. Besides, we conduct two zero-shot learning experiments to further explore the potential of the synthetic gradient. △ Less

Submitted 27 April, 2020; originally announced April 2020.

Comments: ICLR 2020

Tensor Decompositions for temporal knowledge base completion

Authors: Timothée Lacroix, Guillaume Obozinski, Nicolas Usunier

Abstract: Most algorithms for representation learning and link prediction in relational data have been designed for static data. However, the data they are applied to usually evolves with time, such as friend graphs in social networks or user interactions with items in recommender systems. This is also the case for knowledge bases, which contain facts such as (US, has president, B. Obama, [2009-2017]) that… ▽ More Most algorithms for representation learning and link prediction in relational data have been designed for static data. However, the data they are applied to usually evolves with time, such as friend graphs in social networks or user interactions with items in recommender systems. This is also the case for knowledge bases, which contain facts such as (US, has president, B. Obama, [2009-2017]) that are valid only at certain points in time. For the problem of link prediction under temporal constraints, i.e., answering queries such as (US, has president, ?, 2012), we propose a solution inspired by the canonical decomposition of tensors of order 4. We introduce new regularization schemes and present an extension of ComplEx (Trouillon et al., 2016) that achieves state-of-the-art performance. Additionally, we propose a new dataset for knowledge base completion constructed from Wikidata, larger than previous benchmarks by an order of magnitude, as a new reference for evaluating temporal and non-temporal link prediction methods. △ Less

Submitted 10 April, 2020; originally announced April 2020.

A direct approach to detection and attribution of climate change

Authors: Eniko Székely, Sebastian Sippel, Reto Knutti, Guillaume Obozinski, Nicolai Meinshausen

Abstract: We present here a novel statistical learning approach for detection and attribution (D&A) of climate change. Traditional optimal D&A studies try to directly model the observations from model simulations, but practically this is challenging due to high-dimensionality. Dimension reduction techniques reduce the dimensionality, typically using empirical orthogonal functions, but as these techniques ar… ▽ More We present here a novel statistical learning approach for detection and attribution (D&A) of climate change. Traditional optimal D&A studies try to directly model the observations from model simulations, but practically this is challenging due to high-dimensionality. Dimension reduction techniques reduce the dimensionality, typically using empirical orthogonal functions, but as these techniques are unsupervised, the reduced space considered is somewhat arbitrary. Here, we propose a supervised approach where we predict a given external forcing, e.g., anthropogenic forcing, directly from the spatial pattern of climate variables, and use the predicted forcing as a test statistic for D&A. We want the prediction to work well even under changes in the distribution of other external forcings, e.g., solar or volcanic forcings, and therefore formulate the optimization problem from a distributional robustness perspective. △ Less

Submitted 8 October, 2019; originally announced October 2019.

Integer programming on the junction tree polytope for influence diagrams

Authors: Axel Parmentier, Victor Cohen, Vincent Leclère, Guillaume Obozinski, Joseph Salmon

Abstract: Influence Diagrams (ID) are a flexible tool to represent discrete stochastic optimization problems, including Markov Decision Process (MDP) and Partially Observable MDP as standard examples. More precisely, given random variables considered as vertices of an acyclic digraph, a probabilistic graphical model defines a joint distribution via the conditional distributions of vertices given their paren… ▽ More Influence Diagrams (ID) are a flexible tool to represent discrete stochastic optimization problems, including Markov Decision Process (MDP) and Partially Observable MDP as standard examples. More precisely, given random variables considered as vertices of an acyclic digraph, a probabilistic graphical model defines a joint distribution via the conditional distributions of vertices given their parents. In ID, the random variables are represented by a probabilistic graphical model whose vertices are partitioned into three types : chance, decision and utility vertices. The user chooses the distribution of the decision vertices conditionally to their parents in order to maximize the expected utility. Leveraging the notion of rooted junction tree, we present a mixed integer linear formulation for solving an ID, as well as valid inequalities, which lead to a computationally efficient algorithm. We also show that the linear relaxation yields an optimal integer solution for instances that can be solved by the "single policy update", the default algorithm for addressing IDs. △ Less

Submitted 5 July, 2019; v1 submitted 19 February, 2019; originally announced February 2019.

Comments: 36 pages

Learning the effect of latent variables in Gaussian Graphical models with unobserved variables

Authors: Marina Vinyes, Guillaume Obozinski

Abstract: The edge structure of the graph defining an undirected graphical model describes precisely the structure of dependence between the variables in the graph. In many applications, the dependence structure is unknown and it is desirable to learn it from data, often because it is a preliminary step to be able to ascertain causal effects. This problem, known as structure learning, is hard in general, bu… ▽ More The edge structure of the graph defining an undirected graphical model describes precisely the structure of dependence between the variables in the graph. In many applications, the dependence structure is unknown and it is desirable to learn it from data, often because it is a preliminary step to be able to ascertain causal effects. This problem, known as structure learning, is hard in general, but for Gaussian graphical models it is slightly easier because the structure of the graph is given by the sparsity pattern of the precision matrix of the joint distribution, and because independence coincides with decorrelation. A major difficulty too often ignored in structure learning is the fact that if some variables are not observed, the marginal dependence graph over the observed variables will possibly be significantly more complex and no longer reflect the direct dependencies that are potentially associated with causal effects. In this work, we consider a family of latent variable Gaussian graphical models in which the graph of the joint distribution between observed and unobserved variables is sparse, and the unobserved variables are conditionally independent given the others. Prior work was able to recover the connectivity between observed variables, but could only identify the subspace spanned by unobserved variables, whereas we propose a convex optimization formulation based on structured matrix sparsity to estimate the complete connectivity of the complete graph including unobserved variables, given the knowledge of the number of missing variables, and a priori knowledge of their level of connectivity. Our formulation is supported by a theoretical result of identifiability of the latent dependence structure for sparse graphs in the infinite data limit. We propose an algorithm leveraging recent active set methods, which performs well in the experiments on synthetic data. △ Less

Submitted 31 July, 2018; v1 submitted 20 July, 2018; originally announced July 2018.

Canonical Tensor Decomposition for Knowledge Base Completion

Authors: Timothée Lacroix, Nicolas Usunier, Guillaume Obozinski

Abstract: The problem of Knowledge Base Completion can be framed as a 3rd-order binary tensor completion problem. In this light, the Canonical Tensor Decomposition (CP) (Hitchcock, 1927) seems like a natural solution; however, current implementations of CP on standard Knowledge Base Completion benchmarks are lagging behind their competitors. In this work, we attempt to understand the limits of CP for knowle… ▽ More The problem of Knowledge Base Completion can be framed as a 3rd-order binary tensor completion problem. In this light, the Canonical Tensor Decomposition (CP) (Hitchcock, 1927) seems like a natural solution; however, current implementations of CP on standard Knowledge Base Completion benchmarks are lagging behind their competitors. In this work, we attempt to understand the limits of CP for knowledge base completion. First, we motivate and test a novel regularizer, based on tensor nuclear $p$-norms. Then, we present a reformulation of the problem that makes it invariant to arbitrary choices in the inclusion of predicates or their reciprocals in the dataset. These two methods combined allow us to beat the current state of the art on several datasets with a CP decomposition, and obtain even better results using the more advanced ComplEx model. △ Less

Submitted 19 June, 2018; originally announced June 2018.

Tight convex relaxations for sparse matrix factorization

Authors: Emile Richard, Guillaume Obozinski, Jean-Philippe Vert

Abstract: Based on a new atomic norm, we propose a new convex formulation for sparse matrix factorization problems in which the number of nonzero elements of the factors is assumed fixed and known. The formulation counts sparse PCA with multiple factors, subspace clustering and low-rank sparse bilinear regression as potential applications. We compute slow rates and an upper bound on the statistical dimensio… ▽ More Based on a new atomic norm, we propose a new convex formulation for sparse matrix factorization problems in which the number of nonzero elements of the factors is assumed fixed and known. The formulation counts sparse PCA with multiple factors, subspace clustering and low-rank sparse bilinear regression as potential applications. We compute slow rates and an upper bound on the statistical dimension of the suggested norm for rank 1 matrices, showing that its statistical dimension is an order of magnitude smaller than the usual $\ell\_1$-norm, trace norm and their combinations. Even though our convex formulation is in theory hard and does not lead to provably polynomial time algorithmic schemes, we propose an active set algorithm leveraging the structure of the convex problem to solve it and show promising numerical results. △ Less

Submitted 4 December, 2014; v1 submitted 19 July, 2014; originally announced July 2014.

Domain adaptation for sequence labeling using hidden Markov models

Authors: Edouard Grave, Guillaume Obozinski, Francis Bach

Abstract: Most natural language processing systems based on machine learning are not robust to domain shift. For example, a state-of-the-art syntactic dependency parser trained on Wall Street Journal sentences has an absolute drop in performance of more than ten points when tested on textual data from the Web. An efficient solution to make these methods more robust to domain shift is to first learn a word r… ▽ More Most natural language processing systems based on machine learning are not robust to domain shift. For example, a state-of-the-art syntactic dependency parser trained on Wall Street Journal sentences has an absolute drop in performance of more than ten points when tested on textual data from the Web. An efficient solution to make these methods more robust to domain shift is to first learn a word representation using large amounts of unlabeled data from both domains, and then use this representation as features in a supervised learning algorithm. In this paper, we propose to use hidden Markov models to learn word representations for part-of-speech tagging. In particular, we study the influence of using data from the source, the target or both domains to learn the representation and the different ways to represent words using an HMM. △ Less

Submitted 14 December, 2013; originally announced December 2013.

Comments: New Directions in Transfer and Multi-Task: Learning Across Domains and Tasks (NIPS Workshop) (2013)

Convex Relaxation for Combinatorial Penalties

Authors: Guillaume Obozinski, Francis Bach

Abstract: In this paper, we propose an unifying view of several recently proposed structured sparsity-inducing norms. We consider the situation of a model simultaneously (a) penalized by a set- function de ned on the support of the unknown parameter vector which represents prior knowledge on supports, and (b) regularized in Lp-norm. We show that the natural combinatorial optimization problems obtained may b… ▽ More In this paper, we propose an unifying view of several recently proposed structured sparsity-inducing norms. We consider the situation of a model simultaneously (a) penalized by a set- function de ned on the support of the unknown parameter vector which represents prior knowledge on supports, and (b) regularized in Lp-norm. We show that the natural combinatorial optimization problems obtained may be relaxed into convex optimization problems and introduce a notion, the lower combinatorial envelope of a set-function, that characterizes the tightness of our relaxations. We moreover establish links with norms based on latent representations including the latent group Lasso and block-coding, and with norms obtained from submodular functions. △ Less

Submitted 6 May, 2012; originally announced May 2012.

Comments: 35 page

On the Equivalence between Herding and Conditional Gradient Algorithms

Authors: Francis Bach, Simon Lacoste-Julien, Guillaume Obozinski

Abstract: We show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm--namely a conditional gradient algorithm minimizing a quadratic moment discrepancy. This link enables us to invoke convergence results from convex optimization and to consider faster alternatives for the task of approximating integrals in a reproducing kernel Hilbert space. We st… ▽ More We show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm--namely a conditional gradient algorithm minimizing a quadratic moment discrepancy. This link enables us to invoke convergence results from convex optimization and to consider faster alternatives for the task of approximating integrals in a reproducing kernel Hilbert space. We study the behavior of the different variants through numerical simulations. The experiments indicate that while we can improve over herding on the task of approximating integrals, the original herding algorithm tends to approach more often the maximum entropy distribution, shedding more light on the learning bias behind herding. △ Less

Submitted 11 September, 2012; v1 submitted 20 March, 2012; originally announced March 2012.

Journal ref: ICML 2012 International Conference on Machine Learning, Edimburgh : Royaume-Uni (2012)

Learning Hierarchical and Topographic Dictionaries with Structured Sparsity

Authors: Julien Mairal, Rodolphe Jenatton, Guillaume Obozinski, Francis Bach

Abstract: Recent work in signal processing and statistics have focused on defining new regularization functions, which not only induce sparsity of the solution, but also take into account the structure of the problem. We present in this paper a class of convex penalties introduced in the machine learning community, which take the form of a sum of l_2 and l_infinity-norms over groups of variables. They exten… ▽ More Recent work in signal processing and statistics have focused on defining new regularization functions, which not only induce sparsity of the solution, but also take into account the structure of the problem. We present in this paper a class of convex penalties introduced in the machine learning community, which take the form of a sum of l_2 and l_infinity-norms over groups of variables. They extend the classical group-sparsity regularization in the sense that the groups possibly overlap, allowing more flexibility in the group design. We review efficient optimization methods to deal with the corresponding inverse problems, and their application to the problem of learning dictionaries of natural image patches: On the one hand, dictionary learning has indeed proven effective for various signal processing tasks. On the other hand, structured sparsity provides a natural framework for modeling dependencies between dictionary elements. We thus consider a structured sparse regularization to learn dictionaries embedded in a particular structure, for instance a tree or a two-dimensional grid. In the latter case, the results we obtain are similar to the dictionaries produced by topographic independent component analysis. △ Less

Submitted 20 October, 2011; originally announced October 2011.

Journal ref: SPIE Wavelets and Sparsity XIV 81381P (2011)

Group Lasso with Overlaps: the Latent Group Lasso approach

Authors: Guillaume Obozinski, Laurent Jacob, Jean-Philippe Vert

Abstract: We study a norm for structured sparsity which leads to sparse linear predictors whose supports are unions of prede ned overlap** groups of variables. We call the obtained formulation latent group Lasso, since it is based on applying the usual group Lasso penalty on a set of latent variables. A detailed analysis of the norm and its properties is presented and we characterize conditions under whic… ▽ More We study a norm for structured sparsity which leads to sparse linear predictors whose supports are unions of prede ned overlap** groups of variables. We call the obtained formulation latent group Lasso, since it is based on applying the usual group Lasso penalty on a set of latent variables. A detailed analysis of the norm and its properties is presented and we characterize conditions under which the set of groups associated with latent variables are correctly identi ed. We motivate and discuss the delicate choice of weights associated to each group, and illustrate this approach on simulated data and on the problem of breast cancer prognosis from gene expression data. △ Less

Submitted 3 October, 2011; originally announced October 2011.

Structured sparsity through convex optimization

Authors: Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski

Abstract: Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. While naturally cast as a combinatorial optimization problem, variable or feature selection admits a convex relaxation through the regularization by the $\ell_1$-norm. In this paper, we consider situations where we are not only interested in sparsity, but where some structural prior knowledge… ▽ More Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. While naturally cast as a combinatorial optimization problem, variable or feature selection admits a convex relaxation through the regularization by the $\ell_1$-norm. In this paper, we consider situations where we are not only interested in sparsity, but where some structural prior knowledge is available as well. We show that the $\ell_1$-norm can then be extended to structured norms built on either disjoint or overlap** groups of variables, leading to a flexible framework that can deal with various structures. We present applications to unsupervised learning, for structured sparse principal component analysis and hierarchical dictionary learning, and to supervised learning in the context of non-linear variable selection. △ Less

Submitted 20 April, 2012; v1 submitted 12 September, 2011; originally announced September 2011.

Comments: Statistical Science (2012) To appear

Trace Lasso: a trace norm regularization for correlated designs

Authors: Edouard Grave, Guillaume Obozinski, Francis Bach

Abstract: Using the $\ell_1$-norm to regularize the estimation of the parameter vector of a linear model leads to an unstable estimator when covariates are highly correlated. In this paper, we introduce a new penalty function which takes into account the correlation of the design matrix to stabilize the estimation. This norm, called the trace Lasso, uses the trace norm, which is a convex surrogate of the ra… ▽ More Using the $\ell_1$-norm to regularize the estimation of the parameter vector of a linear model leads to an unstable estimator when covariates are highly correlated. In this paper, we introduce a new penalty function which takes into account the correlation of the design matrix to stabilize the estimation. This norm, called the trace Lasso, uses the trace norm, which is a convex surrogate of the rank, of the selected covariates as the criterion of model complexity. We analyze the properties of our norm, describe an optimization algorithm based on reweighted least-squares, and illustrate the behavior of this norm on synthetic data, showing that it is more adapted to strong correlations than competing methods such as the elastic net. △ Less

Submitted 9 September, 2011; originally announced September 2011.

Journal ref: Neural Information Processing Systems, Spain (2012)

Optimization with Sparsity-Inducing Penalties

Authors: Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski

Abstract: Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast as convex optimization problems by regularizing the empirical risk with appropr… ▽ More Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast as convex optimization problems by regularizing the empirical risk with appropriate non-smooth norms. The goal of this paper is to present from a general perspective optimization tools and techniques dedicated to such sparsity-inducing penalties. We cover proximal methods, block-coordinate descent, reweighted $\ell_2$-penalized techniques, working-set and homotopy methods, as well as non-convex formulations and extensions, and provide an extensive set of experiments to compare various algorithms from a computational point of view. △ Less

Submitted 22 November, 2011; v1 submitted 3 August, 2011; originally announced August 2011.

Multi-scale Mining of fMRI data with Hierarchical Structured Sparsity

Authors: Rodolphe Jenatton, Alexandre Gramfort, Vincent Michel, Guillaume Obozinski, Evelyn Eger, Francis Bach, Bertrand Thirion

Abstract: Inverse inference, or "brain reading", is a recent paradigm for analyzing functional magnetic resonance imaging (fMRI) data, based on pattern recognition and statistical learning. By predicting some cognitive variables related to brain activation maps, this approach aims at decoding brain activity. Inverse inference takes into account the multivariate information between voxels and is currently th… ▽ More Inverse inference, or "brain reading", is a recent paradigm for analyzing functional magnetic resonance imaging (fMRI) data, based on pattern recognition and statistical learning. By predicting some cognitive variables related to brain activation maps, this approach aims at decoding brain activity. Inverse inference takes into account the multivariate information between voxels and is currently the only way to assess how precisely some cognitive information is encoded by the activity of neural populations within the whole brain. However, it relies on a prediction function that is plagued by the curse of dimensionality, since there are far more features than samples, i.e., more voxels than fMRI volumes. To address this problem, different methods have been proposed, such as, among others, univariate feature selection, feature agglomeration and regularization techniques. In this paper, we consider a sparse hierarchical structured regularization. Specifically, the penalization we use is constructed from a tree that is obtained by spatially-constrained agglomerative clustering. This approach encodes the spatial structure of the data at different scales into the regularization, which makes the overall prediction procedure more robust to inter-subject variability. The regularization used induces the selection of spatially coherent predictive brain regions simultaneously at different scales. We test our algorithm on real data acquired to study the mental representation of objects, and we show that the proposed algorithm not only delineates meaningful brain regions but yields as well better prediction accuracy than reference methods. △ Less

Submitted 8 December, 2011; v1 submitted 2 May, 2011; originally announced May 2011.

Comments: (2011)

Convex and Network Flow Optimization for Structured Sparsity

Authors: Julien Mairal, Rodolphe Jenatton, Guillaume Obozinski, Francis Bach

Abstract: We consider a class of learning problems regularized by a structured sparsity-inducing norm defined as the sum of l_2- or l_infinity-norms over groups of variables. Whereas much effort has been put in develo** fast optimization techniques when the groups are disjoint or embedded in a hierarchy, we address here the case of general overlap** groups. To this end, we present two different strategi… ▽ More We consider a class of learning problems regularized by a structured sparsity-inducing norm defined as the sum of l_2- or l_infinity-norms over groups of variables. Whereas much effort has been put in develo** fast optimization techniques when the groups are disjoint or embedded in a hierarchy, we address here the case of general overlap** groups. To this end, we present two different strategies: On the one hand, we show that the proximal operator associated with a sum of l_infinity-norms can be computed exactly in polynomial time by solving a quadratic min-cost flow problem, allowing the use of accelerated proximal gradient methods. On the other hand, we use proximal splitting techniques, and address an equivalent formulation with non-overlap** groups, but in higher dimension and with additional constraints. We propose efficient and scalable algorithms exploiting these two strategies, which are significantly faster than alternative approaches. We illustrate these methods with several problems such as CUR matrix factorization, multi-task learning of tree-structured dictionaries, background subtraction in video sequences, image denoising with wavelets, and topographic dictionary learning of natural image patches. △ Less

Submitted 16 September, 2011; v1 submitted 11 April, 2011; originally announced April 2011.

Comments: to appear in the Journal of Machine Learning Research (JMLR)

Journal ref: Journal of Machine Learning Research 12 (2011) 2681?2720

Proximal Methods for Hierarchical Sparse Coding

Authors: Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, Francis Bach

Abstract: Sparse coding consists in representing signals as sparse linear combinations of atoms selected from a dictionary. We consider an extension of this framework where the atoms are further assumed to be embedded in a tree. This is achieved using a recently introduced tree-structured sparse regularization norm, which has proven useful in several applications. This norm leads to regularized problems tha… ▽ More Sparse coding consists in representing signals as sparse linear combinations of atoms selected from a dictionary. We consider an extension of this framework where the atoms are further assumed to be embedded in a tree. This is achieved using a recently introduced tree-structured sparse regularization norm, which has proven useful in several applications. This norm leads to regularized problems that are difficult to optimize, and we propose in this paper efficient algorithms for solving them. More precisely, we show that the proximal operator associated with this norm is computable exactly via a dual approach that can be viewed as the composition of elementary proximal operators. Our procedure has a complexity linear, or close to linear, in the number of atoms, and allows the use of accelerated gradient techniques to solve the tree-structured sparse approximation problem at the same computational cost as traditional ones using the L1-norm. Our method is efficient and scales gracefully to millions of variables, which we illustrate in two types of applications: first, we consider fixed hierarchical dictionaries of wavelets to denoise natural images. Then, we apply our optimization tools in the context of dictionary learning, where learned dictionary elements naturally organize in a prespecified arborescent structure, leading to a better performance in reconstruction of natural image patches. When applied to text documents, our method learns hierarchies of topics, thus providing a competitive alternative to probabilistic topic models. △ Less

Submitted 5 July, 2011; v1 submitted 11 September, 2010; originally announced September 2010.

Journal ref: Journal of Machine Learning Research, 12 (2011) 2297-2334

Network Flow Algorithms for Structured Sparsity

Authors: Julien Mairal, Rodolphe Jenatton, Guillaume Obozinski, Francis Bach

Abstract: We consider a class of learning problems that involve a structured sparsity-inducing norm defined as the sum of $\ell_\infty$-norms over groups of variables. Whereas a lot of effort has been put in develo** fast optimization methods when the groups are disjoint or embedded in a specific hierarchical structure, we address here the case of general overlap** groups. To this end, we show that the… ▽ More We consider a class of learning problems that involve a structured sparsity-inducing norm defined as the sum of $\ell_\infty$-norms over groups of variables. Whereas a lot of effort has been put in develo** fast optimization methods when the groups are disjoint or embedded in a specific hierarchical structure, we address here the case of general overlap** groups. To this end, we show that the corresponding optimization problem is related to network flow optimization. More precisely, the proximal problem associated with the norm we consider is dual to a quadratic min-cost flow problem. We propose an efficient procedure which computes its solution exactly in polynomial time. Our algorithm scales up to millions of variables, and opens up a whole new range of applications for structured sparse models. We present several experiments on image and video data, demonstrating the applicability and scalability of our approach for various problems. △ Less

Submitted 30 August, 2010; originally announced August 2010.

Comments: accepted for publication in Adv. Neural Information Processing Systems, 2010

Report number: RR-7372

Structured Sparse Principal Component Analysis

Authors: Rodolphe Jenatton, Guillaume Obozinski, Francis Bach

Abstract: We present an extension of sparse PCA, or sparse dictionary learning, where the sparsity patterns of all dictionary elements are structured and constrained to belong to a prespecified set of shapes. This \emph{structured sparse PCA} is based on a structured regularization recently introduced by [1]. While classical sparse priors only deal with \textit{cardinality}, the regularization we use enco… ▽ More We present an extension of sparse PCA, or sparse dictionary learning, where the sparsity patterns of all dictionary elements are structured and constrained to belong to a prespecified set of shapes. This \emph{structured sparse PCA} is based on a structured regularization recently introduced by [1]. While classical sparse priors only deal with \textit{cardinality}, the regularization we use encodes higher-order information about the data. We propose an efficient and simple optimization procedure to solve this problem. Experiments with two practical tasks, face recognition and the study of the dynamics of a protein complex, demonstrate the benefits of the proposed structured approach over unstructured approaches. △ Less

Submitted 8 September, 2009; originally announced September 2009.

Support union recovery in high-dimensional multivariate regression

Authors: Guillaume Obozinski, Martin J. Wainwright, Michael I. Jordan

Abstract: In multivariate regression, a $K$-dimensional response vector is regressed upon a common set of $p$ covariates, with a matrix $B^*\in\mathbb{R}^{p\times K}$ of regression coefficients. We study the behavior of the multivariate group Lasso, in which block regularization based on the $\ell_1/\ell_2$ norm is used for support union recovery, or recovery of the set of $s$ rows for which $B^*$ is nonzer… ▽ More In multivariate regression, a $K$-dimensional response vector is regressed upon a common set of $p$ covariates, with a matrix $B^*\in\mathbb{R}^{p\times K}$ of regression coefficients. We study the behavior of the multivariate group Lasso, in which block regularization based on the $\ell_1/\ell_2$ norm is used for support union recovery, or recovery of the set of $s$ rows for which $B^*$ is nonzero. Under high-dimensional scaling, we show that the multivariate group Lasso exhibits a threshold for the recovery of the exact row pattern with high probability over the random design and noise that is specified by the sample complexity parameter $θ(n,p,s):=n/[2ψ(B^*)\log(p-s)]$. Here $n$ is the sample size, and $ψ(B^*)$ is a sparsity-overlap function measuring a combination of the sparsities and overlaps of the $K$-regression coefficient vectors that constitute the model. We prove that the multivariate group Lasso succeeds for problem sequences $(n,p,s)$ such that $θ(n,p,s)$ exceeds a critical level $θ_u$, and fails for sequences such that $θ(n,p,s)$ lies below a critical level $θ_{\ell}$. For the special case of the standard Gaussian ensemble, we show that $θ_{\ell}=θ_u$ so that the characterization is sharp. The sparsity-overlap function $ψ(B^*)$ reveals that, if the design is uncorrelated on the active rows, $\ell_1/\ell_2$ regularization for multivariate regression never harms performance relative to an ordinary Lasso approach and can yield substantial improvements in sample complexity (up to a factor of $K$) when the coefficient vectors are suitably orthogonal. For more general designs, it is possible for the ordinary Lasso to outperform the multivariate group Lasso. We complement our analysis with simulations that demonstrate the sharpness of our theoretical results, even for relatively small problems. △ Less

Submitted 7 March, 2011; v1 submitted 5 August, 2008; originally announced August 2008.

Comments: Published in at http://dx.doi.org/10.1214/09-AOS776 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS776

Journal ref: Annals of Statistics 2011, Vol. 39, No. 1, 1-47