Showing 1–28 of 28 results for author: Aghasi, A

Laplace-HDC: Understanding the geometry of binary hyperdimensional computing

Authors: Saeid Pourmand, Wyatt D. Whiting, Alireza Aghasi, Nicholas F. Marshall

Abstract: This paper studies the geometry of binary hyperdimensional computing (HDC), a computational scheme in which data are encoded using high-dimensional binary vectors. We establish a result about the similarity structure induced by the HDC binding operator and show that the Laplace kernel naturally arises in this setting, motivating our new encoding method Laplace-HDC, which improves upon previous met… ▽ More This paper studies the geometry of binary hyperdimensional computing (HDC), a computational scheme in which data are encoded using high-dimensional binary vectors. We establish a result about the similarity structure induced by the HDC binding operator and show that the Laplace kernel naturally arises in this setting, motivating our new encoding method Laplace-HDC, which improves upon previous methods. We describe how our results indicate limitations of binary HDC in encoding spatial information from images and discuss potential solutions, including using Haar convolutional features and the definition of a translation-equivariant HDC encoding. Several numerical experiments highlighting the improved accuracy of Laplace-HDC in contrast to alternative methods are presented. We also numerically study other aspects of the proposed framework such as robustness and the underlying translation-equivariant encoding. △ Less

Submitted 26 April, 2024; v1 submitted 16 April, 2024; originally announced April 2024.

Comments: 23 pages, 7 figures

Fully Zeroth-Order Bilevel Programming via Gaussian Smoothing

Authors: Alireza Aghasi, Saeed Ghadimi

Abstract: In this paper, we study and analyze zeroth-order stochastic approximation algorithms for solving bilvel problems, when neither the upper/lower objective values, nor their unbiased gradient estimates are available. In particular, exploiting Stein's identity, we first use Gaussian smoothing to estimate first- and second-order partial derivatives of functions with two independent block of variables.… ▽ More In this paper, we study and analyze zeroth-order stochastic approximation algorithms for solving bilvel problems, when neither the upper/lower objective values, nor their unbiased gradient estimates are available. In particular, exploiting Stein's identity, we first use Gaussian smoothing to estimate first- and second-order partial derivatives of functions with two independent block of variables. We then used these estimates in the framework of a stochastic approximation algorithm for solving bilevel optimization problems and establish its non-asymptotic convergence analysis. To the best of our knowledge, this is the first time that sample complexity bounds are established for a fully stochastic zeroth-order bilevel optimization algorithm. △ Less

Submitted 29 March, 2024; originally announced April 2024.

Certain and Approximately Certain Models for Statistical Learning

Authors: Cheng Zhen, Nischal Aryal, Arash Termehchy, Alireza Aghasi, Amandeep Singh Chabada

Abstract: Real-world data is often incomplete and contains missing values. To train accurate models over real-world datasets, users need to spend a substantial amount of time and resources imputing and finding proper values for missing data items. In this paper, we demonstrate that it is possible to learn accurate models directly from data with missing values for certain training data and target models. We… ▽ More Real-world data is often incomplete and contains missing values. To train accurate models over real-world datasets, users need to spend a substantial amount of time and resources imputing and finding proper values for missing data items. In this paper, we demonstrate that it is possible to learn accurate models directly from data with missing values for certain training data and target models. We propose a unified approach for checking the necessity of data imputation to learn accurate models across various widely-used machine learning paradigms. We build efficient algorithms with theoretical guarantees to check this necessity and return accurate models in cases where imputation is unnecessary. Our extensive experiments indicate that our proposed algorithms significantly reduce the amount of time and effort needed for data imputation without imposing considerable computational overhead. △ Less

Submitted 1 March, 2024; v1 submitted 27 February, 2024; originally announced February 2024.

Comments: A technical report for a paper to appear at SIGMOD 2024

Survey of Distributed Algorithms for Resource Allocation over Multi-Agent Systems

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Mohammad Pirani, Ehsan Nekouei, Houman Zarrabi, Reza Keypour, Apostolos I. Rikos, Karl H. Johansson

Abstract: Resource allocation and scheduling in multi-agent systems present challenges due to complex interactions and decentralization. This survey paper provides a comprehensive analysis of distributed algorithms for addressing the distributed resource allocation (DRA) problem over multi-agent systems. It covers a significant area of research at the intersection of optimization, multi-agent systems, and d… ▽ More Resource allocation and scheduling in multi-agent systems present challenges due to complex interactions and decentralization. This survey paper provides a comprehensive analysis of distributed algorithms for addressing the distributed resource allocation (DRA) problem over multi-agent systems. It covers a significant area of research at the intersection of optimization, multi-agent systems, and distributed consensus-based computing. The paper begins by presenting a mathematical formulation of the DRA problem, establishing a solid foundation for further exploration. Real-world applications of DRA in various domains are examined to underscore the importance of efficient resource allocation, and relevant distributed optimization formulations are presented. The survey then delves into existing solutions for DRA, encompassing linear, nonlinear, primal-based, and dual-formulation-based approaches. Furthermore, this paper evaluates the features and properties of DRA algorithms, addressing key aspects such as feasibility, convergence rate, and network reliability. The analysis of mathematical foundations, diverse applications, existing solutions, and algorithmic properties contributes to a broader comprehension of the challenges and potential solutions for this domain. △ Less

Submitted 28 January, 2024; originally announced January 2024.

Comments: Submitted to annual reviews in control

Accelerated Distributed Allocation

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi

Abstract: Distributed allocation finds applications in many scenarios including CPU scheduling, distributed energy resource management, and networked coverage control. In this paper, we propose a fast convergent optimization algorithm with a tunable rate using the signum function. The convergence rate of the proposed algorithm can be managed by changing two parameters. We prove convergence over uniformly-co… ▽ More Distributed allocation finds applications in many scenarios including CPU scheduling, distributed energy resource management, and networked coverage control. In this paper, we propose a fast convergent optimization algorithm with a tunable rate using the signum function. The convergence rate of the proposed algorithm can be managed by changing two parameters. We prove convergence over uniformly-connected multi-agent networks. Therefore, the solution converges even if the network loses connectivity at some finite time intervals. The proposed algorithm is all-time feasible, implying that at any termination time of the algorithm, the resource-demand feasibility holds. This is in contrast to asymptotic feasibility in many dual formulation solutions (e.g., ADMM) that meet resource-demand feasibility over time and asymptotically. △ Less

Submitted 28 January, 2024; originally announced January 2024.

Comments: Conditionally accepted in IEEE SPL

Robust-to-Noise Algorithms for Distributed Resource Allocation and Scheduling

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi

Abstract: Efficient resource allocation and scheduling algorithms are essential for various distributed applications, ranging from wireless networks and cloud computing platforms to autonomous multi-agent systems and swarm robotic networks. However, real-world environments are often plagued by uncertainties and noise, leading to sub-optimal performance and increased vulnerability of traditional algorithms.… ▽ More Efficient resource allocation and scheduling algorithms are essential for various distributed applications, ranging from wireless networks and cloud computing platforms to autonomous multi-agent systems and swarm robotic networks. However, real-world environments are often plagued by uncertainties and noise, leading to sub-optimal performance and increased vulnerability of traditional algorithms. This paper addresses the challenge of robust resource allocation and scheduling in the presence of noise and disturbances. The proposed study introduces a novel sign-based dynamics for develo** robust-to-noise algorithms distributed over a multi-agent network that can adaptively handle external disturbances. Leveraging concepts from convex optimization theory, control theory, and network science the framework establishes a principled approach to design algorithms that can maintain key properties such as resource-demand balance and constraint feasibility. Meanwhile, notions of uniform-connectivity and versatile networking conditions are also addressed. △ Less

Submitted 30 November, 2023; originally announced November 2023.

Comments: IEEE/RSI ICRoM2023

Discretized Distributed Optimization over Dynamic Digraphs

Authors: Mohammadreza Doostmohammadian, Wei Jiang, Muwahida Liaquat, Alireza Aghasi, Houman Zarrabi

Abstract: We consider a discrete-time model of continuous-time distributed optimization over dynamic directed-graphs (digraphs) with applications to distributed learning. Our optimization algorithm works over general strongly connected dynamic networks under switching topologies, e.g., in mobile multi-agent systems and volatile networks due to link failures. Compared to many existing lines of work, there is… ▽ More We consider a discrete-time model of continuous-time distributed optimization over dynamic directed-graphs (digraphs) with applications to distributed learning. Our optimization algorithm works over general strongly connected dynamic networks under switching topologies, e.g., in mobile multi-agent systems and volatile networks due to link failures. Compared to many existing lines of work, there is no need for bi-stochastic weight designs on the links. The existing literature mostly needs the link weights to be stochastic using specific weight-design algorithms needed both at the initialization and at all times when the topology of the network changes. This paper eliminates the need for such algorithms and paves the way for distributed optimization over time-varying digraphs. We derive the bound on the gradient-tracking step-size and discrete time-step for convergence and prove dynamic stability using arguments from consensus algorithms, matrix perturbation theory, and Lyapunov theory. This work, particularly, is an improvement over existing stochastic-weight undirected networks in case of link removal or packet drops. This is because the existing literature may need to rerun time-consuming and computationally complex algorithms for stochastic design, while the proposed strategy works as long as the underlying network is weight-symmetric and balanced. The proposed optimization framework finds applications to distributed classification and learning. △ Less

Submitted 26 March, 2024; v1 submitted 14 November, 2023; originally announced November 2023.

Journal ref: IEEE Transactions on Automation science and Engineering 2024

Distributed Delay-Tolerant Strategies for Equality-Constraint Sum-Preserving Resource Allocation

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Maria Vrakopoulou, Hamid R. Rabiee, Usman A. Khan, Themistoklis Charalambou

Abstract: This paper proposes two nonlinear dynamics to solve constrained distributed optimization problem for resource allocation over a multi-agent network. In this setup, coupling constraint refers to resource-demand balance which is preserved at all-times. The proposed solutions can address various model nonlinearities, for example, due to quantization and/or saturation. Further, it allows to reach fast… ▽ More This paper proposes two nonlinear dynamics to solve constrained distributed optimization problem for resource allocation over a multi-agent network. In this setup, coupling constraint refers to resource-demand balance which is preserved at all-times. The proposed solutions can address various model nonlinearities, for example, due to quantization and/or saturation. Further, it allows to reach faster convergence or to robustify the solution against impulsive noise or uncertainties. We prove convergence over weakly connected networks using convex analysis and Lyapunov theory. Our findings show that convergence can be reached for general sign-preserving odd nonlinearity. We further propose delay-tolerant mechanisms to handle general bounded heterogeneous time-varying delays over the communication network of agents while preserving all-time feasibility. This work finds application in CPU scheduling and coverage control among others. This paper advances the state-of-the-art by addressing (i) possible nonlinearity on the agents/links, meanwhile handling (ii) resource-demand feasibility at all times, (iii) uniform-connectivity instead of all-time connectivity, and (iv) possible heterogeneous and time-varying delays. To our best knowledge, no existing work addresses contributions (i)-(iv) altogether. Simulations and comparative analysis are provided to corroborate our contributions. △ Less

Submitted 27 October, 2023; originally announced October 2023.

Journal ref: SCL 2023

D-SVM over Networked Systems with Non-Ideal Linking Conditions

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Houman Zarrabi

Abstract: This paper considers distributed optimization algorithms, with application in binary classification via distributed support-vector-machines (D-SVM) over multi-agent networks subject to some link nonlinearities. The agents solve a consensus-constraint distributed optimization cooperatively via continuous-time dynamics, while the links are subject to strongly sign-preserving odd nonlinear conditions… ▽ More This paper considers distributed optimization algorithms, with application in binary classification via distributed support-vector-machines (D-SVM) over multi-agent networks subject to some link nonlinearities. The agents solve a consensus-constraint distributed optimization cooperatively via continuous-time dynamics, while the links are subject to strongly sign-preserving odd nonlinear conditions. Logarithmic quantization and clip** (saturation) are two examples of such nonlinearities. In contrast to existing literature that mostly considers ideal links and perfect information exchange over linear channels, we show how general sector-bounded models affect the convergence to the optimizer (i.e., the SVM classifier) over dynamic balanced directed networks. In general, any odd sector-bounded nonlinear map** can be applied to our dynamics. The main challenge is to show that the proposed system dynamics always have one zero eigenvalue (associated with the consensus) and the other eigenvalues all have negative real parts. This is done by recalling arguments from matrix perturbation theory. Then, the solution is shown to converge to the agreement state under certain conditions. For example, the gradient tracking (GT) step size is tighter than the linear case by factors related to the upper/lower sector bounds. To the best of our knowledge, no existing work in distributed optimization and learning literature considers non-ideal link conditions. △ Less

Submitted 13 April, 2023; originally announced April 2023.

Distributed Constraint-Coupled Optimization over Lossy Networks

Authors: Mohammadreza Doostmohammadian, Usman A. Khan, Alireza Aghasi, Themistoklis Charalambous

Abstract: This paper considers distributed resource allocation and sum-preserving constrained optimization over lossy networks, where the links are unreliable and subject to packet drops. We define the conditions to ensure convergence under packet drops and link removal by focusing on two main properties of our allocation algorithm: (i) The weight-stochastic condition in typical consensus schemes is reduced… ▽ More This paper considers distributed resource allocation and sum-preserving constrained optimization over lossy networks, where the links are unreliable and subject to packet drops. We define the conditions to ensure convergence under packet drops and link removal by focusing on two main properties of our allocation algorithm: (i) The weight-stochastic condition in typical consensus schemes is reduced to balanced weights, with no need for readjusting the weights to satisfy stochasticity. (ii) The algorithm does not require all-time connectivity but instead uniform connectivity over some non-overlap** finite time intervals. First, we prove that our algorithm provides primal-feasible allocation at every iteration step and converges under the conditions (i)-(ii) and some other mild conditions on the nonlinear iterative dynamics. These nonlinearities address possible practical constraints in real applications due to, for example, saturation or quantization among others. Then, using (i)-(ii) and the notion of bond-percolation theory, we relate the packet drop rate and the network percolation threshold to the (finite) number of iterations ensuring uniform connectivity and, thus, convergence towards the optimum value. △ Less

Submitted 30 August, 2022; originally announced August 2022.

RIGID: Robust Linear Regression with Missing Data

Authors: Alireza Aghasi, MohammadJavad Feizollahi, Saeed Ghadimi

Abstract: We present a robust framework to perform linear regression with missing entries in the features. By considering an elliptical data distribution, and specifically a multivariate normal model, we are able to conditionally formulate a distribution for the missing entries and present a robust framework, which minimizes the worst case error caused by the uncertainty about the missing data. We show that… ▽ More We present a robust framework to perform linear regression with missing entries in the features. By considering an elliptical data distribution, and specifically a multivariate normal model, we are able to conditionally formulate a distribution for the missing entries and present a robust framework, which minimizes the worst case error caused by the uncertainty about the missing data. We show that the proposed formulation, which naturally takes into account the dependency between different variables, ultimately reduces to a convex program, for which a customized and scalable solver can be delivered. In addition to a detailed analysis to deliver such solver, we also asymptoticly analyze the behavior of the proposed framework, and present technical discussions to estimate the required input parameters. We complement our analysis with experiments performed on synthetic, semi-synthetic, and real data, and show how the proposed formulation improves the prediction accuracy and robustness, and outperforms the competing techniques. Missing data is a common problem associated with many datasets in machine learning. With the significant increase in using robust optimization techniques to train machine learning models, this paper presents a novel robust regression framework that operates by minimizing the uncertainty associated with missing data. The proposed approach allows training models with incomplete data, while minimizing the impact of uncertainty associated with the unavailable data. The ideas developed in this paper can be generalized beyond linear models and elliptical data distributions. △ Less

Submitted 8 November, 2022; v1 submitted 26 May, 2022; originally announced May 2022.

Distributed Finite-Sum Constrained Optimization subject to Nonlinearity on the Node Dynamics

Authors: Mohammadreza Doostmohammadian, Maria Vrakopoulou, Alireza Aghasi, Themistoklis Charalambous

Abstract: Motivated by recent development in networking and parallel data-processing, we consider a distributed and localized finite-sum (or fixed-sum) allocation technique to solve resource-constrained convex optimization problems over multi-agent networks (MANs). Such networks include (smart) agents representing an intelligent entity capable of communication, processing, and decision-making. In particular… ▽ More Motivated by recent development in networking and parallel data-processing, we consider a distributed and localized finite-sum (or fixed-sum) allocation technique to solve resource-constrained convex optimization problems over multi-agent networks (MANs). Such networks include (smart) agents representing an intelligent entity capable of communication, processing, and decision-making. In particular, we consider problems subject to practical nonlinear constraints on the dynamics of the agents in terms of their communications and actuation capabilities (referred to as the node dynamics), e.g., networks of mobile robots subject to actuator saturation and quantized communication. The considered distributed sum-preserving optimization solution further enables adding purposeful nonlinear constraints, for example, sign-based nonlinearities, to reach convergence in predefined-time or robust to impulsive noise and disturbances in faulty environments. Moreover, convergence can be achieved under minimal network connectivity requirements among the agents; thus, the solution is applicable over dynamic networks where the channels come and go due to the agent's mobility and limited range. This paper discusses how various nonlinearity constraints on the optimization problem (e.g., collaborative allocation of resources) can be addressed for different applications via a distributed setup (over a network). △ Less

Submitted 28 March, 2022; originally announced March 2022.

Comments: IEEE VTC2022

1st-Order Dynamics on Nonlinear Agents for Resource Allocation over Uniformly-Connected Networks

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Maria Vrakopoulou, Themistoklis Charalambous

Abstract: A general nonlinear $1$st-order consensus-based solution for distributed constrained convex optimization is proposed with network resource allocation applications. The solution is used to optimize continuously-differentiable strictly convex cost functions over weakly-connected undirected networks, while it is anytime feasible and models various nonlinearities to account for imperfections and const… ▽ More A general nonlinear $1$st-order consensus-based solution for distributed constrained convex optimization is proposed with network resource allocation applications. The solution is used to optimize continuously-differentiable strictly convex cost functions over weakly-connected undirected networks, while it is anytime feasible and models various nonlinearities to account for imperfections and constraints on the (physical model of) agents in terms of limited actuation capabilities, e.g., quantization and saturation. Due to such inherent nonlinearities, the existing linear solutions considering ideal agent models may not necessarily converge with guaranteed optimality and anytime feasibility. Some applications also impose specific nonlinearities, e.g., convergence in fixed/finite-time or sign-based robust disturbance-tolerant dynamics. Our proposed distributed protocol generalizes such nonlinear models. Putting convex set analysis together with nonsmooth Lyapunov analysis, we prove convergence, (i) regardless of the particular type of nonlinearity, and (ii) with weak network-connectivity requirements (uniform-connectivity). △ Less

Submitted 19 November, 2021; v1 submitted 10 September, 2021; originally announced September 2021.

Distributed support-vector-machine over dynamic balanced directed networks

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Themistoklis Charalambous, Usman A. Khan

Abstract: In this paper, we consider the binary classification problem via distributed Support-Vector-Machines (SVM), where the idea is to train a network of agents, with limited share of data, to cooperatively learn the SVM classifier for the global database. Agents only share processed information regarding the classifier parameters and the gradient of the local loss functions instead of their raw data. I… ▽ More In this paper, we consider the binary classification problem via distributed Support-Vector-Machines (SVM), where the idea is to train a network of agents, with limited share of data, to cooperatively learn the SVM classifier for the global database. Agents only share processed information regarding the classifier parameters and the gradient of the local loss functions instead of their raw data. In contrast to the existing work, we propose a continuous-time algorithm that incorporates network topology changes in discrete jumps. This hybrid nature allows us to remove chattering that arises because of the discretization of the underlying CT process. We show that the proposed algorithm converges to the SVM classifier over time-varying weight balanced directed graphs by using arguments from the matrix perturbation theory. △ Less

Submitted 1 April, 2021; originally announced April 2021.

Comments: submitted to CDC21

Fast-Convergent Dynamics for Distributed Allocation of Resources Over Switching Sparse Networks with Quantized Communication Links

Authors: Mohammadreza Doostmohammadian, Alireza Aghasi, Mohammad Pirani, Ehsan Nekouei, Usman A. Khan, Themistoklis Charalambous

Abstract: This paper proposes networked dynamics to solve resource allocation problems over time-varying multi-agent networks. The state of each agent represents the amount of used resources (or produced utilities) while the total amount of resources is fixed. The idea is to optimally allocate the resources among the group of agents by minimizing the overall cost function subject to fixed sum of resources.… ▽ More This paper proposes networked dynamics to solve resource allocation problems over time-varying multi-agent networks. The state of each agent represents the amount of used resources (or produced utilities) while the total amount of resources is fixed. The idea is to optimally allocate the resources among the group of agents by minimizing the overall cost function subject to fixed sum of resources. Each agents' information is restricted to its own state and cost function and those of its immediate in-neighbors. This is motivated by distributed applications such as mobile edge-computing, economic dispatch over smart grids, and multi-agent coverage control. This work provides a fast convergent solution (in comparison with linear dynamics) while considering relaxed network connectivity with quantized communication links. The proposed dynamics reaches optimal solution over switching (possibly disconnected) undirected networks as far as their union over some bounded non-overlap** time-intervals has a spanning-tree. We prove feasibility of the solution, uniqueness of the optimal state, and convergence to the optimal value under the proposed dynamics, where the analysis is applicable to similar 1st-order allocation dynamics with strongly sign-preserving nonlinearities, such as actuator saturation. △ Less

Submitted 25 July, 2022; v1 submitted 15 December, 2020; originally announced December 2020.

Comments: ECC2022

Inverse Constrained Reinforcement Learning

Authors: Usman Anwar, Shehryar Malik, Alireza Aghasi, Ali Ahmed

Abstract: In real world settings, numerous constraints are present which are hard to specify mathematically. However, for the real world deployment of reinforcement learning (RL), it is critical that RL agents are aware of these constraints, so that they can act safely. In this work, we consider the problem of learning constraints from demonstrations of a constraint-abiding agent's behavior. We experimental… ▽ More In real world settings, numerous constraints are present which are hard to specify mathematically. However, for the real world deployment of reinforcement learning (RL), it is critical that RL agents are aware of these constraints, so that they can act safely. In this work, we consider the problem of learning constraints from demonstrations of a constraint-abiding agent's behavior. We experimentally validate our approach and show that our framework can successfully learn the most likely constraints that the agent respects. We further show that these learned constraints are \textit{transferable} to new agents that may have different morphologies and/or reward functions. Previous works in this regard have either mainly been restricted to tabular (discrete) settings, specific types of constraints or assume the environment's transition dynamics. In contrast, our framework is able to learn arbitrary \textit{Markovian} constraints in high-dimensions in a completely model-free setting. The code can be found it: \url{https://github.com/shehryar-malik/icrl}. △ Less

Submitted 21 May, 2021; v1 submitted 19 November, 2020; originally announced November 2020.

Comments: Camera-ready version for ICML 2021

Learning To Solve Differential Equations Across Initial Conditions

Authors: Shehryar Malik, Usman Anwar, Ali Ahmed, Alireza Aghasi

Abstract: Recently, there has been a lot of interest in using neural networks for solving partial differential equations. A number of neural network-based partial differential equation solvers have been formulated which provide performances equivalent, and in some cases even superior, to classical solvers. However, these neural solvers, in general, need to be retrained each time the initial conditions or th… ▽ More Recently, there has been a lot of interest in using neural networks for solving partial differential equations. A number of neural network-based partial differential equation solvers have been formulated which provide performances equivalent, and in some cases even superior, to classical solvers. However, these neural solvers, in general, need to be retrained each time the initial conditions or the domain of the partial differential equation changes. In this work, we posit the problem of approximating the solution of a fixed partial differential equation for any arbitrary initial conditions as learning a conditional probability distribution. We demonstrate the utility of our method on Burger's Equation. △ Less

Submitted 19 April, 2020; v1 submitted 26 March, 2020; originally announced March 2020.

Bilinear Compressed Sensing under known Signs via Convex Programming

Authors: Alireza Aghasi, Ali Ahmed, Paul Hand, Babhru Joshi

Abstract: We consider the bilinear inverse problem of recovering two vectors, $\boldsymbol{x} \in\mathbb{R}^L$ and $\boldsymbol{w} \in\mathbb{R}^L$, from their entrywise product. We consider the case where $\boldsymbol{x}$ and $\boldsymbol{w}$ have known signs and are sparse with respect to known dictionaries of size $K$ and $N$, respectively. Here, $K$ and $N$ may be larger than, smaller than, or equal to… ▽ More We consider the bilinear inverse problem of recovering two vectors, $\boldsymbol{x} \in\mathbb{R}^L$ and $\boldsymbol{w} \in\mathbb{R}^L$, from their entrywise product. We consider the case where $\boldsymbol{x}$ and $\boldsymbol{w}$ have known signs and are sparse with respect to known dictionaries of size $K$ and $N$, respectively. Here, $K$ and $N$ may be larger than, smaller than, or equal to $L$. We introduce $\ell_1$-BranchHull, which is a convex program posed in the natural parameter space and does not require an approximate solution or initialization in order to be stated or solved. Under the assumptions that $\boldsymbol{x}$ and $\boldsymbol{w}$ satisfy a comparable-effective-sparsity condition and are $S_1$- and $S_2$-sparse with respect to a random dictionary, we present a recovery guarantee in a noisy case. We show that $\ell_1$-BranchHull is robust to small dense noise with high probability if the number of measurements satisfy $L\geqΩ\left((S_1+S_2)\log^{2}(K+N)\right)$. Numerical experiments show that the scaling constant in the theorem is not too large. We also introduce variants of $\ell_1$-BranchHull for the purposes of tolerating noise and outliers, and for the purpose of recovering piecewise constant signals. We provide an ADMM implementation of these variants and show they can extract piecewise constant behavior from real images. △ Less

Submitted 27 June, 2019; v1 submitted 25 June, 2019; originally announced June 2019.

Comments: arXiv admin note: substantial text overlap with arXiv:1809.08359

Simultaneous Phase Retrieval and Blind Deconvolution via Convex Programming

Authors: Ali Ahmed, Alireza Aghasi, Paul Hand

Abstract: We consider the task of recovering two real or complex $m$-vectors from phaseless Fourier measurements of their circular convolution. Our method is a novel convex relaxation that is based on a lifted matrix recovery formulation that allows a nontrivial convex relaxation of the bilinear measurements from convolution. We prove that if the two signals belong to known random subspaces of dimensions… ▽ More We consider the task of recovering two real or complex $m$-vectors from phaseless Fourier measurements of their circular convolution. Our method is a novel convex relaxation that is based on a lifted matrix recovery formulation that allows a nontrivial convex relaxation of the bilinear measurements from convolution. We prove that if the two signals belong to known random subspaces of dimensions $k$ and $n$, then they can be recovered up to the inherent scaling ambiguity with $m \gg (k+n) \log^2 m$ phaseless measurements. Our method provides the first theoretical recovery guarantee for this problem by a computationally efficient algorithm and does not require a solution estimate to be computed for initialization. Our proof is based on Rademacher complexity estimates. Additionally, we provide an alternating direction method of multipliers (ADMM) implementation and provide numerical experiments that verify the theory. △ Less

Submitted 11 May, 2019; v1 submitted 26 April, 2019; originally announced April 2019.

Comments: arXiv admin note: substantial text overlap with arXiv:1806.08091

A convex program for bilinear inversion of sparse vectors

Authors: Alireza Aghasi, Ali Ahmed, Paul Hand, Babhru Joshi

Abstract: We consider the bilinear inverse problem of recovering two vectors, $\boldsymbol{x}\in\mathbb{R}^L$ and $\boldsymbol{w}\in\mathbb{R}^L$, from their entrywise product. We consider the case where $\boldsymbol{x}$ and $\boldsymbol{w}$ have known signs and are sparse with respect to known dictionaries of size $K$ and $N$, respectively. Here, $K$ and $N$ may be larger than, smaller than, or equal to… ▽ More We consider the bilinear inverse problem of recovering two vectors, $\boldsymbol{x}\in\mathbb{R}^L$ and $\boldsymbol{w}\in\mathbb{R}^L$, from their entrywise product. We consider the case where $\boldsymbol{x}$ and $\boldsymbol{w}$ have known signs and are sparse with respect to known dictionaries of size $K$ and $N$, respectively. Here, $K$ and $N$ may be larger than, smaller than, or equal to $L$. We introduce $\ell_1$-BranchHull, which is a convex program posed in the natural parameter space and does not require an approximate solution or initialization in order to be stated or solved. We study the case where $\boldsymbol{x}$ and $\boldsymbol{w}$ are $S_1$- and $S_2$-sparse with respect to a random dictionary and present a recovery guarantee that only depends on the number of measurements as $L\geqΩ(S_1+S_2)\log^{2}(K+N)$. Numerical experiments verify that the scaling constant in the theorem is not too large. One application of this problem is the sweep distortion removal task in dielectric imaging, where one of the signals is a nonnegative reflectivity, and the other signal lives in a known subspace, for example that given by dominant wavelet coefficients. We also introduce a variants of $\ell_1$-BranchHull for the purposes of tolerating noise and outliers, and for the purpose of recovering piecewise constant signals. We provide an ADMM implementation of these variants and show they can extract piecewise constant behavior from real images. △ Less

Submitted 21 September, 2018; originally announced September 2018.

Blind Deconvolutional Phase Retrieval via Convex Programming

Authors: Ali Ahmed, Alireza Aghasi, Paul Hand

Abstract: We consider the task of recovering two real or complex $m$-vectors from phaseless Fourier measurements of their circular convolution. Our method is a novel convex relaxation that is based on a lifted matrix recovery formulation that allows a nontrivial convex relaxation of the bilinear measurements from convolution. We prove that if the two signals belong to known random subspaces of dimensions… ▽ More We consider the task of recovering two real or complex $m$-vectors from phaseless Fourier measurements of their circular convolution. Our method is a novel convex relaxation that is based on a lifted matrix recovery formulation that allows a nontrivial convex relaxation of the bilinear measurements from convolution. We prove that if the two signals belong to known random subspaces of dimensions $k$ and $n$, then they can be recovered up to the inherent scaling ambiguity with $m >> (k+n) \log^2 m$ phaseless measurements. Our method provides the first theoretical recovery guarantee for this problem by a computationally efficient algorithm and does not require a solution estimate to be computed for initialization. Our proof is based Rademacher complexity estimates. Additionally, we provide an ADMM implementation of the method and provide numerical experiments that verify the theory. △ Less

Submitted 21 June, 2018; originally announced June 2018.

Fast Convex Pruning of Deep Neural Networks

Authors: Alireza Aghasi, Afshin Abdi, Justin Romberg

Abstract: We develop a fast, tractable technique called Net-Trim for simplifying a trained neural network. The method is a convex post-processing module, which prunes (sparsifies) a trained network layer by layer, while preserving the internal responses. We present a comprehensive analysis of Net-Trim from both the algorithmic and sample complexity standpoints, centered on a fast, scalable convex optimizati… ▽ More We develop a fast, tractable technique called Net-Trim for simplifying a trained neural network. The method is a convex post-processing module, which prunes (sparsifies) a trained network layer by layer, while preserving the internal responses. We present a comprehensive analysis of Net-Trim from both the algorithmic and sample complexity standpoints, centered on a fast, scalable convex optimization program. Our analysis includes consistency results between the initial and retrained models before and after Net-Trim application and guarantees on the number of training samples needed to discover a network that can be expressed using a certain number of nonzero terms. Specifically, if there is a set of weights that uses at most $s$ terms that can re-create the layer outputs from the layer inputs, we can find these weights from $\mathcal{O}(s\log N/s)$ samples, where $N$ is the input size. These theoretical results are similar to those for sparse regression using the Lasso, and our analysis uses some of the same recently-developed tools (namely recent results on the concentration of measure and convex analysis). Finally, we propose an algorithmic framework based on the alternating direction method of multipliers (ADMM), which allows a fast and simple implementation of Net-Trim for network pruning and compression. △ Less

Submitted 25 February, 2019; v1 submitted 17 June, 2018; originally announced June 2018.

BranchHull: Convex bilinear inversion from the entrywise product of signals with known signs

Authors: Alireza Aghasi, Ali Ahmed, Paul Hand, Babhru Joshi

Abstract: We consider the bilinear inverse problem of recovering two vectors, $x$ and $w$, in $\mathbb{R}^L$ from their entrywise product. For the case where the vectors have known signs and belong to known subspaces, we introduce the convex program BranchHull, which is posed in the natural parameter space that does not require an approximate solution or initialization in order to be stated or solved. Under… ▽ More We consider the bilinear inverse problem of recovering two vectors, $x$ and $w$, in $\mathbb{R}^L$ from their entrywise product. For the case where the vectors have known signs and belong to known subspaces, we introduce the convex program BranchHull, which is posed in the natural parameter space that does not require an approximate solution or initialization in order to be stated or solved. Under the structural assumptions that $x$ and $w$ are members of known $K$ and $N$ dimensional random subspaces, we present a recovery guarantee for the noiseless case and a noisy case. In the noiseless case, we prove that the BranchHull recovers $x$ and $w$ up to the inherent scaling ambiguity with high probability when $L\ \gg\ 2(K+N)$. The analysis provides a precise upper bound on the coefficient for the sample complexity. In a noisy case, we show that with high probability the BranchHull is robust to small dense noise when $L = Ω(K+N)$. BranchHull is motivated by the sweep distortion removal task in dielectric imaging, where one of the signals is a nonnegative reflectivity, and the other signal lives in a known wavelet subspace. Additional potential applications are blind deconvolution and self-calibration. △ Less

Submitted 4 January, 2019; v1 submitted 14 February, 2017; originally announced February 2017.

Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee

Authors: Alireza Aghasi, Afshin Abdi, Nam Nguyen, Justin Romberg

Abstract: We introduce and analyze a new technique for model reduction for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Our Net-Trim algorithm prunes (sparsifies) a trained network layer-wise, removing connections at each layer by solving a convex op… ▽ More We introduce and analyze a new technique for model reduction for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Our Net-Trim algorithm prunes (sparsifies) a trained network layer-wise, removing connections at each layer by solving a convex optimization program. This program seeks a sparse set of weights at each layer that keeps the layer inputs and outputs consistent with the originally trained model. The algorithms and associated analysis are applicable to neural networks operating with the rectified linear unit (ReLU) as the nonlinear activation. We present both parallel and cascade versions of the algorithm. While the latter can achieve slightly simpler models with the same generalization performance, the former can be computed in a distributed manner. In both cases, Net-Trim significantly reduces the number of connections in the network, while also providing enough regularization to slightly reduce the generalization error. We also provide a mathematical analysis of the consistency between the initial network and the retrained model. To analyze the model sample complexity, we derive the general sufficient conditions for the recovery of a sparse transform matrix. For a single layer taking independent Gaussian random vectors of length $N$ as inputs, we show that if the network response can be described using a maximum number of $s$ non-zero weights per node, these weights can be learned from $\mathcal{O}(s\log N)$ samples. △ Less

Submitted 23 November, 2017; v1 submitted 16 November, 2016; originally announced November 2016.

Sweep Distortion Removal from THz Images via Blind Demodulation

Authors: Alireza Aghasi, Barmak Heshmat, Albert Redo-Sanchez, Justin Romberg, Ramesh Raskar

Abstract: Heavy sweep distortion induced by alignments and inter-reflections of layers of a sample is a major burden in recovering 2D and 3D information in time resolved spectral imaging. This problem cannot be addressed by conventional denoising and signal processing techniques as it heavily depends on the physics of the acquisition. Here we propose and implement an algorithmic framework based on low-rank… ▽ More Heavy sweep distortion induced by alignments and inter-reflections of layers of a sample is a major burden in recovering 2D and 3D information in time resolved spectral imaging. This problem cannot be addressed by conventional denoising and signal processing techniques as it heavily depends on the physics of the acquisition. Here we propose and implement an algorithmic framework based on low-rank matrix recovery and alternating minimization that exploits the forward model for THz acquisition. The method allows recovering the original signal in spite of the presence of temporal-spatial distortions. We address a blind-demodulation problem, where based on several observations of the sample texture modulated by an undesired sweep pattern, the two classes of signals are separated. The performance of the method is examined in both synthetic and experimental data, and the successful reconstructions are demonstrated. The proposed general scheme can be implemented to advance inspection and imaging applications in THz and other time-resolved sensing modalities. △ Less

Submitted 29 March, 2016; originally announced April 2016.

Learning Shapes by Convex Composition

Authors: Alireza Aghasi, Justin Romberg

Abstract: We present a mathematical and algorithmic scheme for learning the principal geometric elements in an image or 3D object. We build on recent work that convexifies the basic problem of finding a combination of a small number shapes that overlap and occlude one another in such a way that they "match" a given scene as closely as possible. This paper derives general sufficient conditions under which th… ▽ More We present a mathematical and algorithmic scheme for learning the principal geometric elements in an image or 3D object. We build on recent work that convexifies the basic problem of finding a combination of a small number shapes that overlap and occlude one another in such a way that they "match" a given scene as closely as possible. This paper derives general sufficient conditions under which this convex shape composition identifies a target composition. From a computational standpoint, we present two different methods for solving the associated optimization programs. The first method simply recasts the problem as a linear program, while the second uses the alternating direction method of multipliers with a series of easily computed proximal operators. △ Less

Submitted 4 July, 2016; v1 submitted 23 February, 2016; originally announced February 2016.

Sparse Shape Reconstruction

Authors: Alireza Aghasi, Justin Romberg

Abstract: This paper introduces a new shape-based image reconstruction technique applicable to a large class of imaging problems formulated in a variational sense. Given a collection of shape priors (a shape dictionary), we define our problem as choosing the right elements and geometrically composing them through basic set operations to characterize desired regions in the image. This combinatorial problem c… ▽ More This paper introduces a new shape-based image reconstruction technique applicable to a large class of imaging problems formulated in a variational sense. Given a collection of shape priors (a shape dictionary), we define our problem as choosing the right elements and geometrically composing them through basic set operations to characterize desired regions in the image. This combinatorial problem can be relaxed and then solved using classical descent methods. The main component of this relaxation is forming certain compactly supported functions which we call "knolls", and reformulating the shape representation as a basis expansion in terms of such functions. To select suitable elements of the dictionary, our problem ultimately reduces to solving a nonlinear program with sparsity constraints. We provide a new sparse nonlinear reconstruction technique to approach this problem. The performance of proposed technique is demonstrated with some standard imaging problems including image segmentation, X-ray tomography and diffusive tomography. △ Less

Submitted 28 February, 2013; originally announced March 2013.

A Hybrid Global Minimization Scheme for Accurate Source Localization in Sensor Networks

Authors: Hamidreza Aghasi, Hamidreza Amindavar, Alireza Aghasi

Abstract: We consider the localization problem of multiple wideband sources in a multi-path environment by coherently taking into account the attenuation characteristics and the time delays in the reception of the signal. Our proposed method leaves the space for unavailability of an accurate signal attenuation model in the environment by considering the model as an unknown function with reasonable prior ass… ▽ More We consider the localization problem of multiple wideband sources in a multi-path environment by coherently taking into account the attenuation characteristics and the time delays in the reception of the signal. Our proposed method leaves the space for unavailability of an accurate signal attenuation model in the environment by considering the model as an unknown function with reasonable prior assumptions about its functional space. Such approach is capable of enhancing the localization performance compared to only utilizing the signal attenuation information or the time delays. In this paper, the localization problem is modeled as a cost function in terms of the source locations, attenuation model parameters and the multi-path parameters. To globally perform the minimization, we propose a hybrid algorithm combining the differential evolution algorithm with the Levenberg-Marquardt algorithm. Besides the proposed combination of optimization schemes, supporting the technical details such as closed forms of cost function sensitivity matrices are provided. Finally, the validity of the proposed method is examined in several localization scenarios, taking into account the noise in the environment, the multi-path phenomenon and considering the sensors not being synchronized. △ Less

Submitted 2 August, 2011; v1 submitted 14 February, 2011; originally announced February 2011.