-
High-Dimensional Inference over Networks: Linear Convergence and Statistical Guarantees
Authors:
Ying Sun,
Marie Maros,
Gesualdo Scutari,
Guang Cheng
Abstract:
We study sparse linear regression over a network of agents, modeled as an undirected graph and no server node. The estimation of the $s$-sparse parameter is formulated as a constrained LASSO problem wherein each agent owns a subset of the $N$ total observations. We analyze the convergence rate and statistical guarantees of a distributed projected gradient tracking-based algorithm under high-dimens…
▽ More
We study sparse linear regression over a network of agents, modeled as an undirected graph and no server node. The estimation of the $s$-sparse parameter is formulated as a constrained LASSO problem wherein each agent owns a subset of the $N$ total observations. We analyze the convergence rate and statistical guarantees of a distributed projected gradient tracking-based algorithm under high-dimensional scaling, allowing the ambient dimension $d$ to grow with (and possibly exceed) the sample size $N$. Our theory shows that, under standard notions of restricted strong convexity and smoothness of the loss functions, suitable conditions on the network connectivity and algorithm tuning, the distributed algorithm converges globally at a {\it linear} rate to an estimate that is within the centralized {\it statistical precision} of the model, $O(s\log d/N)$. When $s\log d/N=o(1)$, a condition necessary for statistical consistency, an $\varepsilon$-optimal solution is attained after $\mathcal{O}(κ\log (1/\varepsilon))$ gradient computations and $O (κ/(1-ρ) \log (1/\varepsilon))$ communication rounds, where $κ$ is the restricted condition number of the loss function and $ρ$ measures the network connectivity. The computation cost matches that of the centralized projected gradient algorithm despite having data distributed; whereas the communication rounds reduce as the network connectivity improves. Overall, our study reveals interesting connections between statistical efficiency, network connectivity \& topology, and convergence rate in high dimensions.
△ Less
Submitted 20 January, 2022;
originally announced January 2022.
-
On Decentralized Tracking with ADMM for Problems with Time-Varying Curvature
Authors:
Marie Maros,
Joakim Jalén
Abstract:
We analyze the performance of the alternating direction method of multipliers (ADMM) to track, in a decentralized manner, a solution of a stochastic sequence of optimization problems parametrized by a discrete time Markov process. The main advantage of considering a stochastic model is that we allow the objective functions to occasionally lose strong convexity and/or Lipschitz continuity of their…
▽ More
We analyze the performance of the alternating direction method of multipliers (ADMM) to track, in a decentralized manner, a solution of a stochastic sequence of optimization problems parametrized by a discrete time Markov process. The main advantage of considering a stochastic model is that we allow the objective functions to occasionally lose strong convexity and/or Lipschitz continuity of their gradients. Due to the stochastic nature of our model, the tracking statement is given in a mean square deviation error.
△ Less
Submitted 15 March, 2019;
originally announced March 2019.
-
Eco-PANDA: A Computationally Economic, Geometrically Converging, Dual Optimization Method on Time-Varying Undirected Graphs
Authors:
Marie Maros,
Joakim Jaldén
Abstract:
In this paper we consider distributed convex optimization over time-varying undirected graphs. We propose a linearized version of primarily averaged network dual ascent (PANDA) while requiring less computational costs. The proposed method, economic primarily averaged network dual ascent (Eco-PANDA), provably converges at R-linear rate to the optimal point given that the agents' objective functions…
▽ More
In this paper we consider distributed convex optimization over time-varying undirected graphs. We propose a linearized version of primarily averaged network dual ascent (PANDA) while requiring less computational costs. The proposed method, economic primarily averaged network dual ascent (Eco-PANDA), provably converges at R-linear rate to the optimal point given that the agents' objective functions are strongly convex and have Lipschitz continuous gradients. Therefore, the method is competitive, in terms of type of rate, with both DIGing and PANDA. The proposed method halves the communication costs of methods like DIGing while still converging R-linearly and having the same per iterate complexity.
△ Less
Submitted 29 October, 2018;
originally announced October 2018.
-
A geometrically converging dual method for distributed optimization over time-varying graphs
Authors:
Marie Maros,
Joakim Jaldén
Abstract:
In this paper we consider a distributed convex optimization problem over time-varying undirected networks. We propose a dual method, primarily averaged network dual ascent (PANDA), that is proven to converge R-linearly to the optimal point given that the agents objective functions are strongly convex and have Lipschitz continuous gradients. Like dual decomposition, PANDA requires half the amount o…
▽ More
In this paper we consider a distributed convex optimization problem over time-varying undirected networks. We propose a dual method, primarily averaged network dual ascent (PANDA), that is proven to converge R-linearly to the optimal point given that the agents objective functions are strongly convex and have Lipschitz continuous gradients. Like dual decomposition, PANDA requires half the amount of variable exchanges per iterate of methods based on DIGing, and can provide with practical improved performance as empirically demonstrated.
△ Less
Submitted 12 October, 2018;
originally announced October 2018.
-
Dynamic Power Allocation for Smart Grids via ADMM
Authors:
Marie Maros,
Joakim Jaldén
Abstract:
Electric power distribution systems will encounter fluctuations in supply due to the introduction of renewable sources with high variability in generation capacity. It is therefore necessary to provide algorithms that are capable of dynamically finding approximate solutions. We propose two semi-distributed algorithms based on ADMM and discuss their advantages and disadvantages. One of the algorith…
▽ More
Electric power distribution systems will encounter fluctuations in supply due to the introduction of renewable sources with high variability in generation capacity. It is therefore necessary to provide algorithms that are capable of dynamically finding approximate solutions. We propose two semi-distributed algorithms based on ADMM and discuss their advantages and disadvantages. One of the algorithms computes a feasible approximate of the optimal power allocation at each instance. We require coordination between the nodes to guarantee feasibility of each of the iterates. We bound the distance from the approximate solutions to the optimal solution as a function of the variation in optimal power allocation. Finally, we verify our results via experiments.
△ Less
Submitted 4 May, 2018;
originally announced May 2018.
-
On the Q-linear convergence of Distributed Generalized ADMM under non-strongly convex function components
Authors:
Marie Maros,
Joakim Jaldén
Abstract:
Solving optimization problems in multi-agent networks where each agent only has partial knowledge of the problem has become an increasingly important problem. In this paper we consider the problem of minimizing the sum of $n$ convex functions. We assume that each function is only known by one agent. We show that Generalized Distributed ADMM converges Q-linearly to the solution of the mentioned opt…
▽ More
Solving optimization problems in multi-agent networks where each agent only has partial knowledge of the problem has become an increasingly important problem. In this paper we consider the problem of minimizing the sum of $n$ convex functions. We assume that each function is only known by one agent. We show that Generalized Distributed ADMM converges Q-linearly to the solution of the mentioned optimization problem if the over all objective function is strongly convex but the functions known by each agent are allowed to be only convex. Establishing Q-linear convergence allows for tracking statements that can not be made if only R-linear convergence is guaranteed. Further, we establish the equivalence between Generalized Distributed ADMM and P-EXTRA for a sub-set of mixing matrices. This equivalence yields insights in the convergence of P-EXTRA when overshooting to accelerate convergence.
△ Less
Submitted 4 May, 2018;
originally announced May 2018.
-
PANDA: A Dual Linearly Converging Method for Distributed Optimization over Time-Varying Undirected Graphs
Authors:
Marie Maros,
Joakim Jaldén
Abstract:
In this paper we consider a distributed convex optimization problem over time-varying networks. We propose a dual method that converges R-linearly to the optimal point given that the agents' objective functions are strongly convex and have Lipschitz continuous gradients. The proposed method requires half the amount of variable exchanges per iterate than methods based on DIGing, and yields improved…
▽ More
In this paper we consider a distributed convex optimization problem over time-varying networks. We propose a dual method that converges R-linearly to the optimal point given that the agents' objective functions are strongly convex and have Lipschitz continuous gradients. The proposed method requires half the amount of variable exchanges per iterate than methods based on DIGing, and yields improved practical performance as empirically demonstrated.
△ Less
Submitted 20 April, 2018; v1 submitted 22 March, 2018;
originally announced March 2018.
-
ADMM for Distributed Dynamic Beamforming
Authors:
Marie Maros,
Joakim Jaldén
Abstract:
This paper shows the capability the alternating direction method of multipliers (ADMM) has to track, in a distributed manner, the optimal down-link beam-forming solution in a multiple input multiple output (MISO) multi-cell network given a dynamic channel. Each time the channel changes, ADMM is allowed to perform one algorithm iteration. In order to implement the proposed scheme, the base stations…
▽ More
This paper shows the capability the alternating direction method of multipliers (ADMM) has to track, in a distributed manner, the optimal down-link beam-forming solution in a multiple input multiple output (MISO) multi-cell network given a dynamic channel. Each time the channel changes, ADMM is allowed to perform one algorithm iteration. In order to implement the proposed scheme, the base stations are not required to exchange channel state information (CSI), but will require to exchange interference values once. We show ADMM's tracking ability in terms of the algorithm's Lyapunov function given that the primal and dual solutions to the convex optimization problem at hand can be understood as a continuous map** from the problem's parameters. We show that this holds true even considering that the problem looses strong convexity when it is made distributed. We then show that these requirements hold for the down-link, and consequently up-link, beam-forming case. Numerical examples corroborating the theoretical findings are also provided.
△ Less
Submitted 12 August, 2016;
originally announced September 2016.
