-
Understanding A Class of Decentralized and Federated Optimization Algorithms: A Multi-Rate Feedback Control Perspective
Authors:
Xinwei Zhang,
Mingyi Hong,
Nicola Elia
Abstract:
Distributed algorithms have been playing an increasingly important role in many applications such as machine learning, signal processing, and control. Significant research efforts have been devoted to develo** and analyzing new algorithms for various applications. In this work, we provide a fresh perspective to understand, analyze, and design distributed optimization algorithms. Through the lens…
▽ More
Distributed algorithms have been playing an increasingly important role in many applications such as machine learning, signal processing, and control. Significant research efforts have been devoted to develo** and analyzing new algorithms for various applications. In this work, we provide a fresh perspective to understand, analyze, and design distributed optimization algorithms. Through the lens of multi-rate feedback control, we show that a wide class of distributed algorithms, including popular decentralized/federated schemes, can be viewed as discretizing a certain continuous-time feedback control system, possibly with multiple sampling rates, such as decentralized gradient descent, gradient tracking, and federated averaging. This key observation not only allows us to develop a generic framework to analyze the convergence of the entire algorithm class. More importantly, it also leads to an interesting way of designing new distributed algorithms. We develop the theory behind our framework and provide examples to highlight how the framework can be used in practice.
△ Less
Submitted 1 November, 2022; v1 submitted 26 April, 2022;
originally announced April 2022.
-
A Fast Saddle-Point Dynamical System Approach to Robust Deep Learning
Authors:
Yasaman Esfandiari,
Aditya Balu,
Keivan Ebrahimi,
Umesh Vaidya,
Nicola Elia,
Soumik Sarkar
Abstract:
Recent focus on robustness to adversarial attacks for deep neural networks produced a large variety of algorithms for training robust models. Most of the effective algorithms involve solving the min-max optimization problem for training robust models (min step) under worst-case attacks (max step). However, they often suffer from high computational cost from running several inner maximization itera…
▽ More
Recent focus on robustness to adversarial attacks for deep neural networks produced a large variety of algorithms for training robust models. Most of the effective algorithms involve solving the min-max optimization problem for training robust models (min step) under worst-case attacks (max step). However, they often suffer from high computational cost from running several inner maximization iterations (to find an optimal attack) inside every outer minimization iteration. Therefore, it becomes difficult to readily apply such algorithms for moderate to large size real world data sets. To alleviate this, we explore the effectiveness of iterative descent-ascent algorithms where the maximization and minimization steps are executed in an alternate fashion to simultaneously obtain the worst-case attack and the corresponding robust model. Specifically, we propose a novel discrete-time dynamical system-based algorithm that aims to find the saddle point of a min-max optimization problem in the presence of uncertainties. Under the assumptions that the cost function is convex and uncertainties enter concavely in the robust learning problem, we analytically show that our algorithm converges asymptotically to the robust optimal solution under a general adversarial budget constraints as induced by $\ell_p$ norm, for $1\leq p\leq \infty$. Based on our proposed analysis, we devise a fast robust training algorithm for deep neural networks. Although such training involves highly non-convex robust optimization problems, empirical results show that the algorithm can achieve significant robustness compared to other state-of-the-art robust models on benchmark data sets.
△ Less
Submitted 1 March, 2021; v1 submitted 18 October, 2019;
originally announced October 2019.
-
Youla Coding and Computation of Gaussian Feedback Capacity
Authors:
Chong Li,
Nicola Elia
Abstract:
In this paper, we propose an approach to numerically compute the feedback capacity of stationary finite dimensional Gaussian channels and construct (arbitrarily close to) capacity-achieving feedback codes. In particular, we first extend the interpretation of feedback communication over stationary finite dimensional Gaussian channels as feedback control systems by showing that, the problem of findi…
▽ More
In this paper, we propose an approach to numerically compute the feedback capacity of stationary finite dimensional Gaussian channels and construct (arbitrarily close to) capacity-achieving feedback codes. In particular, we first extend the interpretation of feedback communication over stationary finite dimensional Gaussian channels as feedback control systems by showing that, the problem of finding stabilizing feedback controllers with maximal reliable transmission rate over Youla parameters coincides with the problem of finding strictly causal filters to achieve feedback capacity derived in [2]. This extended interpretation provides an approach to construct deterministic feedback coding schemes with double exponential decaying error probability. We next propose asymptotic capacity-achieving upper bounds, which can be numerically evaluated by solving finite dimensional convex optimizations. From the filters that achieve the upper bounds, we apply the Youla-based interpretation to construct feasible filters, i.e., feedback codes, leading to a sequence of lower bounds. We prove the sequence of lower bounds is asymptotically capacity-achieving.
△ Less
Submitted 12 January, 2018;
originally announced January 2018.
-
A Web-based Tool for Identifying Strategic Intervention Points in Complex Systems
Authors:
Sotiris Moschoyiannis,
Nicholas Elia,
Alexandra S. Penn,
David J. B. Lloyd,
Chris Knight
Abstract:
Steering a complex system towards a desired outcome is a challenging task. The lack of clarity on the system's exact architecture and the often scarce scientific data upon which to base the operationalisation of the dynamic rules that underpin the interactions between participant entities are two contributing factors. We describe an analytical approach that builds on Fuzzy Cognitive Map** (FCM)…
▽ More
Steering a complex system towards a desired outcome is a challenging task. The lack of clarity on the system's exact architecture and the often scarce scientific data upon which to base the operationalisation of the dynamic rules that underpin the interactions between participant entities are two contributing factors. We describe an analytical approach that builds on Fuzzy Cognitive Map** (FCM) to address the latter and represent the system as a complex network. We apply results from network controllability to address the former and determine minimal control configurations - subsets of factors, or system levers, which comprise points for strategic intervention in steering the system. We have implemented the combination of these techniques in an analytical tool that runs in the browser, and generates all minimal control configurations of a complex network. We demonstrate our approach by reporting on our experience of working alongside industrial, local-government, and NGO stakeholders in the Humber region, UK. Our results are applied to the decision-making process involved in the transition of the region to a bio-based economy.
△ Less
Submitted 1 August, 2016;
originally announced August 2016.
-
Control approach to computing the feedback capacity for stationary finite dimensional Gaussian channels
Authors:
Chong Li,
Nicola Elia
Abstract:
We firstly extend the interpretation of feedback communication over stationary finite dimensional Gaussian channels as feedback control systems by showing that, the problem of finding stabilizing feedback controllers with maximal reliable transmission rate over Youla parameters coincides with the problem of finding strictly causal filters to achieve feedback capacity recently derived in [1]. The a…
▽ More
We firstly extend the interpretation of feedback communication over stationary finite dimensional Gaussian channels as feedback control systems by showing that, the problem of finding stabilizing feedback controllers with maximal reliable transmission rate over Youla parameters coincides with the problem of finding strictly causal filters to achieve feedback capacity recently derived in [1]. The aforementioned interpretation provides an approach to construct deterministic feedback coding schemes (with double exponential decaying error probability). We next propose an asymptotic capacity-achieving upper bounds, which can be numerically evaluated by solving finite dimensional dual optimizations. From the filters that achieve upper bounds, we derive feasible filters which lead to a sequence of lower bounds. Thus, from the lower bound filters we obtain communication systems that achieve the lower bound rate. Extensive examples show the sequence of lower bounds is asymptotic capacity-achieving as well.
△ Less
Submitted 30 November, 2015;
originally announced November 2015.
-
Upper Bound on the Capacity of Gaussian Channels with Noisy Feedback
Authors:
Chong Li,
Nicola Elia
Abstract:
We consider an additive Gaussian channel with additive Gaussian noise feedback. We derive an upper bound on the n-block capacity (defined by Cover [1]). It is shown that this upper bound can be obtained by solving a convex optimization problem. With stationarity assumptions on Gaussian noise processes, we characterize the limit of the n-block upper bound and prove that this limit is the upper boun…
▽ More
We consider an additive Gaussian channel with additive Gaussian noise feedback. We derive an upper bound on the n-block capacity (defined by Cover [1]). It is shown that this upper bound can be obtained by solving a convex optimization problem. With stationarity assumptions on Gaussian noise processes, we characterize the limit of the n-block upper bound and prove that this limit is the upper bound of the noisy feedback (shannon) capacity.
△ Less
Submitted 7 January, 2012;
originally announced January 2012.
-
Bounds on the Achievable Rate of Noisy feedback Gaussian Channels under Linear Feedback Coding Scheme
Authors:
Chong Li,
Nicola Elia
Abstract:
In this paper, we investigate the additive Gaussian noise channel with noisy feedback. We consider the setup of linear coding of the feedback information and Gaussian signaling of the message (i.e. Cover-Pombra Scheme). Then, we derive the upper and lower bounds on the largest achievable rate for this setup. We show that these two bounds can be obtained by solving two convex optimization problems.…
▽ More
In this paper, we investigate the additive Gaussian noise channel with noisy feedback. We consider the setup of linear coding of the feedback information and Gaussian signaling of the message (i.e. Cover-Pombra Scheme). Then, we derive the upper and lower bounds on the largest achievable rate for this setup. We show that these two bounds can be obtained by solving two convex optimization problems. Finally, we present some simulations and discussion.
△ Less
Submitted 13 August, 2011;
originally announced August 2011.
-
The Information Flow and Capacity of Channels with Noisy Feedback
Authors:
Chong Li,
Nicola Elia
Abstract:
In this paper, we consider some long-standing problems in communication systems with access to noisy feedback. We introduce a new notion, the residual directed information, to capture the effective information flow (i.e. mutual information between the message and the channel outputs) in the forward channel. In light of this new concept, we investigate discrete memoryless channels (DMC) with noisy…
▽ More
In this paper, we consider some long-standing problems in communication systems with access to noisy feedback. We introduce a new notion, the residual directed information, to capture the effective information flow (i.e. mutual information between the message and the channel outputs) in the forward channel. In light of this new concept, we investigate discrete memoryless channels (DMC) with noisy feedback and prove that the noisy feedback capacity is not achievable by using any typical closed-loop encoder (non-trivially taking feedback information to produce channel inputs). We then show that the residual directed information can be used to characterize the capacity of channels with noisy feedback. Finally, we provide computable bounds on the noisy feedback capacity, which are characterized by the causal conditional directed information.
△ Less
Submitted 2 December, 2011; v1 submitted 13 August, 2011;
originally announced August 2011.
-
Convergence of Fundamental Limitations in Feedback Communication, Estimation, and Feedback Control over Gaussian Channels
Authors:
Jialing Liu,
Nicola Elia
Abstract:
In this paper, we establish the connections of the fundamental limitations in feedback communication, estimation, and feedback control over Gaussian channels, from a unifying perspective for information, estimation, and control. The optimal feedback communication system over a Gaussian necessarily employs the Kalman filter (KF) algorithm, and hence can be transformed into an estimation system an…
▽ More
In this paper, we establish the connections of the fundamental limitations in feedback communication, estimation, and feedback control over Gaussian channels, from a unifying perspective for information, estimation, and control. The optimal feedback communication system over a Gaussian necessarily employs the Kalman filter (KF) algorithm, and hence can be transformed into an estimation system and a feedback control system over the same channel. This follows that the information rate of the communication system is alternatively given by the decay rate of the Cramer-Rao bound (CRB) of the estimation system and by the Bode integral (BI) of the control system. Furthermore, the optimal tradeoff between the channel input power and information rate in feedback communication is alternatively characterized by the optimal tradeoff between the (causal) one-step prediction mean-square error (MSE) and (anti-causal) smoothing MSE (of an appropriate form) in estimation, and by the optimal tradeoff between the regulated output variance with causal feedback and the disturbance rejection measure (BI or degree of anti-causality) in feedback control. All these optimal tradeoffs have an interpretation as the tradeoff between causality and anti-causality. Utilizing and motivated by these relations, we provide several new results regarding the feedback codes and information theoretic characterization of KF. Finally, the extension of the finite-horizon results to infinite horizon is briefly discussed under specific dimension assumptions (the asymptotic feedback capacity problem is left open in this paper).
△ Less
Submitted 1 October, 2009;
originally announced October 2009.
-
Phase Transitions on Fixed Connected Graphs and Random Graphs in the Presence of Noise
Authors:
Jialing Liu,
Vikas Yadav,
Hullas Sehgal,
Joshua M. Olson,
Haifeng Liu,
Nicola Elia
Abstract:
In this paper, we study the phase transition behavior emerging from the interactions among multiple agents in the presence of noise. We propose a simple discrete-time model in which a group of non-mobile agents form either a fixed connected graph or a random graph process, and each agent, taking bipolar value either +1 or -1, updates its value according to its previous value and the noisy measur…
▽ More
In this paper, we study the phase transition behavior emerging from the interactions among multiple agents in the presence of noise. We propose a simple discrete-time model in which a group of non-mobile agents form either a fixed connected graph or a random graph process, and each agent, taking bipolar value either +1 or -1, updates its value according to its previous value and the noisy measurements of the values of the agents connected to it. We present proofs for the occurrence of the following phase transition behavior: At a noise level higher than some threshold, the system generates symmetric behavior (vapor or melt of magnetization) or disagreement; whereas at a noise level lower than the threshold, the system exhibits spontaneous symmetry breaking (solid or magnetization) or consensus. The threshold is found analytically. The phase transition occurs for any dimension. Finally, we demonstrate the phase transition behavior and all analytic results using simulations. This result may be found useful in the study of the collective behavior of complex systems under communication constraints.
△ Less
Submitted 24 August, 2008;
originally announced August 2008.
-
Capacity-achieving Feedback Scheme for Gaussian Finite-State Markov Channels with Channel State Information
Authors:
Jialing Liu,
Nicola Elia,
Sekhar Tatikonda
Abstract:
In this paper, we propose capacity-achieving communication schemes for Gaussian finite-state Markov channels (FSMCs) subject to an average channel input power constraint, under the assumption that the transmitters can have access to delayed noiseless output feedback as well as instantaneous or delayed channel state information (CSI). We show that the proposed schemes reveals connections between fe…
▽ More
In this paper, we propose capacity-achieving communication schemes for Gaussian finite-state Markov channels (FSMCs) subject to an average channel input power constraint, under the assumption that the transmitters can have access to delayed noiseless output feedback as well as instantaneous or delayed channel state information (CSI). We show that the proposed schemes reveals connections between feedback communication and feedback control.
△ Less
Submitted 7 October, 2010; v1 submitted 14 August, 2008;
originally announced August 2008.
-
Gaussian Channels with Feedback: Optimality, Fundamental Limitations, and Connections of Communication, Estimation, and Control
Authors:
Jialing Liu,
Nicola Elia
Abstract:
Gaussian channels with memory and with noiseless feedback have been widely studied in the information theory literature. However, a coding scheme to achieve the feedback capacity is not available. In this paper, a coding scheme is proposed to achieve the feedback capacity for Gaussian channels. The coding scheme essentially implements the celebrated Kalman filter algorithm, and is equivalent to…
▽ More
Gaussian channels with memory and with noiseless feedback have been widely studied in the information theory literature. However, a coding scheme to achieve the feedback capacity is not available. In this paper, a coding scheme is proposed to achieve the feedback capacity for Gaussian channels. The coding scheme essentially implements the celebrated Kalman filter algorithm, and is equivalent to an estimation system over the same channel without feedback. It reveals that the achievable information rate of the feedback communication system can be alternatively given by the decay rate of the Cramer-Rao bound of the associated estimation system. Thus, combined with the control theoretic characterizations of feedback communication (proposed by Elia), this implies that the fundamental limitations in feedback communication, estimation, and control coincide. This leads to a unifying perspective that integrates information, estimation, and control. We also establish the optimality of the Kalman filtering in the sense of information transmission, a supplement to the optimality of Kalman filtering in the sense of information processing proposed by Mitter and Newton. In addition, the proposed coding scheme generalizes the Schalkwijk-Kailath codes and reduces the coding complexity and coding delay. The construction of the coding scheme amounts to solving a finite-dimensional optimization problem. A simplification to the optimal stationary input distribution developed by Yang, Kavcic, and Tatikonda is also obtained. The results are verified in a numerical example.
△ Less
Submitted 26 December, 2005;
originally announced December 2005.
