-
Deformable Butterfly: A Highly Structured and Sparse Linear Transform
Authors:
Rui Lin,
Jie Ran,
King Hung Chiu,
Graziano Chesi,
Ngai Wong
Abstract:
We introduce a new kind of linear transform named Deformable Butterfly (DeBut) that generalizes the conventional butterfly matrices and can be adapted to various input-output dimensions. It inherits the fine-to-coarse-grained learnable hierarchy of traditional butterflies and when deployed to neural networks, the prominent structures and sparsity in a DeBut layer constitutes a new way for network…
▽ More
We introduce a new kind of linear transform named Deformable Butterfly (DeBut) that generalizes the conventional butterfly matrices and can be adapted to various input-output dimensions. It inherits the fine-to-coarse-grained learnable hierarchy of traditional butterflies and when deployed to neural networks, the prominent structures and sparsity in a DeBut layer constitutes a new way for network compression. We apply DeBut as a drop-in replacement of standard fully connected and convolutional layers, and demonstrate its superiority in homogenizing a neural network and rendering it favorable properties such as light weight and low inference complexity, without compromising accuracy. The natural complexity-accuracy tradeoff arising from the myriad deformations of a DeBut layer also opens up new rooms for analytical and practical research. The codes and Appendix are publicly available at: https://github.com/ruilin0212/DeBut.
△ Less
Submitted 25 March, 2022;
originally announced March 2022.
-
Exploiting Elasticity in Tensor Ranks for Compressing Neural Networks
Authors:
Jie Ran,
Rui Lin,
Hayden K. H. So,
Graziano Chesi,
Ngai Wong
Abstract:
Elasticities in depth, width, kernel size and resolution have been explored in compressing deep neural networks (DNNs). Recognizing that the kernels in a convolutional neural network (CNN) are 4-way tensors, we further exploit a new elasticity dimension along the input-output channels. Specifically, a novel nuclear-norm rank minimization factorization (NRMF) approach is proposed to dynamically and…
▽ More
Elasticities in depth, width, kernel size and resolution have been explored in compressing deep neural networks (DNNs). Recognizing that the kernels in a convolutional neural network (CNN) are 4-way tensors, we further exploit a new elasticity dimension along the input-output channels. Specifically, a novel nuclear-norm rank minimization factorization (NRMF) approach is proposed to dynamically and globally search for the reduced tensor ranks during training. Correlation between tensor ranks across multiple layers is revealed, and a graceful tradeoff between model size and accuracy is obtained. Experiments then show the superiority of NRMF over the previous non-elastic variational Bayesian matrix factorization (VBMF) scheme.
△ Less
Submitted 10 May, 2021;
originally announced May 2021.
-
HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compression
Authors:
Rui Lin,
Ching-Yun Ko,
Zhuolun He,
Cong Chen,
Yuan Cheng,
Hao Yu,
Graziano Chesi,
Ngai Wong
Abstract:
The emerging edge computing has promoted immense interests in compacting a neural network without sacrificing much accuracy. In this regard, low-rank tensor decomposition constitutes a powerful tool to compress convolutional neural networks (CNNs) by decomposing the 4-way kernel tensor into multi-stage smaller ones. Building on top of Tucker-2 decomposition, we propose a generalized Higher Order T…
▽ More
The emerging edge computing has promoted immense interests in compacting a neural network without sacrificing much accuracy. In this regard, low-rank tensor decomposition constitutes a powerful tool to compress convolutional neural networks (CNNs) by decomposing the 4-way kernel tensor into multi-stage smaller ones. Building on top of Tucker-2 decomposition, we propose a generalized Higher Order Tucker Articulated Kernels (HOTCAKE) scheme comprising four steps: input channel decomposition, guided Tucker rank selection, higher order Tucker decomposition and fine-tuning. By subjecting each CONV layer to HOTCAKE, a highly compressed CNN model with graceful accuracy trade-off is obtained. Experiments show HOTCAKE can compress even pre-compressed models and produce state-of-the-art lightweight networks.
△ Less
Submitted 28 February, 2020;
originally announced February 2020.
-
Stabilization of Linear Systems Across a Time-Varying AWGN Fading Channel
Authors:
Lanlan Su,
Vijay Gupta,
Graziano Chesi
Abstract:
This technical note investigates the minimum average transmit power required for mean-square stabilization of a discrete-time linear process across a time-varying additive white Gaussian noise (AWGN) fading channel that is presented between the sensor and the controller. We assume channel state information at both the transmitter and the receiver, and allow the transmit power to vary with the chan…
▽ More
This technical note investigates the minimum average transmit power required for mean-square stabilization of a discrete-time linear process across a time-varying additive white Gaussian noise (AWGN) fading channel that is presented between the sensor and the controller. We assume channel state information at both the transmitter and the receiver, and allow the transmit power to vary with the channel state to obtain the minimum required average transmit power via optimal power adaptation. We consider both the case of independent and identically distributed fading and fading subject to a Markov chain. Based on the proposed necessary and sufficient conditions for mean-square stabilization, we show that the minimum average transmit power to ensure stabilizability can be obtained by solving a geometric program.
△ Less
Submitted 31 July, 2019; v1 submitted 30 July, 2019;
originally announced July 2019.
-
Distributed Resource Allocation over Time-varying Balanced Digraphs with Discrete-time Communication
Authors:
Lanlan Su,
Mengmou Li,
Vijay Gupta,
Graziano Chesi
Abstract:
This work is concerned with the problem of distributed resource allocation in continuous-time setting but with discrete-time communication over infinitely jointly connected and balanced digraphs. We provide a passivity-based perspective for the continuous-time algorithm, based on which an intermittent communication scheme is developed. Particularly, a periodic communication scheme is first derived…
▽ More
This work is concerned with the problem of distributed resource allocation in continuous-time setting but with discrete-time communication over infinitely jointly connected and balanced digraphs. We provide a passivity-based perspective for the continuous-time algorithm, based on which an intermittent communication scheme is developed. Particularly, a periodic communication scheme is first derived through analyzing the passivity degradation over output sampling of the distributed dynamics at each node. Then, an asynchronous distributed event-triggered scheme is further developed. The sampled-based event-triggered communication scheme is exempt from Zeno behavior as the minimum inter-event time is lower bounded by the sampling period. The parameters in the proposed algorithm rely only on local information of each individual nodes, which can be designed in a truly distributed fashion
△ Less
Submitted 15 January, 2021; v1 submitted 30 July, 2019;
originally announced July 2019.
-
Input-Feedforward-Passivity-Based Distributed Optimization Over Jointly Connected Balanced Digraphs
Authors:
Mengmou Li,
Graziano Chesi,
Yiguang Hong
Abstract:
In this paper, a distributed optimization problem is investigated via input feedforward passivity. First, an input-feedforward-passivity-based continuous-time distributed algorithm is proposed. It is shown that the error system of the proposed algorithm can be decomposed into a group of individual input feedforward passive (IFP) systems that interact with each other using output feedback informati…
▽ More
In this paper, a distributed optimization problem is investigated via input feedforward passivity. First, an input-feedforward-passivity-based continuous-time distributed algorithm is proposed. It is shown that the error system of the proposed algorithm can be decomposed into a group of individual input feedforward passive (IFP) systems that interact with each other using output feedback information. Based on this IFP framework, convergence conditions of a suitable coupling gain are derived over weight-balanced and uniformly jointly strongly connected (UJSC) topologies. It is also shown that the IFP-based algorithm converges exponentially when the topology is strongly connected. Second, a novel distributed derivative feedback algorithm is proposed based on the passivation of IFP systems. While most works on directed topologies require knowledge of eigenvalues of the graph Laplacian, the derivative feedback algorithm is fully distributed, namely, it is robust against randomly changing weight-balanced digraphs with any positive coupling gain and without knowing any global information. Finally, numerical examples are presented to illustrate the proposed distributed algorithms.
△ Less
Submitted 1 October, 2020; v1 submitted 9 May, 2019;
originally announced May 2019.
