-
Positive concave deep equilibrium models
Authors:
Mateusz Gabor,
Tomasz Piotrowski,
Renato L. G. Cavalcante
Abstract:
Deep equilibrium (DEQ) models are widely recognized as a memory efficient alternative to standard neural networks, achieving state-of-the-art performance in language modeling and computer vision tasks. These models solve a fixed point equation instead of explicitly computing the output, which sets them apart from standard neural networks. However, existing DEQ models often lack formal guarantees o…
▽ More
Deep equilibrium (DEQ) models are widely recognized as a memory efficient alternative to standard neural networks, achieving state-of-the-art performance in language modeling and computer vision tasks. These models solve a fixed point equation instead of explicitly computing the output, which sets them apart from standard neural networks. However, existing DEQ models often lack formal guarantees of the existence and uniqueness of the fixed point, and the convergence of the numerical scheme used for computing the fixed point is not formally established. As a result, DEQ models are potentially unstable in practice. To address these drawbacks, we introduce a novel class of DEQ models called positive concave deep equilibrium (pcDEQ) models. Our approach, which is based on nonlinear Perron-Frobenius theory, enforces nonnegative weights and activation functions that are concave on the positive orthant. By imposing these constraints, we can easily ensure the existence and uniqueness of the fixed point without relying on additional complex assumptions commonly found in the DEQ literature, such as those based on monotone operator theory in convex analysis. Furthermore, the fixed point can be computed with the standard fixed point algorithm, and we provide theoretical guarantees of its geometric convergence, which, in particular, simplifies the training process. Experiments demonstrate the competitiveness of our pcDEQ models against other implicit models.
△ Less
Submitted 24 June, 2024; v1 submitted 6 February, 2024;
originally announced February 2024.
-
Inverse Feasibility in Over-the-Air Federated Learning
Authors:
Tomasz Piotrowski,
Rafail Ismayilov,
Matthias Frey,
Renato L. G. Cavalcante
Abstract:
We introduce the concept of inverse feasibility for linear forward models as a tool to enhance OTA FL algorithms. Inverse feasibility is defined as an upper bound on the condition number of the forward operator as a function of its parameters. We analyze an existing OTA FL model using this definition, identify areas for improvement, and propose a new OTA FL model. Numerical experiments illustrate…
▽ More
We introduce the concept of inverse feasibility for linear forward models as a tool to enhance OTA FL algorithms. Inverse feasibility is defined as an upper bound on the condition number of the forward operator as a function of its parameters. We analyze an existing OTA FL model using this definition, identify areas for improvement, and propose a new OTA FL model. Numerical experiments illustrate the main implications of the theoretical results. The proposed framework, which is based on inverse problem theory, can potentially complement existing notions of security and privacy by providing additional desirable characteristics to networks.
△ Less
Submitted 24 May, 2024; v1 submitted 25 November, 2022;
originally announced November 2022.
-
ToFFi -- Toolbox for Frequency-based Fingerprinting of Brain Signals
Authors:
Michał K. Komorowski,
Krzysztof Rykaczewski,
Tomasz Piotrowski,
Katarzyna Jurewicz,
Jakub Wojciechowski,
Anne Keitel,
Joanna Dreszer,
Włodzisław Duch
Abstract:
Spectral fingerprints (SFs) are unique power spectra signatures of human brain regions of interest (ROIs, Keitel & Gross, 2016). SFs allow for accurate ROI identification and can serve as biomarkers of differences exhibited by non-neurotypical groups. At present, there are no open-source, versatile tools to calculate spectral fingerprints. We have filled this gap by creating a modular, highly-conf…
▽ More
Spectral fingerprints (SFs) are unique power spectra signatures of human brain regions of interest (ROIs, Keitel & Gross, 2016). SFs allow for accurate ROI identification and can serve as biomarkers of differences exhibited by non-neurotypical groups. At present, there are no open-source, versatile tools to calculate spectral fingerprints. We have filled this gap by creating a modular, highly-configurable MATLAB Toolbox for Frequency-based Fingerprinting (ToFFi). It can transform MEG/EEG signals into unique spectral representations using ROIs provided by anatomical (AAL, Desikan-Killiany), functional (Schaefer), or other custom volumetric brain parcellations. Toolbox design supports reproducibility and parallel computations.
△ Less
Submitted 19 October, 2021;
originally announced October 2021.
-
Relaxed Zero-Forcing Beamformer under Temporally-Correlated Interference
Authors:
Takehiro Kono,
Masahiro Yukawa,
Tomasz Piotrowski
Abstract:
The relaxed zero-forcing (RZF) beamformer is a quadratically-and-linearly constrained minimum variance beamformer. The central question addressed in this paper is whether RZF performs better than the widely-used minimum variance distortionless response and zero-forcing beamformers under temporally-correlated interference. First, RZF is rederived by imposing an ellipsoidal constraint that bounds th…
▽ More
The relaxed zero-forcing (RZF) beamformer is a quadratically-and-linearly constrained minimum variance beamformer. The central question addressed in this paper is whether RZF performs better than the widely-used minimum variance distortionless response and zero-forcing beamformers under temporally-correlated interference. First, RZF is rederived by imposing an ellipsoidal constraint that bounds the amount of interference leakage for mitigating the intrinsic gap between the output variance and the mean squared error (MSE) which stems from the temporal correlations. Second, an analysis of RZF is presented for the single-interference case, showing how the MSE is affected by the spatio-temporal correlations between the desired and interfering sources as well as by the signal and noise powers. Third, numerical studies are presented for the multiple-interference case, showing the remarkable advantages of RZF in its basic performance as well as in its application to brain activity reconstruction from EEG data. The analytical and experimental results clarify that the RZF beamformer gives near-optimal performance in some situations.
△ Less
Submitted 11 September, 2021;
originally announced September 2021.
-
Fixed points of nonnegative neural networks
Authors:
Tomasz J. Piotrowski,
Renato L. G. Cavalcante,
Mateusz Gabor
Abstract:
We use fixed point theory to analyze nonnegative neural networks, which we define as neural networks that map nonnegative vectors to nonnegative vectors. We first show that nonnegative neural networks with nonnegative weights and biases can be recognized as monotonic and (weakly) scalable map**s within the framework of nonlinear Perron-Frobenius theory. This fact enables us to provide conditions…
▽ More
We use fixed point theory to analyze nonnegative neural networks, which we define as neural networks that map nonnegative vectors to nonnegative vectors. We first show that nonnegative neural networks with nonnegative weights and biases can be recognized as monotonic and (weakly) scalable map**s within the framework of nonlinear Perron-Frobenius theory. This fact enables us to provide conditions for the existence of fixed points of nonnegative neural networks having inputs and outputs of the same dimension, and these conditions are weaker than those recently obtained using arguments in convex analysis. Furthermore, we prove that the shape of the fixed point set of nonnegative neural networks with nonnegative weights and biases is an interval, which under mild conditions degenerates to a point. These results are then used to obtain the existence of fixed points of more general nonnegative neural networks. From a practical perspective, our results contribute to the understanding of the behavior of autoencoders, and we also offer valuable mathematical machinery for future developments in deep equilibrium models.
△ Less
Submitted 17 June, 2024; v1 submitted 30 June, 2021;
originally announced June 2021.
-
Iterative Neural Networks with Bounded Weights
Authors:
Tomasz Piotrowski,
Krzysztof Rykaczewski
Abstract:
A recent analysis of a model of iterative neural network in Hilbert spaces established fundamental properties of such networks, such as existence of the fixed points sets, convergence analysis, and Lipschitz continuity. Building on these results, we show that under a single mild condition on the weights of the network, one is guaranteed to obtain a neural network converging to its unique fixed poi…
▽ More
A recent analysis of a model of iterative neural network in Hilbert spaces established fundamental properties of such networks, such as existence of the fixed points sets, convergence analysis, and Lipschitz continuity. Building on these results, we show that under a single mild condition on the weights of the network, one is guaranteed to obtain a neural network converging to its unique fixed point. We provide a bound on the norm of this fixed point in terms of norms of weights and biases of the network. We also show why this model of a feed-forward neural network is not able to accomodate Hopfield networks under our assumption.
△ Less
Submitted 19 August, 2019; v1 submitted 16 August, 2019;
originally announced August 2019.
