-
A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks
Authors:
Vignesh Kothapalli,
Tom Tirer,
Joan Bruna
Abstract:
Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in GNNs is not well understood. In this paper, we focus on node-wise classification, illustrated with community detection on stochastic block model graphs, and explore the feature evolution through the lens of the "Neural C…
▽ More
Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in GNNs is not well understood. In this paper, we focus on node-wise classification, illustrated with community detection on stochastic block model graphs, and explore the feature evolution through the lens of the "Neural Collapse" (NC) phenomenon. When training instance-wise deep classifiers (e.g. for image classification) beyond the zero training error point, NC demonstrates a reduction in the deepest features' within-class variability and an increased alignment of their class means to certain symmetric structures. We start with an empirical study that shows that a decrease in within-class variability is also prevalent in the node-wise classification setting, however, not to the extent observed in the instance-wise case. Then, we theoretically study this distinction. Specifically, we show that even an "optimistic" mathematical model requires that the graphs obey a strict structural condition in order to possess a minimizer with exact collapse. Interestingly, this condition is viable also for heterophilic graphs and relates to recent empirical studies on settings with improved GNNs' generalization. Furthermore, by studying the gradient dynamics of the theoretical model, we provide reasoning for the partial collapse observed empirically. Finally, we present a study on the evolution of within- and between-class feature variability across layers of a well-trained GNN and contrast the behavior with spectral methods.
△ Less
Submitted 26 October, 2023; v1 submitted 4 July, 2023;
originally announced July 2023.
-
Performance Analysis of Automotive SAR With Radar Based Motion Estimation
Authors:
Oded Bialer,
Tom Tirer
Abstract:
Automotive synthetic aperture radar (SAR) can achieve a significant angular resolution enhancement for detecting static objects, which is essential for automated driving. Obtaining high resolution SAR images requires precise ego vehicle velocity estimation. A small velocity estimation error can result in a focused SAR image with objects at offset angles. In this paper, we consider an automotive SA…
▽ More
Automotive synthetic aperture radar (SAR) can achieve a significant angular resolution enhancement for detecting static objects, which is essential for automated driving. Obtaining high resolution SAR images requires precise ego vehicle velocity estimation. A small velocity estimation error can result in a focused SAR image with objects at offset angles. In this paper, we consider an automotive SAR system that produces SAR images of static objects based on ego vehicle velocity estimation from the radar return signal without the overhead in complexity and cost of using an auxiliary global navigation satellite system (GNSS) and inertial measurement unit (IMU). We derive a novel analytical approximation for the automotive SAR angle estimation error variance when the velocity is estimated by the radar. The developed analytical analysis closely predicts the true SAR angle estimation variance, and also provides insights on the effects of the radar parameters and the environment condition on the automotive SAR angle estimation error. We evaluate via the analytical analysis and simulation tests the radar settings and environment condition in which the automotive SAR attains a significant performance gain over the angular resolution of the short aperture physical antenna array. We show that, perhaps surprisingly, when the velocity is estimated by the radar the performance advantage of automotive SAR is realized only in limited conditions. Hence since its implementation comes with an increase in computation and system complexity as well as an increase in the detection delay it should be used carefully and selectively.
△ Less
Submitted 21 April, 2022;
originally announced April 2022.
-
Direction of Arrival Estimation and Phase-Correction for Non-Coherent Sub-Arrays: A Convex Optimization Approach
Authors:
Tom Tirer,
Oded Bialer
Abstract:
Estimating the direction of arrival (DOA) of sources is an important problem in aerospace and vehicular communication, localization and radar. In this paper, we consider a challenging multi-source DOA estimation task, where the receiving antenna array is composed of non-coherent sub-arrays, i.e., sub-arrays that observe different unknown phase shifts at every snapshot (e.g., due to waiving the dem…
▽ More
Estimating the direction of arrival (DOA) of sources is an important problem in aerospace and vehicular communication, localization and radar. In this paper, we consider a challenging multi-source DOA estimation task, where the receiving antenna array is composed of non-coherent sub-arrays, i.e., sub-arrays that observe different unknown phase shifts at every snapshot (e.g., due to waiving the demanding synchronization of local oscillators across the entire array). We formulate this problem as the reconstruction of joint sparse and low-rank matrices, and solve the problem's convex relaxation. To scale the optimization complexity with the number of snapshots better than general-purpose solvers, we design an optimization scheme, based on integrating the alternating direction method of multipliers and the accelerated proximal gradient techniques, that exploits the structure of the problem. While the DOAs can be estimated from the solution of the aforementioned convex problem, we further show how an improvement is obtained if, instead, one estimates from this solution only the sub-arrays' phase shifts. This is done using another, computationally-light, convex relaxation that is practically tight. Using the estimated phase shifts, "phase-corrected" observations are created and a final plain ("coherent") DOA estimation method can be applied. Numerical experiments show the performance advantages of the proposed strategies over existing methods.
△ Less
Submitted 15 February, 2022;
originally announced February 2022.
-
On the Convergence Rate of Projected Gradient Descent for a Back-Projection based Objective
Authors:
Tom Tirer,
Raja Giryes
Abstract:
Ill-posed linear inverse problems appear in many scientific setups, and are typically addressed by solving optimization problems, which are composed of data fidelity and prior terms. Recently, several works have considered a back-projection (BP) based fidelity term as an alternative to the common least squares (LS), and demonstrated excellent results for popular inverse problems. These works have…
▽ More
Ill-posed linear inverse problems appear in many scientific setups, and are typically addressed by solving optimization problems, which are composed of data fidelity and prior terms. Recently, several works have considered a back-projection (BP) based fidelity term as an alternative to the common least squares (LS), and demonstrated excellent results for popular inverse problems. These works have also empirically shown that using the BP term, rather than the LS term, requires fewer iterations of optimization algorithms. In this paper, we examine the convergence rate of the projected gradient descent (PGD) algorithm for the BP objective. Our analysis allows to identify an inherent source for its faster convergence compared to using the LS objective, while making only mild assumptions. We also analyze the more general proximal gradient method under a relaxed contraction condition on the proximal map** of the prior. This analysis further highlights the advantage of BP when the linear measurement operator is badly conditioned. Numerical experiments with both $\ell_1$-norm and GAN-based priors corroborate our theoretical results.
△ Less
Submitted 8 August, 2021; v1 submitted 2 May, 2020;
originally announced May 2020.
-
Back-Projection based Fidelity Term for Ill-Posed Linear Inverse Problems
Authors:
Tom Tirer,
Raja Giryes
Abstract:
Ill-posed linear inverse problems appear in many image processing applications, such as deblurring, super-resolution and compressed sensing. Many restoration strategies involve minimizing a cost function, which is composed of fidelity and prior terms, balanced by a regularization parameter. While a vast amount of research has been focused on different prior models, the fidelity term is almost alwa…
▽ More
Ill-posed linear inverse problems appear in many image processing applications, such as deblurring, super-resolution and compressed sensing. Many restoration strategies involve minimizing a cost function, which is composed of fidelity and prior terms, balanced by a regularization parameter. While a vast amount of research has been focused on different prior models, the fidelity term is almost always chosen to be the least squares (LS) objective, that encourages fitting the linearly transformed optimization variable to the observations. In this paper, we examine a different fidelity term, which has been implicitly used by the recently proposed iterative denoising and backward projections (IDBP) framework. This term encourages agreement between the projection of the optimization variable onto the row space of the linear operator and the pseudo-inverse of the linear operator ("back-projection") applied on the observations. We analytically examine the difference between the two fidelity terms for Tikhonov regularization and identify cases (such as a badly conditioned linear operator) where the new term has an advantage over the standard LS one. Moreover, we demonstrate empirically that the behavior of the two induced cost functions for sophisticated convex and non-convex priors, such as total-variation, BM3D, and deep generative models, correlates with the obtained theoretical analysis.
△ Less
Submitted 24 February, 2020; v1 submitted 16 June, 2019;
originally announced June 2019.
-
Image Restoration by Iterative Denoising and Backward Projections
Authors:
Tom Tirer,
Raja Giryes
Abstract:
Inverse problems appear in many applications, such as image deblurring and inpainting. The common approach to address them is to design a specific algorithm for each problem. The Plug-and-Play (P&P) framework, which has been recently introduced, allows solving general inverse problems by leveraging the impressive capabilities of existing denoising algorithms. While this fresh strategy has found ma…
▽ More
Inverse problems appear in many applications, such as image deblurring and inpainting. The common approach to address them is to design a specific algorithm for each problem. The Plug-and-Play (P&P) framework, which has been recently introduced, allows solving general inverse problems by leveraging the impressive capabilities of existing denoising algorithms. While this fresh strategy has found many applications, a burdensome parameter tuning is often required in order to obtain high-quality results. In this work, we propose an alternative method for solving inverse problems using off-the-shelf denoisers, which requires less parameter tuning. First, we transform a typical cost function, composed of fidelity and prior terms, into a closely related, novel optimization problem. Then, we propose an efficient minimization scheme with a plug-and-play property, i.e., the prior term is handled solely by a denoising operation. Finally, we present an automatic tuning mechanism to set the method's parameters. We provide a theoretical analysis of the method, and empirically demonstrate its competitiveness with task-specific techniques and the P&P approach for image inpainting and deblurring.
△ Less
Submitted 10 October, 2018; v1 submitted 18 October, 2017;
originally announced October 2017.
-
Generalizing CoSaMP to Signals from a Union of Low Dimensional Linear Subspaces
Authors:
Tom Tirer,
Raja Giryes
Abstract:
The idea that signals reside in a union of low dimensional subspaces subsumes many low dimensional models that have been used extensively in the recent decade in many fields and applications. Until recently, the vast majority of works have studied each one of these models on its own. However, a recent approach suggests providing general theory for low dimensional models using their Gaussian mean w…
▽ More
The idea that signals reside in a union of low dimensional subspaces subsumes many low dimensional models that have been used extensively in the recent decade in many fields and applications. Until recently, the vast majority of works have studied each one of these models on its own. However, a recent approach suggests providing general theory for low dimensional models using their Gaussian mean width, which serves as a measure for the intrinsic low dimensionality of the data. In this work we use this novel approach to study a generalized version of the popular compressive sampling matching pursuit (CoSaMP) algorithm, and to provide general recovery guarantees for signals from a union of low dimensional linear subspaces, under the assumption that the measurement matrix is Gaussian. We discuss the implications of our results for specific models, and use the generalized algorithm as an inspiration for a new greedy method for signal reconstruction in a combined sparse-synthesis and cosparse-analysis model. We perform experiments that demonstrate the usefulness of the proposed strategy.
△ Less
Submitted 6 March, 2017;
originally announced March 2017.
