-
Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks
Authors:
Giovanni Luca Marchetti,
Christopher Hillar,
Danica Kragic,
Sophia Sanborn
Abstract:
In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features -- a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case t…
▽ More
In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features -- a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.
△ Less
Submitted 14 June, 2024; v1 submitted 13 December, 2023;
originally announced December 2023.
-
Bispectral Neural Networks
Authors:
Sophia Sanborn,
Christian Shewmake,
Bruno Olshausen,
Christopher Hillar
Abstract:
We present a neural network architecture, Bispectral Neural Networks (BNNs) for learning representations that are invariant to the actions of compact commutative groups on the space over which a signal is defined. The model incorporates the ansatz of the bispectrum, an analytically defined group invariant that is complete -- that is, it preserves all signal structure while removing only the variat…
▽ More
We present a neural network architecture, Bispectral Neural Networks (BNNs) for learning representations that are invariant to the actions of compact commutative groups on the space over which a signal is defined. The model incorporates the ansatz of the bispectrum, an analytically defined group invariant that is complete -- that is, it preserves all signal structure while removing only the variation due to group actions. Here, we demonstrate that BNNs are able to simultaneously learn groups, their irreducible representations, and corresponding equivariant and complete-invariant maps purely from the symmetries implicit in data. Further, we demonstrate that the completeness property endows these networks with strong invariance-based adversarial robustness. This work establishes Bispectral Neural Networks as a powerful computational primitive for robust invariant representation learning
△ Less
Submitted 19 May, 2023; v1 submitted 7 September, 2022;
originally announced September 2022.
-
Biologically Plausible Sequence Learning with Spiking Neural Networks
Authors:
Zuozhu Liu,
Thiparat Chotibut,
Christopher Hillar,
Shaowei Lin
Abstract:
Motivated by the celebrated discrete-time model of nervous activity outlined by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the McCulloch-Pitts network (MPN), for sequence learning in spiking neural networks. Our model has a local learning rule, such that the synaptic weight updates depend only on the information directly accessible by the synapse. By exploiting asymmetr…
▽ More
Motivated by the celebrated discrete-time model of nervous activity outlined by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the McCulloch-Pitts network (MPN), for sequence learning in spiking neural networks. Our model has a local learning rule, such that the synaptic weight updates depend only on the information directly accessible by the synapse. By exploiting asymmetry in the connections between binary neurons, we show that MPN can be trained to robustly memorize multiple spatiotemporal patterns of binary vectors, generalizing the ability of the symmetric Hopfield network to memorize static spatial patterns. In addition, we demonstrate that the model can efficiently learn sequences of binary pictures as well as generative models for experimental neural spike-train data. Our learning rule is consistent with spike-timing-dependent plasticity (STDP), thus providing a theoretical ground for the systematic design of biologically inspired networks with large and robust long-range sequence storage capacity.
△ Less
Submitted 25 November, 2019;
originally announced November 2019.
-
On the uniqueness and stability of dictionaries for sparse representation of noisy signals
Authors:
Charles J. Garfinkle,
Christopher J. Hillar
Abstract:
Learning optimal dictionaries for sparse coding has exposed characteristic sparse features of many natural signals. However, universal guarantees of the stability of such features in the presence of noise are lacking. Here, we provide very general conditions guaranteeing when dictionaries yielding the sparsest encodings are unique and stable with respect to measurement or modeling error. We demons…
▽ More
Learning optimal dictionaries for sparse coding has exposed characteristic sparse features of many natural signals. However, universal guarantees of the stability of such features in the presence of noise are lacking. Here, we provide very general conditions guaranteeing when dictionaries yielding the sparsest encodings are unique and stable with respect to measurement or modeling error. We demonstrate that some or all original dictionary elements are recoverable from noisy data even if the dictionary fails to satisfy the spark condition, its size is overestimated, or only a polynomial number of distinct sparse supports appear in the data. Importantly, we derive these guarantees without requiring any constraints on the recovered dictionary beyond a natural upper bound on its size. Our results also yield an effective procedure sufficient to affirm if a proposed solution to the dictionary learning problem is unique within bounds commensurate with the noise. We suggest applications to data analysis, engineering, and neuroscience and close with some remaining challenges left open by our work.
△ Less
Submitted 14 May, 2019; v1 submitted 22 June, 2016;
originally announced June 2016.
-
Randomization, Sums of Squares, and Faster Real Root Counting for Tetranomials and Beyond
Authors:
Osbert Bastani,
Christopher J. Hillar,
Dimitar Popov,
J. Maurice Rojas
Abstract:
Suppose f is a real univariate polynomial of degree D with exactly 4 monomial terms. We present an algorithm, with complexity polynomial in log D on average (relative to the stable log-uniform measure), for counting the number of real roots of f. The best previous algorithms had complexity super-linear in D. We also discuss connections to sums of squares and A-discriminants, including explicit obs…
▽ More
Suppose f is a real univariate polynomial of degree D with exactly 4 monomial terms. We present an algorithm, with complexity polynomial in log D on average (relative to the stable log-uniform measure), for counting the number of real roots of f. The best previous algorithms had complexity super-linear in D. We also discuss connections to sums of squares and A-discriminants, including explicit obstructions to expressing positive definite sparse polynomials as sums of squares of few sparse polynomials. Our key tool is the introduction of efficiently computable chamber cones, bounding regions in coefficient space where the number of real roots of f can be computed easily. Much of our theory extends to n-variate (n+3)-nomials.
△ Less
Submitted 13 January, 2011;
originally announced January 2011.
-
Most tensor problems are NP-hard
Authors:
Christopher Hillar,
Lek-Heng Lim
Abstract:
We prove that multilinear (tensor) analogues of many efficiently computable problems in numerical linear algebra are NP-hard. Our list here includes: determining the feasibility of a system of bilinear equations, deciding whether a 3-tensor possesses a given eigenvalue, singular value, or spectral norm; approximating an eigenvalue, eigenvector, singular vector, or the spectral norm; and determinin…
▽ More
We prove that multilinear (tensor) analogues of many efficiently computable problems in numerical linear algebra are NP-hard. Our list here includes: determining the feasibility of a system of bilinear equations, deciding whether a 3-tensor possesses a given eigenvalue, singular value, or spectral norm; approximating an eigenvalue, eigenvector, singular vector, or the spectral norm; and determining the rank or best rank-1 approximation of a 3-tensor. Furthermore, we show that restricting these problems to symmetric tensors does not alleviate their NP-hardness. We also explain how deciding nonnegative definiteness of a symmetric 4-tensor is NP-hard and how computing the combinatorial hyperdeterminant of a 4-tensor is NP-, #P-, and VNP-hard. We shall argue that our results provide another view of the boundary separating the computational tractability of linear/convex problems from the intractability of nonlinear/nonconvex ones.
△ Less
Submitted 30 June, 2013; v1 submitted 7 November, 2009;
originally announced November 2009.
