Expressivity of Neural Networks with Random Weights and Learned Biases

Abstract

Landmark universal function approximation results for neural networks with trained weights and biases provided impetus for the ubiquitous use of neural networks as learning models in Artificial Intelligence (AI) and neuroscience. Recent work has pushed the bounds of universal approximation by showing that arbitrary functions can similarly be learned by tuning smaller subsets of parameters, for example the output weights, within randomly initialized networks. Motivated by the fact that biases can be interpreted as biologically plausible mechanisms for adjusting unit outputs in neural networks, such as tonic inputs or activation thresholds, we investigate the expressivity of neural networks with random weights where only biases are optimized. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can be trained to perform multiple tasks by learning biases only. We further show that an equivalent result holds for recurrent neural networks predicting dynamical system trajectories. Our results are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on multi-task methods such as bias fine-tuning and unit masking.

1 Introduction

1.1 Context and Motivation

The universal approximation theorems Hornik et al. (1989); Funahashi (1989); Hornik (1991) of the late 1900s highlighted the expressivity of neural network models, i.e. their ability to approximate or express a broad class of functions through tuning of weights and biases, heralding the central role neural networks play in Machine Learning (ML) and neuroscience today. Since these foundational studies, a rich literature has explored the limits of the expressivity of neural networks by finding smaller parameter subsets whose tuning still results in the approximation of wide classes of functions or dynamics. Prior work has explored approximation capabilities of feed-forward (FFNs) and Recurrent Neural Networks (RNNs) where only the output weights are trained Rosenblatt et al. (1962); Rahimi and Recht (2008); Ding et al. (2014); Neufeld and Schmocker (2023); Jaeger and Haas (2004); Sussillo and Abbott (2009); Gonon et al. (2023); Hart et al. (2021), and deep FFNs where only subsets of parameters Rosenfeld and Tsotsos (2019), normalization parameters Burkholz (2023); Giannou et al. (2023), or binary masks–either over units or parameters–are trained Malach et al. (2020). Recently, a study has also explored the approximation abilities of transformers where only context is tuned Petrov et al. (2024). Here, motivated by recent insights from neuroscience, we initiate the investigation of expressivity in both FFNs and RNNs where only biases are trained, an avenue that remains largely unexplored.

Learned biases are of fundamental relevance to neuroscience and ML. In the latter domain, recent work has highlighted the optimization or the careful selection of bias vectors: fine-tuning of biases for multi-task learning Zaken et al. (2021) and the context tokens used for in-context learning Von Oswald et al. (2023) can all be viewed as methods of carefully setting the bias in a model where other connectivity parameters are fixed in order to perform a task. In neuroscience, single neurons employ diverse mechanisms to adjust their response to inputs, beyond synaptic plasticity. Importantly, some of these mechanisms can be construed as manipulating the firing onset of neurons which, in a firing rate model, is represented by the bias. Among these, we have shunting inhibition Holt and Koch (1997), threshold adaptation Azouz and Gray (2000), and a host of other mechanisms that participate in sha** the input-output transfer function of neurons (reviewed in Ferguson and Cardin (2020), see also Mazzucato et al. (2019)). Experimental evidence also suggests that bias-related signals play a role in learning Zhang and Linden (2003); Sehgal et al. (2013) but, despite this, most work modelling learning in neuroscience has focused on synapses.

If tuning the biases of a neural network will only span a reduced set of functions, or output dynamics, then this would solidify the role of synaptic plasticity as the critical component in biological and artificial learning. Conversely, if one can express arbitrary dynamics solely by changing biases, this would motivate deeper investigation of when and how non-synaptic plasticity mechanisms might shoulder some of the effort of learning. In this paper we address the question of the expressivity of bias learning. In a regime where all weight parameters are randomly initialized and frozen, and only hidden layer biases are optimized, we provide theoretical guarantees demonstrating that:

1. feed-forward neural networks with wide hidden layers are universal function approximators with high probability;

2. RNNs with wide hidden layers can arbitrarily approximate finite-time trajectories from smooth dynamical systems with high probability.

We provide empirical support for, and a deeper interrogation of, these theoretical findings with an array of numerical experiments.

1.2 Related Works

Neuroscience. Evidence from experimental and theoretical neuroscience shows that cortical circuits flexibly regulate their dynamical phases and toggle between different regimes solely by changing the distribution of input biases to a circuit. Bias-induced modulations can explain the context-dependent effects of expectation Mazzucato et al. (2019), movements Wyrick and Mazzucato (2021) and arousal Papadopoulos et al. (2024) on sensory processing across modalities and brain areas. Changes in the bias distribution modulate the gain of the input-output neuronal transfer function Mazzucato et al. (2019). Moreover, two bias-related biological mechanisms, neural firing threshold and network inputs, have also been noted for their ability to shape network dynamics: theoretical work shows that threshold heterogeneity can improve network capacity Gast et al. (2024, 2023), while a mix of theory and experimental evidence suggests a role for inputs from upstream brain areas in reconfiguring circuit dynamics on fast timescales Perich et al. (2018); Remington et al. (2018). Within the RNN reservoir computing approach to modelling in neuroscience, where recurrent weights are random and fixed, bias modulations can toggle between multiple phases (including fixed point, chaos, and multistable regimes) and, strikingly, enable RNN multi-tasking in the absence of any parameter optimization Ogawa et al. (2023). A repertoire of dynamical motifs can also be generated in RNN reservoirs with dynamic feedback loops Logiaco et al. (2021). However, in all these studies input biases or other parameters were either i.i.d. generated or set to specific values, but never learned. The role of plastic biases has not been investigated yet and will be the main focus of our efforts below.

Machine Learning. We mention primarily two bodies of literature in ML/AI that are related to this study. Many efforts have explored neural networks that are trained to quickly meta-learn new tasks via dynamics in activation space alone, without any adaptation of weights (see e.g. Feldkamp et al. (1997); Klos et al. (2020); Cotter and Conwell (1990, 1991)). Like our work, this thread of research proposes a mechanism by which a network might adapt to any new task without changing weights. However, prior work differs form the current study in that it requires an initial meta-training of all parameters in a network, weights included, before operating in the "fast learning" regime where network variables maintain context information that allow the networks to rapidly adapt to new tasks. In some cases these context variables can be thought of as biases Cotter and Conwell (1991). The closest ML contributions to our work are on the topic of masking–particularly the Strong Lottery Ticket Hypothesis (SLTH). This hypothesis conjectured that a given desired parameterization of a neural network can be found as a sub-network in a larger, appropriately initialized, network Ramanujan et al. (2020). SLTH is typically formulated with respect to weights, i.e. a subnetwork is defined by deleting weights from the full network. However, a few studies have investigated SLTH when subnetworks are constructed by deleting units Malach et al. (2020). While our study is different at face value for its focus on function approximation via bias optimization, rather than finding “lottery ticket" subnetworks, a key step in our analytic derivations relies on masking in a fashion analogous to proofs of the unit masking version of SLTH. Thus, our work also provides two results that may be of interest to the SLTH theory. First an alternative proof (complementing the first proof in Malach et al. (2020)) of the SLTH for units in single-hidden layer neural networks; second, a first proof of the SLTH over units for RNNs (see Section §2 for more details). Flavours of the lottery ticket hypothesis for RNNs has been explored empirically Yu et al. (2019); García-Arias et al. (2021); Schlake et al. (2022) but we have not encountered its proof, over weights or units, in the literature.

2 Theory Results

2.1 Feed-forward neural networks

This section studies the single-layer FFN, whose output is given by

$$y_{n}(x,\theta)=\sum_{i=1}^{n}A_{:i}\phi(B_{i:}x+b),$$ (1)

with $A\in\mathbb{R}^{l\times n}$, $B\in\mathbb{R}^{n\times d}$, $b\in\mathbb{R}^{n}$, and $\theta=\{B,A,b\}$. We shall investigate the approximation properties of this neural network when all the weights in $B$ and $A$ are fixed and sampled uniformly from the $n(l+d)$ dimensional centered hypercube, with half-edges of length $\gamma$, and only $b$ is tuned. We begin by outlining the activation function assumptions necessary for our theoretical results.

Definition 1.

The function $\phi$ is a suitable activation if, when employed in the neural network of Eq.1, it allows for universal approximation of the following kind: for any continuous $h:U\to\mathbb{R}^{l}$ and any $\epsilon>0$ $\exists n\in\mathbb{N}$ and parameters $\theta$ s.t. $\int_{U}||h(x)-y_{n}(x,\theta)||dx\leq\epsilon$, where $U\subset\mathbb{R}^{d}$ is compact and $||\cdot||$ is the $1$-norm and will be throughout the paper.

From the universal approximation theorems of 1993 Leshno et al. (1993); Hornik (1993), a sufficient condition for $\phi$ to be a suitable activation is that it is non-polynomial. In this paper we conceptualize universal approximation as the approximation of continuous functions on compact sets with respect to an $L^{1}$ functional norm, but we remark that the literature has also studied other conditions on $h$ (for example, continuity and measurability) and other forms of convergence (for example, with respect to the $L^{\infty}$ norm). For a review of the literature, see Pinkus (1999).

Definition 2.

A suitable activation $\phi$ is referred to as a $\gamma$-parameter bounding activation if it allows for universal approximation even when each individual parameter, e.g. an element of a weight matrix or bias vector, is bounded.

Proposition 1.

The ReLU and the Heaviside step function are $\gamma$-parameter bounding activations for any $\gamma>0$.

The proof is in Appendix §2.1 for completeness. A key subtlety of parameter-bounding is that it is a bound on individual, scalar, parameters. Thus, as a network grows in width the bias vector and weight matrix norms will still grow accordingly. This may be important, as research suggests band-limited parameters cannot universally approximate, at least for certain activation types Li et al. (2023). We leave it to future work to determine which other activations are parameter bounding.

We make one final definition:

Definition 3.

If $\phi$ is a $\gamma$-parameter bounding activation, is continuous, and if $\exists\tau\in\mathbb{R}$ such that for $x<\tau$ $\phi(x)=0$ then we say that $\phi$ is a $\gamma$-bias-learning activation.

Obviously the ReLU is a bias-learning activation. We leave the study of discontinuous functions like the Heaviside to future work. We conclude with the main result of this section. Define $p_{R}$ to be a uniform distribution on $[-R,+R]$, where $\gamma<R<\infty$.

Theorem 1.

Assume that $\phi$ is $\gamma$-bias learning and, for compact $U\subset\mathbb{R}^{d}$, $h:U\to\mathbb{R}^{l}$ is continuous. Then for any degree of accuracy $\epsilon>0$ and probability of error $\delta\in(0,1)$, there exists a hidden-layer width $m\in\mathbb{N}$ and bias vector $b\in\mathbb{R}^{m}$ such that, with a probability of $1-\delta$, a neural network given by Eq.1 with each individual weight sampled from $p_{R}$ approximates $h$ with error less than $\epsilon$.

Corollary 1.

Assume that $d=l$, i.e. the input and output space of $h$ has the same dimension. Then the results of Theorem 1 also hold for single-hidden-layer res-nets.

Proof Intuition: We provide intuition about the proof of Theorem 1 and its Corollary, whose details can be found in the Appendix. According to the universal approximation theorem, given a continuous function, we can find a one-hidden-layer network, $\mathcal{N}_{1}$, that is close to that function in the $L^{1}$ norm on the (compact) space of its inputs. If $\mathcal{N}_{1}$ has been constructed using $\gamma$-parameter bounding activation functions, then we know that each parameter will be on the interval $[-\gamma,\gamma]$. Next, we construct a second network, $\mathcal{N}_{2}$, to approximate $\mathcal{N}_{1}$ by randomly sampling each of its parameters, weight or bias, from $p_{R}$. For $\mathcal{N}_{2}$ to approximate $\mathcal{N}_{1}$, each parameter of $\mathcal{N}_{2}$ should fall within a tiny window of an analogous parameter in $\mathcal{N}_{1}$. This window must have half-length less than $\epsilon$ to yield the desired error bound. Without loss of generality we can assume $\epsilon<R-\gamma$. Then, if we sample parameters uniformly on $[-R,R]$ there will be a non-zero probability that a given parameter of $\mathcal{N}_{2}$ will end up within the tiny $\epsilon$-window centered at a corresponding parameter value in $\mathcal{N}_{1}$; because $\epsilon<R-\gamma$ we know that the $\epsilon$-window won’t stretch outside the distribution support. If we randomly sample a very large number of units to construct the hidden layer of $\mathcal{N}_{2}$ the probability of finding a subnetwork of $\mathcal{N}_{2}$ corresponding to $\mathcal{N}_{1}$ can be made arbitrarily close to $1$. If the activation function is bias-learning we can use biases to pick out this subnetwork by setting them appropriately smaller than the threshold given in Definition 3. We stress that this proof requires exceedingly massive hidden layer widths.

2.2 Recurrent neural networks

This section studies a discrete-time RNN given by

$$r_{t}=-\alpha r_{t-1}+\beta\phi(Wr_{t-1}+Bx_{t-1}+b),\quad y_{t}=Cr_{t},$$ (2)

where $r_{t}\in\mathbb{R}^{m}$ for all $0\leq t\leq T$ for some $T\in\mathbb{N}$, $\alpha$ and $\beta$ control the time scale of the dynamics, $W\in\mathbb{R}^{m\times m}$, $C\in\mathbb{R}^{l\times m}$, and $B$ and $b$ are as in the previous section. The parameters are now $\theta=\{W,B,C,b\}$. The time-dependent input $x_{t}$ belongs to a compact subset $U_{x}\subset\mathbb{R}^{d}$ for all $t$. Note that when $\alpha=0$, $\beta=1$ one gets the standard vanilla RNN formulation; alternatively, $\alpha$ and $\beta$ can be set to approximate continuous-time dynamics using Euler’s method.

We will approximate the following class of dynamical systems by learning only biases:

$$z_{t+1}=F(z_{t},x_{t}),\quad y_{t}=Qz_{t},\quad z_{0}\in U_{z},$$ (3)

where $t$ and $x_{t}$ are as defined for the RNN, $F:U_{z}\times U_{x}\to\mathbb{R}^{s}$ is continuous, and $Q\in\mathbb{R}^{l\times s}$. Because we build from the classic universal approximation results, we must be working with functions operating on compact sets. To guarantee that this will be the case we must make several more assumptions about the dynamical system. First, $U_{z}\subset\mathbb{R}^{s}$ is assumed to be a compact invariant set of the dynamical system: if the system is in $U_{z}$ it remains there for all $t$ for all inputs in $U_{x}$. Second, we assume that the dynamical system is well-defined on a slightly larger compact set $\tilde{U}_{z}\times U_{x}$, where $\tilde{U}_{z}=\{z_{0}+c_{0}:z_{0}\in U_{z},\>||c_{0}||<c\}$ for some $c>0$, with $\tilde{U}_{z}\supset U_{z}$. Letting $p_{R}$ be defined as in Section 2.1, the main result of this section is:

Theorem 2.

Consider the RNN in Eq.2 with $\phi$ a $\gamma$-bias learning activation, and input, output, and recurrent weight parameters for each hidden unit sampled from $p_{R}$. We can find a hidden-layer width, a bias vector, and a hidden-state initial condition for the RNN such that, with a probability that is arbitrarily close to $1$, the RNN approximates finite trajectories from the dynamical system defined in Eq.3 to below any positive, non-zero, error.

This proof is weaker than the proof given in the previous section in that it shows point-wise convergence, rather than $L^{1}$, convergence. We conjecture that this can be extended straightforwardly but we leave this result to future work.

Proof Intuition: The complete proof is in the Appendix; here we provide intuition. As in Theorem 1 the proof proceeds in two steps. First, the dynamical system is approximated by an RNN, $\mathcal{R}_{1}$, using universal approximation theory for RNNs (see e.g., Schäfer and Zimmermann (2006)). $\mathcal{R}_{1}$ is then approximated by a much wider, random RNN, $\mathcal{R}_{2}$, with parameters sampled from $p_{R}$. Analogous to Theorem 1, we show that one can find a sub-network of hidden units in $\mathcal{R}_{2}$ that approximates $\mathcal{R}_{1}$ for very large hidden widths of $\mathcal{R}_{2}$.

3 Numerical Results

3.1 Multi-task Learning with Bias-learning Feed-forward Networks

We first validated the theory by checking whether a single-hidden-layer, bias-learning FFN could perform digit recognition on the MNIST dataset Deng (2012) increasingly well as its hidden layer was widened. Initial weights were sampled from a uniform distribution on $[-0.1,0.1]$ and then frozen. As expected, the network learned the task and validation accuracy increased with layer width. The largest gains in performance occurred before 5000 units (Fig. 1A).

Figure 1: A. Validation accuracy on MNIST vs layer width. B. Validation accuracy on multiple image classification tasks for bias-only (blue) and fully-trained (orange) networks. In panels A and B, training was on 20 epochs and error bars on 5 runs are omitted because the standard errors are of order $10^{-3}$. C. Top: K-mean clustering of Task Variance (TV) reveals task-specific clusters. Bottom: Spearman correlation between TV and bias vectors (mean across neurons in each cluster). In this and all other figures Adam Kingma and Ba (2014) was used for optimization.

Intuitively, bias learning should allow a single random set of weights to be used to learn multiple tasks by simply optimizing task-specific bias vectors. We confirmed this intuition by training a single-hidden-layer FFN with 32000 hidden units on 7 different tasks: MNIST Deng (2012), KMNIST Clanuwat et al. (2018), Fashion MNIST Xiao et al. (2017), Ethiopic-MNIST, Vai-MNIST, and Osmanya-MNIST from Afro-MNIST Wu et al. (2020), and Kannada-MNIST Prabhu (2019). All tasks involved classifying 28$\times$28 grayscale images into 10 classes. The random weights were fixed across tasks while different biases were learned. We compared bias learning against a fully-trained neural network with the same size and architecture (Fig. 1B). We found that the bias-only network achieved similar performance to the fully-trained network on most tasks (only significantly worse on KMNIST). A crucial difference here is that the networks had matched size and architecture, so that the number of trainable parameters in the bias-only network (32 000 parameters) was several orders of magnitude smaller than in the fully-trained case (25 440 010 parameters). Notably, a different set of biases was learned for each task. We conclude that bias-only learning in FFNs is a viable avenue to perform multi-tasking with randomly initialized and fixed weights, but that it requires a much wider hidden layer than fully trained networks.

Next, we investigated the neural mechanism underlying bias learning of multiple tasks. We investigated the task-specific functional organization of the hidden units by estimating single-unit Task Variance (TV) Yang et al. (2019), defined as the variance of a hidden unit activation across the test set for each task. The TV provides a measure of the extent that a given hidden unit contributes to the given task. A unit with high TV in a particular task indicates that its responses vary across stimuli, suggesting that the unit is recruited for solving that task. A unit with high TV in one task and low TV for all other tasks is specialized to one particular task. We clustered the hidden units TV using K-means clustering ($K$ chosen by cross-validation) on the vectors of TVs for each unit and found that distinct functional clusters of hidden units emerged (Fig. 1C). Most units reflected strong task specialization, i.e., they were only used for specific tasks (ex: cluster 3 for KMNIST and cluster 10 for Osmanya). Others were used for many or all tasks (ex: clusters 1 and 8), although a smaller fraction of clusters exhibited such non-selective activation patterns. Overall, we conclude that multi-task bias-learning leads to the emergence of task-specific functional organization.

We finally explored the relationship between the bias of a hidden unit and its task variance (TV). If the neural networks are using biases to shut-off units, analogous to the intuition in our theory work (Section 2.1), then the network units that do not actively participate in a task should be more quiet due to a low bias value learned during training on that particular task. In other words, this intuition would suggest that units should exhibit a correlation between bias and TV, especially in task-specific clusters. In our experiments, all clusters did exhibit the statistical trend of a positive correlation between bias and TV, although to a varying degree across clusters (see numbers at the bottom of Fig. 1C).

3.2 Relationship Between Bias Learning and Masked Learning in FFNs

Figure 2: Comparing bias and masked learning on same weights. A. Learning curves for bias (orange) and mask (black) learning. Inset: we observed a trend where bias-learning achieved roughly $1$ percentage point higher accuracy on MNIST over mask-learning ($0.915\pm 0.0028\mathrm{SD}$ bias vs. $0.905\pm 0.0028\mathrm{SD}$ mask). B. Histograms of hidden unit variances, calculated over $10,000$ MNIST samples, for bias-trained (orange) and mask-trained (black). Histogram counts are log-scaled. C. Scatter plot of hidden unit variances from C but taking only units that are non-zero/not approximately-zero in mask/bias-trained networks; bias-trained on x-axis and mask-trained on y-axis. Plot A is mean$\pm 1$sd over 4 mask-trained/bias-trained nets. Plot B, and C are both one representative network trained with biases and with masks. Learned parameters were initialized to uniform on $[-0.1,0.1]$, weights to uniform on $[-\frac{1}{\sqrt{d}},\frac{1}{\sqrt{d}}]$; all other hyper-parameters can be found in the codebase.

As our theory shows that bias-learning networks can universally approximate simply by turning units off, we wished to test whether bias learning performs similarly to learning masks, and to what extent solutions learned by these approaches are different from each other. We compared training mask to bias learning on networks with the same random input/output weight matrices. For mask-training, we approximated binary masks with ‘soft’ sigmoid masks by optimizing the sigmoid parameters. The sigmoid was steepened over the course of training to approximately learn a mask, and at test time the slope of the sigmoid was made steep enough to effectively binarize the learned soft mask. We compared masks learned in this fashion with learned biases on single-hidden-layer ReLU networks with $10,000$ hidden units. We observed a trend of bias-training slightly improving upon mask-training (Fig.2A), an expected increase given the added flexibility of biases over masks. Further research is needed to determine if this trend is reliable across datasets and different network parametrizations, and whether there might be scenarios where one style of learning works better or worse.

Next, we compared the solutions found via bias and mask learning. We calculated the variance of each hidden unit across $10,000$ MNIST images, in both the bias and mask-trained paradigms, as a measure of the hidden-layer representation of MNIST. The histogram over hidden units of these variances is plotted for trained masks/biases (black/orange) for the same network weights (Fig.2B). The mask-trained networks found slightly sparser solutions (more $0$s, far left histogram bin) and performed the task with lower unit variance values (see middle/right of histogram) compared to the bias-trained networks. We also plotted the hidden unit variances for units in mask and bias-trained networks and computed their correlation (Fig.2C). The scatter plot and correlation were calculated after removing unit variances that were zero in the mask-trained net and sufficiently close to zero (with variance values less than the mean variance divided by 100) in the bias-trained network. Despite the differences observed in the histogram, the correlation coefficient was $0.461\pm 0.022$ (mean $\pm$ SD across $n=4$ networks), suggesting that there is some overlap in the way that these two methods solve MNIST.

3.3 Bias-learning of Dynamical Systems with Recurrent Neural Networks

3.3.1 Autonomous dynamical systems

Figure 3: Learning autonomous dynamical systems. A: Cosine generated by a bias-learning recurrent network with 200 hidden units (dashed orange) and its target (solid black). B. Eigenvalue spectra for the recurrent weight matrix (left) and the Jacobian at the start of training (right, grey squares) and mid-training (right, orange circles), when the network produced a decaying oscillation with period 23.75, close to the target period of 25. Units’ activity approached a fixed point with respect to which the Jacobian was computed. Later in training, the rightmost eigenvalues approached 1 in magnitude and their phases were such that the oscillation had period $\sim 25$. C. Van der Pol oscillator (target in solid black; with dynamics $\ddot{x}=\mu(1-x^{2})\dot{x}-x$, for $\mu=2$) generated by the bias-learning recurrent network with 675 hidden units for a recurrent gain of 1 (dashed orange; see panel D) and a gain of 0.9 (dashed dark orange). Output represents the position of the oscillator, rescaled to [-1, 1]. D. (Left) Sensitivity to distribution of recurrent weights. The fully-trained and bias-learning networks had the same number of learnable parameters (size of hidden layer for fully-trained network was 25). Initial recurrent weight matrix had elements sampled from $(g/\sqrt{m})\mathcal{N}(0,1)$, where $g$ is the gain (Gain recurrent init.). Error bars denote SEM for $n=10$. (Right) Schematics of the fully-trained (top) and bias-learning (bottom) autonomous RNNs. Colored links denote trained weights. In all panels, output matrix elements were $\sim\mathcal{N}(0,1/m^{2})$ and biases $b\sim\mathcal{U}(0,1)$.

We studied the expressivity of bias learning in RNNs trained to generate linear and nonlinear dynamical systems autonomously (i.e., with $x_{t}\equiv 0$ in Eq. 2). We found that RNNs with fixed and random Gaussian weights and trained biases were able to generate a simple cosine function (Fig. 3A). We then elucidated the mechanism underlying RNN bias learning by comparing the Jacobian matrix after learning with the random recurrent weight matrix (which was held fixed during learning). We found that although the random weight matrix maintained a fixed and circular eigenvalue distribution (Fig. 3B, left), learning the biases shaped the Jacobian matrix to develop complex conjugate pairs of large eigenvalues underlying the oscillations (Fig. 3B, right). Therefore, bias learning strongly relies on the ability to shape the “effective connectivity matrix”, i.e. the Jacobian, which involves the derivative of the activation and the recurrent weight matrix.

We next investigated whether bias learning relied on the statistics of the fixed recurrent weights. In light of Fig. 3B, we thus hypothesized that bias learning would be affected by changes in the weight distribution, because bias learning can only control the derivative. We initialized an i.i.d. Gaussian distributed weight matrix $W_{ij}\sim{\cal N}(0,g^{2}/N)$, where $g$ is referred to as its ‘gain’. We then trained bias-learning networks to generate a van der Pol oscillator (Fig. 3C). We found that bias learning required a large enough gain (at least $g=1$) and failed for $g<1$ (Fig. 3D). This was not purely due to a restricted dynamic range for the network activity since the network was able to reproduce the first peak of the oscillator and then flatlined (Fig. 3C). In contrast, fully-trained networks with the same number of training parameters (Fig. 3D) were not sensitive to the value of the gain at initialization. According to our theory, because ReLU is $\gamma$-parameter bounding, a bias-learning RNN should be able to produce the correct output even at low gain, but this could occur for a prohibitively large hidden layer. This result thus highlights that, when the hidden-layer size is fixed, the initial distribution of weights limits the capability of bias-learning networks.

3.3.2 Non-autonomous dynamical systems

Finally, we explored the capabilities of an RNN trained on a non-autonomous dynamical system, namely the $x$-dimension of the Lorenz system where the other dimensions are unobserved (see Fig.4 caption for more details). As in the autonomous network, only the biases of the input layer were trained and the weights were initialized randomly. However, here the networks also received an external input. The objective was formulated as follows: given a recurrent state and the value of a dynamical system at some time-point $t$, predict the future value at $t+\tau$, where this offset ($\tau=27$) was chosen to be the half-width at the half-max of the $x$ auto-correlation function. The numerical integration time step was $0.01$; each model had a hidden-layer width of 1024 units and was trained on different windows of 1080 (i.e., 40$\tau$) time points.

Figure 4: Learning non-autonomous dynamical systems. A. The outputs of the fully-trained (top) and bias-only (bottom) networks on a window of a Lorenz system unseen during training ($\dot{x}=\sigma(y-x),\dot{y}=x(\rho-z)-y,\dot{z}=xy-\beta z$). The system was generated using Euler’s method with a step size $\Delta t$ of 0.01 from an initialization at (0,1,0), with $\sigma=10$, $\rho=28$, and $\beta=\frac{8}{3}$. Standard deviation error bars were computed over 5 seeds, but are not visible. B. Generalization to sequences longer (4320 time-points) than those trained on. C. Output of the bias-only network diverges from the ground truth signal when using its own outputs as context (starting from the grey line).

We found that both the fully-trained and bias-only networks accurately predicted future points of the system, evidenced by a consistent $R^{2}$ metric of >0.99 ($n$=5) on a window of the generated Lorenz attractor held out from training (Fig. 4A). Furthermore, the models showed remarkable stability when predicting windows that were several times larger than the ones used during training, continuing to reconstruct the system without a notable change in accuracy (Fig. 4B). However, when the networks were fed their own previous predictions as input, their prediction accuracy decreased, demonstrating the devastating effect of small compounding deviations propagated through time (Fig. 4C).

4 Discussion

In this paper, we presented theoretical results demonstrating that feed-forward and recurrent neural networks with fixed random weights but learnable biases can approximate arbitrary functions with high probability. We showcased the expressivity of bias-learned networks in auto-regressive modelling, multi-task learning, dynamic pattern generation, and dynamical system forecasting. Finally, we interrogated the representations learned by bias learning via analysis of task specialization, comparison with mask-learning, and an eigenvalue analysis in recurrent networks. In what follows we discuss key insights, limitations, and future directions.

Our results highlight three key insights. First, certain activation functions enable our universal approximation results when weights are drawn from a uniform distribution on a hyper-cube with any strictly positive edge length. Characterizing functions that do or do not support this property–we speculate that non-differentiable points might be an important component–and the link between hyper-cube edge length and network scaling (Remark 1 on Lemma B.4 in Appendix) both seem worthy of future work. Second, in the context of multi-tasking, we showed that bias learning finds solutions that rely on task-selective clusters (Fig.1), similar to the fully trained case Yang et al. (2019). Third, we believe our finding that bias learning yields solutions that are different, though related to, mask learning (Fig.2) suggests that further investigation of our method might shed light on this and other non-synaptic learning approaches.

Our three main study limitations inspire directions for future research. First, the mathematical convergence results for dynamical systems are only point-wise over initial conditions, and are for finite-time trajectories. We believe that one might strengthen these results at least to $L^{p}$ convergence over trajectories, and that one might be able to break the finite-time limitations by studying convergence in stationary distribution. Second, our theory massively overestimates how wide the hidden layer of a bias-learned network should scale to reach a desired level of error (Remark 2 on Lemma B.4 in Appendix). This limitation might be addressed by drawing on scaling arguments from the SLTH literature (see e.g. Malach et al. (2020)). Third, a potential confounding factor in our comparison of bias and mask learning is that our mask learning approach used a learning schedule in the steepness parameter for the soft-masks. It is possible that the altered learning dynamics due to this scheduling contributed to mask and bias learning finding different solutions. Addressing this confound is an important direction for future work.

Future directions should focus on capturing greater biological detail, and better hidden layer scaling. Experimental Ferguson and Cardin (2020) and theoretical work Wyrick and Mazzucato (2021); Ogawa et al. (2023) showed that neural pathways that modulated biases, like firing threshold or tonic inputs, may effect other neuronal properties, like neuron input-output gain. As our proofs rely on masking, they demonstrate universal approximation not just for bias learning but for any learned mechanism that can mask neurons, possibly including gain modulation. Exploring the flexibility of paradigms where gain (Stroud et al. (2018)) and biases are learned in concert is an interesting direction to exploit this. Moreover, the observed distribution of synaptic weights in the brain is not uniform but long-tailed Song et al. (2005), and searching for weight distributions that improve expressivity and hidden-layer scaling is an exciting future direction. If hidden-layer scaling can be improved, bias-learning might open up options for temporal credit assignment that perform poorly due to the $N^{2}$ scaling of synapses. Alternatively, it would be fascinating if, regardless of whether learned parameters are weights or biases, one needs roughly the same number of parameters to achieve a given degree of task performance.

Combined with decades old results on adaptive behaviour in networks without weight changes Cotter and Conwell (1990), and related findings in ML (see in-context learning Brown et al. (2020)), we hope that this study will inspire more research on learning phenomena that transcend synaptic adaptation.

Code Availability

Code will be released upon publication.

Throughout the appendix the proofs are restated for ease of reference. We will always take $||\cdot||$ to be the $1$-norm unless stated otherwise.

Appendix A Random Neural Network Formulation

The proofs of this section revolve around masked, random, neural networks:

$$\tilde{r}_{m}^{\mathcal{M}}=-\alpha r_{0}+\mathcal{M}\odot\beta\phi(Wr_{0}+Bx+b),\quad\tilde{y}_{m}^{\mathcal{M}}=A\tilde{r}_{m}^{\mathcal{M}},$$ (A.1)

where $0\leq\alpha<\infty$, $0<\beta<\infty$, $m\in\mathbb{N}$, $r_{0},\>\tilde{r}_{m}^{\mathcal{M}}\in\mathbb{R}^{m}$, $x\in\mathbb{R}^{d}$, $\tilde{y}_{m}^{\mathcal{M}}\in\mathbb{R}^{l}$, $\mathcal{M}\in\{0,1\}^{m}$, and all other matrices and vectors have real elements with the dimensions required by the above definitions. We assume that $\phi$ is $\gamma$-parameter bounding and that each individual (scalar) parameter, be it weight or bias, is sampled randomly–before masking–from a uniform distribution on $[-\bar{\gamma},\bar{\gamma}]$ (note that here we are using $\bar{\gamma}$ where we used $R$ in the main text). In this way the parameters are random variables with compact support. If $\mathcal{M}=\mathbf{1}$ then we drop the superscript. To account for feed-forward neural networks we simply assume that $W$ is the zero matrix.

W.l.o.g. assume there are $n$ non-zero elements in $\mathcal{M}$. We construct $W^{\mathcal{M}}\in\mathbb{R}^{n\times n}$–the recurrent matrix restricted to participating (non-masked) hidden units–by beginning with $W$ and deleting the $i^{th}$ row and $i^{th}$ column of the matrix if $\mathcal{M}_{i}=0$. We construct $B^{\mathcal{M}}\in\mathbb{R}^{n\times d}$, $A^{\mathcal{M}}\in\mathbb{R}^{l\times n}$, and $b^{\mathcal{M}}\in\mathbb{R}^{n}$ by deleting the $i^{th}$ row of $B$, $A$, and $i^{th}$ element of $b$ if $\mathcal{M}_{i}=0$.

Consider the case where the $i^{th}$ element of $r_{0}$ is $0$ whenever $\mathcal{M}_{i}=0$. Then, regardless of whether Eq.A.1 represents a feed-forward network or the transition function for an RNN, the masked units will always be zero. We can thus simply track the $n$ units that correspond with $1$’s in $\mathcal{M}$ as the outputs, $y^{\mathcal{M}}$ will depend solely on these. We observe that the behaviour of these units can be described by the following network:

$$r_{m}^{\mathcal{M}}=-\alpha r_{0}+\beta\phi(W^{\mathcal{M}}r_{0}+B^{\mathcal{M}}x+b^{\mathcal{M}}),\quad y_{m}^{\mathcal{M}}=A^{\mathcal{M}}r_{m}^{\mathcal{M}}.$$ (A.2)

It is networks of the form of Eq.A.2 that will be the primary subject of study in what follows. Note that the ‘$\sim$’, over the $r$, is dropped to denote the fact that $r$ is a different vector on account of drop** the zero units. In the feed-forward case we use subscripts, as we have done above, to denote hidden layer width. Whenever we discuss RNNs or dynamical systems we will instead use the subscript to denote time.

Appendix B Proofs from Section 2.1

See 1

Proof.

We prove this solely for the ReLU, as the logic for the Heaviside is effectively the same. Let $\phi$ thus be a ReLU. First, observe the following useful property: for all $\alpha>0$ we have $\alpha\phi(x)=\phi(\alpha x)$. From this, consider the neural network of hidden layer width $n$ with ReLU activations, $y_{n}(\theta)$, and observe:

$$y_{n}(\theta)=\alpha^{2}\sum_{i=1}^{n}\frac{A_{:i}}{\alpha}\phi\bigg{(}\frac{B_{i:}}{\alpha}x+\frac{b_{i}}{\alpha}\bigg{)}=\alpha^{2}y_{n}\Big{(}\frac{\theta}{\alpha}\Big{)}.$$ (B.1)

Moreover, if $\alpha\in\mathbb{N}$ we have

$$y_{n}(\theta)=\sum_{i=1}^{\alpha^{2}n}\tilde{A}_{:i}\phi\big{(}\tilde{B}_{i:}x+\tilde{b}\big{)}=y_{\alpha^{2}n}(\tilde{\theta}),$$ (B.2)

where $\tilde{\theta}=[\frac{\theta_{1}}{\alpha},\dots,\frac{\theta_{n}}{\alpha},\dots\frac{\theta_{1}}{\alpha},\dots,\frac{\theta_{n}}{\alpha}]$ so that each element is simply a re-scaled and repeated version of the original parameters; we have $\alpha^{2}$ repeats for each term to replace the $\alpha^{2}$ factor in the LHS of Eq. B.1.

Now, given an arbitrary compact set $U\in\mathbb{R}^{d}$, continuous function $h:\mathbb{R}^{d}\to\mathbb{R}^{l}$, and $\varepsilon>0$, by the universal approximator theory (see e.g. Leshno et al. [1993], Hornik [1993]) we can find $n$ such that

$$\int_{U}||h(x)-y_{n}(x,\theta)||\leq\epsilon$$ (B.3)

holds. Because $n$ is finite we can bound every individual (scalar) parameter by $M$, for some sufficiently large $M$. Suppose we want the parameters to be bounded instead by $\gamma$ with $M>\gamma>0$. If we select $\alpha\in\mathbb{N}$ s.t. $\alpha>\frac{M}{\gamma}$ then we can find $y_{\alpha^{2}n}(x,\tilde{\theta})$ such that $y_{\alpha^{2}n}(x,\tilde{\theta})=y_{n}(x,\theta)$. Thus we have found a parameter-bounding ReLU neural network satisfying Eq.B.3, completing the proof.

∎

Remark: The intuition behind this result, for the ReLU, is credited to a reply to the Universal Approximation Theorem with Bounded Parameters question on Mathematics Stack Exchange.

The following lemma constitutes the core of Theorem 1. It shows that one can achieve universal approximation, in the sense needed for the theorem, using masking. The theorem then follows by manipulating biases to achieve masking. As mentioned in the main text, the following lemma is very closely related to past results on the SLTH over units for MLPs with one hidden layer. Specifically, we believe that one should be able to prove an analogue to our theorem by combining Theorem 3.2 in Malach et al. [2020] with Theorem 1 of Rahimi and Recht [2008] (or a similar result on learning with random networks). We leave the details of this to future studies.

Lemma 1.

Let $h:U\to\mathbb{R}^{l}$ be a continuous function on compact support $U\subset\mathbb{R}^{d}$. Then for any $\epsilon>0,\delta\in(0,1)$, we can find a layer width $m\in\mathbb{N}$ such that with probability at least $1-\delta$ $\exists\mathcal{M}\in\{0,1\}^{m}$ satisfying the following:

$$\int_{U}||h(x)-y_{m}^{\mathcal{M}}(x)||dx\leq\epsilon.$$ (B.4)

Proof.

First, we find a neural network with parameters that approximate the desired function $h$. Given the assumptions on $\phi$, we can use Proposition D.1 to find $n$ and parameters $\theta^{*}=\{A^{*},B^{*},b^{*}\}$ such that

$$\int_{U}||h(x)-y_{n}(x,\theta^{*})||dx\leq\frac{\epsilon}{2},$$ (B.5)

because $U$ is compact and $h$ is continuous.

We make a brief comment about the domain of a given activation function in $y_{n}$. $\phi$ will be operating on a compact domain $\{ux+b:u\in[-\bar{\gamma},\bar{\gamma}]^{d},x\in U,b\in[-\bar{\gamma},\bar{\gamma}]\}$, as a consequence of the compactness of the support of the parameters, and of the assumed compactness of $U$. By its continuity, $\phi$ is Lipschitz and bounded on this domain. We label Lipschitz constant and bound $K_{\phi}$ and $M_{\phi}$ respectively. We further define $M_{x}$ to be the bound for $x$ on $U$, and $|U|$ to be the value $\int_{U}dx$, which is finite by the boundedness of the domain.

Next, we construct a masked random network that approximates $y_{n}$ with high probability. By Lemma 3, we can find a random feed-forward neural network of hidden layer width $m$ such that a mask, $\mathcal{M}$, exists satisfying $|\theta^{*}_{i}-\theta_{i}^{\mathcal{M}}|<\varepsilon$ for some arbitrarily $\varepsilon>0$. In particular, we can choose $\varepsilon$ as:

$$|\theta^{*}_{i}-\theta_{i}^{\mathcal{M}}|<\varepsilon=\frac{\epsilon}{2|U|\max\big{(}nl(K_{\phi}\bar{\gamma}[1+M_{x}]+M_{\phi}),1\big{)}}$$ (B.6)

for all $i$ with probability at least $1-\delta$. If we are in the regime of probability $1-\delta$ where the mask satisfying the above error bound exists then we get

$$||y_{m}^{\mathcal{M}}(x)-y_{n}(x)||\leq nl[K_{\phi}\bar{\gamma}(1+M_{x})+M_{\phi}]\varepsilon\leq\frac{\epsilon}{2|U|},$$ (B.7)

where, in addition to Eq.B.6, we use the assumptions on $\phi$ and $U$ stated before the start of the proof and repeated application of the triangle inequality (see Lemma 4 for the derivation of the above bound).

Integrating the error over the domain and using Eq.B.5 and Eq.B.7 gives

$$\int_{U}||h(x)-y_{m}^{\mathcal{M}}(x)||dx\leq\int_{U}||h(x)-y_{n}(x)||dx+\int_{U}||y_{m}^{\mathcal{M}}(x)-y_{n}(x)||dx\leq\epsilon,$$ (B.8)

with probability $1-\delta$.

∎

See 1

Proof.

Observe that, once we have choosen an $m$ satisfying the desiderata of Lemma B.4, because $\phi$ is assumed to be $\gamma$-bias-learning, $m$ is some finite value and all variables that make up the input of $\phi$ are bounded, we can implement the mask by setting $b_{i}$ to be very negative for every $i$ such that $\mathcal{M}_{i}=0$. For every $b_{i}$ such that $\mathcal{M}=1$ we simply leave $b_{i}$ at its original randomly chosen value. ∎

Corollary 2.

Assume $d=l$, that is, the output and input spaces are the same. Then the results of Lemma B.4 and Theorem 1 also hold for res-nets; that is, networks whose output is of the form $x+y_{m}^{\mathcal{M}}(x)$.

Proof.

This follows by observing that $h(x)+x$ is also a continuous function and then replacing $h(x)$ with $h(x)+x$ in Eq.B.4 and rearranging. ∎

Remark: While the error can be made arbitrarily small, the limit of zero error itself is undefined. This is because our proof relies on first approximating the given smooth function with a neural network with all parameters tuned and then approximating this second network using bias-learning to pick-out a matching sub-network from a large random reservoir; the probability of perfectly matching the fully tuned network with the bias-learned network is zero. This could be addressed by using an integral representation for continuous functions instead of directly using a finite-width neural network to approximate the given function (see e.g. Rahimi and Recht [2008], Li et al. [2023]). As one will see below, this remark also applies to the recurrent neural network result.

Appendix C Proof from Section 2.2

Analogous to the section containing the feed-forward proofs, we first state and prove a lemma which comprises the core of the proof for recurrent neural networks. This lemma shows that one can achieve universal approximation with high probability using masking in a randomly initialized RNN, and in this way provides a proof of the SLTH over units for RNNs. The proof of the main theorem in this section then follows quite straightforwardly.

Lemma 2.

Consider a discrete time, partially observed dynamical system of the form of, and satisfying the same conditions as, the one in Eq.3. Let $0<T<\infty$, initial condition $z_{0}\in U_{z}$, input $x_{t}\in U_{x}$ $\forall t$, $\epsilon>0$ and $\delta\in(0,1)$. Then we can find an RNN initial condition $r_{0}$ and a layer width $m\in\mathbb{N}$ such that with probability at least $1-\delta$ $\exists\mathcal{M}\in\{0,1\}^{m}$ satisfying the following:

$$\sum_{t=1}^{T}||y_{t}-y_{t}^{\mathcal{M}}||<\epsilon.$$ (C.1)

Proof.

In what follows W.L.O.G. we assume $\epsilon<1$. It is well known that we can arbitrarily approximate this dynamical system with an RNN Schäfer and Zimmermann [2006]; we provide a simple proof of this in Proposition 3. In particular, for arbitrary $1>\epsilon>0$ we can find an RNN of the form in Eq.2, with hidden layer width $n\in\mathbb{N}$ and output $\hat{y}$, satisfying:

$$\sum_{t=1}^{T}||\hat{y}_{t}-y_{t}||<\frac{\epsilon}{2},$$ (C.2)

for any initial condition of the dynamical system selected within the invariant set $U_{z}$. We note that this is a point-wise convergence. It can be shown (see Prop.3) that the hidden states of this RNN remain on a compact set, $U_{r}$, when approximating finite time trajectories of the original dynamical system with initial conditions in $U_{z}$. Given this compactness, we can then show that the following set

$$\tilde{U}=\{ u(r+r^{\prime})+vx+b:u\in[-\bar{\gamma},\bar{\gamma}]^{n},v\in[-\bar{\gamma},\bar{\gamma}]^{d},$$ (C.3)

$$r\in U_{r},x\in U,b\in[-\bar{\gamma},\bar{\gamma}],r^{\prime}\in\mathbb{R}^{n},||r^{\prime}||\leq 1\},$$ (C.4)

is itself compact by the compactness of the sets from which it is formed. It will turn out that this set will contain the arguments of $\phi$ that appear in the proof. By its own compactness and the assumptions on $\phi$ we observe that $\phi$ has Lipschitz constant $K_{\phi}$. The size of $U_{r}$–and thus $\tilde{U}$–and $K_{\phi}$ will, in general, depend upon the first approximating neural network. By the compactness of $U$ and $U_{r}$, we can bound $x$ and $r$ on these sets to get bounds $R_{x}$ and $R_{r}$ respectively.

Let the parameters of the above-defined approximating RNN be given by $\theta^{*}=\{A^{*},W^{*},B^{*},b^{*}\}$. Then by Lemma 3 we can find a random RNN of hidden width $m$ and with parameters $\theta$ such that a mask, $\mathcal{M}$, exists satisfying

$$|\theta^{*}_{i}-\theta_{i}^{\mathcal{M}}|<\varepsilon=\frac{\epsilon}{2lT\big{[}\max(\bar{\gamma},1)\max\big{(}n\beta K_{\phi}\tilde{R}\sum_{t=0}^{T-1}(\alpha+\bar{\gamma}\beta K_{\phi}n)^{t},1\big{)}+R_{r}\big{]}},$$ (C.5)

for all $i$ with probability at least $1-\delta$, where $\tilde{R}=R_{x}+R_{r}+1$. One can show using induction (see Lemma 5 for the derivation) that if $|\theta^{*}_{i}-\theta_{i}^{\mathcal{M}}|<\varepsilon$ for all $i$ we get

$$\sum_{t=1}^{T}||y_{t}^{\mathcal{M}}-\hat{y}_{t}||<lT\big{[}\bar{\gamma}n\beta K_{\phi}\tilde{R}\sum_{t=0}^{T-1}(\alpha+\bar{\gamma}\beta K_{\phi}n)^{t}+R_{r}\big{]}\varepsilon.$$ (C.6)

The triangle inequality on $||y_{t}^{\mathcal{M}}-\hat{y}_{t}+\hat{y}_{t}-y_{t}||$, along with Equations C.2, C.5, and C.6 completes the proof.

∎

See 2

Proof.

This follows directly from Theorem 2, by observing that one can replace the mask by simply setting biases to some sufficiently low value. ∎

Appendix D Supplementary Lemmas

The following result is well known in the literature; see e.g. Proposition 1 of Leshno et al. [1993].

Proposition 2.

For any $\epsilon>0$ $\exists n\in\mathbb{N}$ s.t.

$$\int_{U}||h(x)-y_{n}(x,\theta^{*})||dx\leq\epsilon$$ (D.1)

Corollary 3.

The above holds if we restrict the output weight matrix of the neural network to have rank equal to the output dimension.

Proof.

This is because the set of full rank matrices is dense in $\mathbb{R}^{m\times n}$ for $m,n\in\mathbb{N}$. ∎

Consider matrices $W^{*}\in\mathbb{R}^{n\times n}$, $B^{*}\in\mathbb{R}^{n\times d}$, $A^{*}\in\mathbb{R}^{l\times n}$, and vector $b^{*}\in\mathbb{R}^{n}$. We can vectorize and concatenate their elements into the single long vector $\theta\in\mathbb{R}^{\pi}$, where $\pi=n(n+d+l+1)$. Assume that $|\theta_{i}^{*}|<\gamma$ for all $i$.

Next, construct $W\in\mathbb{R}^{m\times m}$, $B\in\mathbb{R}^{m\times d}$, $A\in\mathbb{R}^{l\times m}$, and vector $b\in\mathbb{R}^{m}$, by sampling each element randomly from a uniform distribution on $[-\bar{\gamma},\bar{\gamma}]$ where $\bar{\gamma}=\gamma+\Delta\gamma$ for $\Delta\gamma>0$. We analogously group these into a single vector, $\theta\in\mathbb{R}^{m(m+d+l+1)}$ Observe that for each $\mathcal{M}\in\{0,1\}^{m}$, we can construct sub-matrices of $W$, $B$, $A$, and sub-vector of $b$ by deleting column and row pairs in $W$, rows in $B$, columns in $A$, and elements of $b$ whose indices correspond to $i\in\{1,\dots,m\}$ such that $\mathcal{M}_{i}=0$. For a given $\mathcal{M}$, we define $\theta^{\mathcal{M}}$ to be the vector constructed by flattening and concatenating these sub-matrices and vector. We then have the following lemma:

Lemma 3.

For $\theta^{*}$, defined above, and arbitrary $\varepsilon>0$, $\delta\in(0,1)$, we can find $m>n$ such that with probability at least $1-\delta$ $\exists\mathcal{M}\in\{0,1\}^{m}$ with only $n$ non-zero elements such that $|\theta_{i}^{*}-\theta_{i}^{\mathcal{M}}|<\varepsilon$ for all $i\in\{1,\dots,\pi\}$. In particular, any $m\geq\frac{n\log\delta}{\log[1-(\frac{\epsilon}{\bar{\gamma}})^{\pi}]}$ will satisfy the result, where $\epsilon=\min(\varepsilon,\Delta\gamma)$.

Proof.

In what follows we set $\epsilon=\min(\varepsilon,\Delta\gamma)$. This simplifies the below probability bound that we derive because it means the probability of falling within an $\epsilon$ window of a desired parameter will not change, even if the desired parameter is very close to its bound, $\pm\gamma$. We will refer to the event that the desiderata of the lemma are satisfied for $\epsilon$, rather than $\varepsilon$, as $A_{1}$; that is: $\exists\mathcal{M}\in\{0,1\}^{m}$ with only $n$ non-zero elements such that $|\theta_{i}^{*}-\theta_{i}^{\mathcal{M}}|<\epsilon$ for all $i\in\{1,\dots,n\}$. The event that the desiderata are not satisfied is $A_{1}^{c}$.

Assume that $m^{\star}=kn$ for some $k\in\mathbb{N}^{+}$. Consider the ‘block’ mask $\mathcal{M}^{k_{1}}$ s.t. $\mathcal{M}_{i}^{k_{1}}=1$ only for $i\in\{(k_{1}-1)n+1,\dots,k_{1}n\}$, with $0<k_{1}\leq k$. Note that the $n$ elements selected by these block masks are non-overlap** for two different $k_{1}$. Let event $A_{2}$ be the event that there is a block mask that occurs satisfying the desiderata of the lemma with error $\epsilon$. Clearly $A_{2}\subset A_{1}\implies A_{1}^{c}\subset A_{2}^{c}\implies P(A_{1}^{c})\leq P(A_{2}^{c})$. $A_{2}^{c}$ is the probability that there is no block mask satisfying the desiderata. Observe that

$$P(A_{2}^{c}) =P\bigg{[}\bigcap_{k_{1}=1}^{k}\{k_{1}^{th}\>\mathrm{block}\>\mathrm{mask}\>\mathrm{doesn^{\prime}t}\>\mathrm{work}\}\bigg{]}=\prod_{k_{1}=1}^{k}P(\{k_{1}^{th}\>\mathrm{block}\>\mathrm{mask}\>\mathrm{doesn^{\prime}t}\>\mathrm{work}\})$$

$$=\prod_{k_{1}=1}^{k}1-P(\{k_{1}^{th}\>\mathrm{block}\>\mathrm{mask}\>\mathrm{works}\})=\bigg{[}1-\Big{(}\frac{\epsilon}{\bar{\gamma}}\Big{)}^{\pi}\bigg{]}^{\frac{m^{\star}}{n}},$$ (D.2)

which follows from the fact that the elements of the matrices are independently sampled and the elements corresponding to sub-matrices selected by a given block mask are independent of those associated with another block mask. By making $m^{\star}$ very large we can make $P(A_{2}^{c})$ arbitrarily small. Because $P(A_{1}^{c})\leq P(A_{2}^{c})$–and the desiderata of the lemma with error $\epsilon$ are not satisfied solely on $A_{1}^{c}$–the result follows by selecting $m^{\star}=m$ such that $P(A_{2}^{c})\leq\delta$. We thus see that the probability of finding a sufficient mask occurs with probability at least $1-\delta$. Lastly, because we have found a mask that satisfies per-parameter error $\epsilon$, and because $\epsilon\leq\varepsilon$, we have proved the lemma. ∎

Remark 1: We note that, in Eq.D.2, $n$ will likely also depend implicitly on $\gamma$. If $\gamma$ is very small then we will need to stack many ReLUs on top of each other to attain a large enough dynamic range to approximate the desired function (see §2.1), leading to a larger number of units. Conversely, if $\gamma$ is very large we will need to sample a large number of units before we get parameters appropriately close to the desired subnetwork configuration. This suggests the existence of some sweet spot in the value $\gamma$, which we leave for future work to explore.

Remark 2: We observe that this bound appears to be very weak. For example, if one wished to use it to find a masked network to match an MLP with input, hidden, and output dimensions of only $1$, $3$, and $1$ respectively, with a per-parameter error of $\epsilon=0.05$ an error probability of $\delta=0.1$, and with $\gamma=0.1$, this bound would suggest we need a hidden layer of $m\geq 8.34\times 10^{12}$ neurons in the bias learning network. In light of the numerical experiments, it is clear that while the math here provides proofs of existence for bias learning it does not say anything useful about the hidden layer width scaling required.

For the following proposition we consider the discrete time dynamical system that we wish to approximate to be as in Eq.3.

Proposition 3.

For finite $0<T<\infty$, any initial condition $z_{0}\in U_{z}$, input $x_{t}\in U_{x}$ $\forall t$, $\epsilon>0$, and any $|\alpha|<\infty,\beta>0$ we can find an RNN of the style of Eq. 2 of hidden width $n\in\mathbb{N}$ and an initial value $r_{0}$ for the RNN such that:

$$\sum_{t=1}^{T}||\hat{y}_{t}-y_{t}||<\epsilon$$ (D.3)

where $\hat{y}_{t}$ is the output of the RNN and $y_{t}$ is that of the dynamical system. Moreover, when approximating the dynamical system in this way the RNN hidden states will remain in a compact set which we denote $U_{r}$.

The main portion of this result is well known, see e.g. Schäfer and Zimmermann [2006]. For completeness, we provide an example proof below.

Proof.

In what follows, W.L.O.G we will assume that the error is smaller than $c$. We want to approximate the dynamical system:

$$z_{t+1}=F(z_{t},x_{t}),\quad y_{t}=Cz_{t},\quad z_{0}\in U_{z},$$ (D.4)

defined on set $\tilde{U}_{z}=\{z_{0}+c_{0}:z_{0}\in U_{z},\>||c_{0}||<c\}$, where $U_{z}$ is an invariant set (see §2.2).

We define the set:

$$U_{zx}=\{[z+c_{0}\>\>x]:z\in U_{z},x\in U,||c_{0}||<c\}.$$ (D.5)

Importantly, this set is compact given the compactness assumptions on $U$ and $U_{z}$. Also note that, since $F$ is assumed continuous, it will be $K_{F}$-Lipschitz on this compact set for some constant $K_{F}$. We can thus use the corollary to Proposition D.1 to find a neural network of hidden dimension $n\in\mathbb{N}$ that approximates $F$ with a maximum-rank output matrix, $A$. We write this neural network:

$$\hat{z}=-\alpha z+\beta A\phi(Wz+Bx+b)=\hat{F}(z,x),$$ (D.6)

assuming $z\in U_{z}$ and $x\in U$, with $A\in\mathbb{R}^{s\times n}$, $W\in\mathbb{R}^{n\times s}$, $B\in\mathbb{R}^{n\times d}$, and $b\in\mathbb{R}^{n}$. In particular, we can find arbitrary $\epsilon$ with $0<\epsilon<c$ such that:

$$||\hat{F}(z,x)-F(z,x)||<\varepsilon=\frac{\epsilon}{T\max(R_{C}\sum_{t=0}^{T-1}K^{t}_{F},1)},$$ (D.7)

where $R_{C}=||C||$. Fix $T\geq 1$. To prove that we can approximate the underlying dynamical system, we use induction starting at time $t=1$. The base case will be

$$||\hat{z}_{1}-z_{1}||=||\hat{F}(z_{0},x_{0})-F(z_{0},x_{0})||\leq\varepsilon,$$ (D.8)

by our choice of $n$ and initial condition, and that $[z_{0},x_{0}]\in U_{zx}$. Importantly, this implies also that $||\hat{z}_{1}-z_{1}||<\varepsilon$. Because $\varepsilon<c$ this means that $[\hat{z_{1}}\>\>x_{1}]^{\top}\in U_{zx}$.

For $t=1$, $\varepsilon=\sum_{t^{\prime}=0}^{t-1}K_{F}^{t^{\prime}}\varepsilon$. We thus make the induction hypothesis that $||\hat{z}_{t}-z_{t}||<\sum_{t^{\prime}=0}^{t-1}K_{F}^{t^{\prime}}\varepsilon$ and that $[\hat{z}_{t}\>\>x_{t}]^{\top}\in U_{zx}$. If $T=1$ we are finished. If $T>1$ we assume $1<t<T$ and use this hypothesis to prove the induction step:

$$||\hat{z}_{t+1}-z_{t+1}|| \leq||\hat{F}(\hat{z}_{t},x_{t})-F(\hat{z}_{t},x_{t})||+||F(\hat{z}_{t},x_{t})-F(z_{t},x_{t})||$$ (D.9)

$$\leq\varepsilon+K_{F}||\hat{z}_{t}-z_{t}||=\varepsilon\sum_{t^{\prime}=0}^{t}K_{F}^{t^{\prime}}<\frac{c}{T}.$$ (D.10)

Because $\frac{c}{T}\leq c$, $[\hat{z}_{t+1}\>\>x_{t+1}]^{\top}\in U_{zx}$. Then

$$\sum_{t=1}^{T}||\hat{y}_{t}-y_{t}||\leq R_{C}T\varepsilon\sum_{t=0}^{T}K_{F}^{t}\leq\epsilon.$$ (D.11)

While we have approximated the dynamical system it is not yet in the standard rate-style RNN form. However, we can obtain the rate form by changing from tracking $\hat{z}$ to a different dynamical variable: $r_{t}\in\mathbb{R}^{n}$. Because $A$ is assumed to be maximum rank, we can find a minimal norm $r_{0}\in\mathbb{R}^{n}$ such that $z_{0}=Ar_{0}$, and $||r_{0}||\leq r^{\prime}$ for any $r^{\prime}$ that also satisfies $z_{0}=Ar^{\prime}$. Take this as the initial value for an RNN with dynamics:

$$r_{t+1}=-\alpha r_{t}+\phi(WAr_{t}+Bx_{t}+b)$$ (D.12)

It is easy to see that $\hat{z}_{t}=Ar_{t}$ for all $0\leq t\leq T$. It follows that $\hat{y}_{t}=CAr_{t}\>\forall t$. Thus, this RNN approximates the original partially observed dynamical system.

Lastly, we explain how the hidden states of the RNN will remain in a compact set. Each $r_{t}=\phi(W\hat{z}_{t-1}+Bx_{t-1}+b)\equiv\Phi(z_{0},x_{0},\dots,x_{t-1})$ for all $1\leq t\leq T$ and any choice of $\hat{z}_{0}=z_{0}\in U_{z}$. $\Phi$ is continuous by the continuity of the activation functions and the boundedness of the parameters. Thus, the set $U_{r}^{(1)}=\{r:r=\Phi(z_{0},x_{0},\dots,x_{t-1}),z_{0}\in U_{z},x_{s}\in U_{x}\>\forall\>0\leq s<t\}$ contains all $r_{t}$ and is compact, as the arguments of $\Phi$ are taken from compact sets.

It remains to treat the initial values of the RNN, $r_{0}$. These are defined implicitly as $Ar_{0}=z_{0},z_{0}\in U_{z}$. We will show that we can span all $z_{0}\in U_{z}$ by selecting $r_{0}$ from a compact set. By assumption $A$ is full rank so $\mathrm{col}(A)=\mathbb{R}^{s}$ and, thus, we can find a subset of the columns of $A$ that form a basis for $\mathbb{R}^{s}$. Let these columns be $\{A_{:i}\}_{i\in\nu}$. Construct a matrix $A_{\nu}$ by deleting the columns whose indices are not contained in $\nu$. Because the columns of $A_{\nu}$ are a basis the matrix is invertible. Define $U_{r}^{(0)}=\{r:r_{i}=\tilde{r}_{i}\>\mathrm{if}\>i\in\nu,r_{i}=0\>\mathrm{if}\>i\not\in\nu,\tilde{r}=A_{\nu}^{-1}z_{0},z_{0}\in U_{z}\}$. For each $z_{0}$ in $U_{z}$ we have an initial condition $r_{0}\in U_{r}^{(0)}$ and, moreover, $r_{0}$ is a continuous map** of $z_{0}$ (via $A_{\nu}^{-1}$). Since $z_{0}$ is taken from a compact set $U_{r}^{(0)}$ must also be compact.

Finally, define $U_{r}=U_{r}^{(0)}\cup U_{r}^{(1)}$. From the above this contains all hidden states of the RNN for the finite time trajectories of interest from any initial condition. As a union of two compact sets it is also compact. We will use $U_{r}$ in Lemma 2. We note that the size of the set will likely depend, via the dependency on $A$, on the function being approximated and on the neural network hidden layer size $n$.

∎

Lemma 4.

Eq.B.7 holds under the assumptions of Lemma 1.

Proof.

This follows by adding and subtracting $A_{ij}^{\mathcal{M}}\phi(B_{j:}x+b_{j})$ inside the summations in the norm and the expanded neural network expression, and then successively bounding things using the assumptions on the domain and the activation function:

$$||y_{m}^{\mathcal{M}}-y_{n}|| \leq\sum_{i=1}^{l}\sum_{j=1}^{n}\Big{|}A_{ij}^{\mathcal{M}}\big{[}\phi(B_{j:}^{\mathcal{M}}x+b_{j}^{\mathcal{M}})-\phi(B_{j:}x+b_{j})\big{]}-(A_{ij}-A_{ij}^{\mathcal{M}})\phi(B_{j:}x+b_{j})\Big{|}$$ (D.13)

$$\leq\sum_{i=1}^{l}\sum_{j=1}^{n}\Big{[}\bar{\gamma}K_{\phi}\big{|}(B_{j:}^{\mathcal{M}}-B_{j:})x+b_{j}^{\mathcal{M}}-b_{j}\big{|}+\varepsilon M_{\phi}\Big{]}$$ (D.14)

$$\leq nl\big{[}\bar{\gamma}K_{\phi}(\varepsilon M_{x}+\varepsilon)+\varepsilon M_{\phi}\big{]}=nl\big{[}\bar{\gamma}K_{\phi}(M_{x}+1)+M_{\phi}\big{]}\varepsilon.$$ (D.15)

∎

Lemma 5.

Eq.C.6 holds under the assumptions of Lemma 2.

Proof.

We have:

$$||r_{0}^{\mathcal{M}}-r_{0}|| =0$$ (D.16)

$$||r_{1}^{\mathcal{M}}-r_{1}|| \leq\alpha||r_{0}^{\mathcal{M}}-r_{0}||+\beta K_{\phi}\big{[}\bar{\gamma}n||r_{0}^{\mathcal{M}}-r_{0}||+\varepsilon n(R_{r}+R_{x}+1)\big{]}$$ (D.17)

$$=\beta K_{\phi}\bar{\gamma}n\tilde{R}\varepsilon\leq 1,$$ (D.18)

where we used that $r_{0}=r_{0}^{\mathcal{M}}$ and thus both are contained in $U_{r}$ so that the arguments of $\phi$ are contained in $\tilde{U}$, allowing us to use the Lipschitz result and constant $K_{\phi}$. We note that the final inequality follows from the choice of $\varepsilon$ and means that, since $r_{1}\in U_{r}$, $r_{1}^{\mathcal{M}}=r_{1}+r^{\prime}$ with $||r^{\prime}||\leq 1$. This means that for the next step forward in time the arguments of $\phi$ will once again be in $\tilde{U}$ so that the Lipschitz conditions will continue to hold. Using $t=1$ as a base case we make the induction hypothesis:

$$||r_{t}^{\mathcal{M}}-r_{t}|| \leq n\beta K_{\phi}\tilde{R}\sum_{t=0}^{t-1}(\alpha+\beta K_{\phi}n\bar{\gamma})^{t}\varepsilon$$ (D.19)

$$||r_{t}^{\mathcal{M}}-r_{t}|| \leq 1.$$ (D.20)

Using the induction hypothesis it is straightforward to prove the induction step, and to show that $||r_{t+1}^{\mathcal{M}}-r_{t+1}||$ remains smaller than $1$ and thus that the arguments of $\phi$ remain in $\tilde{U}$. Proving the induction step and then bounding the output matrix elements by $\bar{\gamma}$ yields the result.

∎