Structured and Balanced Multi-component
and Multi-layer Networks
Structured and Balanced Multi-component
and Multi-layer Neural Networks
Abstract
In this work, we propose a balanced multi-component and multi-layer neural network (MMNN) structure to approximate functions with complex features with both accuracy and efficiency in terms of degrees of freedom and computation cost. The main idea is motivated by a multi-component, each of which can be approximated effectively by a single-layer network, and multi-layer decomposition in a “divide-and-conquer” type of strategy to deal with a complex function. While an easy modification to fully connected neural networks (FCNNs) or multi-layer perceptrons (MLPs) through the introduction of balanced multi-component structures in the network, MMNNs achieve a significant reduction of training parameters, a much more efficient training process, and a much improved accuracy compared to FCNNs or MLPs. Extensive numerical experiments are presented to illustrate the effectiveness of MMNNs in approximating high oscillatory functions and its automatic adaptivity in capturing localized features. Our codes and implementation details are available here.
Key words. deep neural networks, rectified linear unit, function compositions, Fourier series
1 Introduction
The key use of neural networks is to approximate an input-to-output relation, i.e., a map** or a function in the mathematics term. In this work, we continue our study of numerical understanding of neural network approximation of functions from representation to learning dynamics. In our earlier study [38], we demonstrated that a one-hidden-layer (also known as a two-layer or shallow) network is essentially a “low-pass filter” when approximating a function in practice. Due to the strong correlation among the family of activation functions (parameterized by the weight and bias), such as (rectified linear unit), the Gram matrix, the element of which is the pairwise correlation (inner product) of the activation functions, has a fast spectral decay. If initialized randomly, the eigenvectors of the Gram matrix correspond to generalized Fourier modes from low frequency to high frequency ordered corresponding to decreasing eigenvalues. Due to the ill-conditioning of the representation, no matter how wide a one-hidden-layer network is, it can only learn and approximate smooth functions or sample low-frequency modes effectively and stably (with respect to noise or machine round-off errors).
In this work, we propose a balanced multi-component and multi-layer neural network (MMNN) structure based on our previous understanding of a one-hidden-layer network. First, we show that a multi-layer network with a multi-component structure, each of which can be approximated well and effectively by a one-hidden-layer network, can overcome the limitation of a shallow network by smooth decomposition and transformation. Compared to a fully connected neural network of a similar structure, our proposed MMNN is much more effective in terms of representation, training, and accuracy in approximating functions, especially for functions containing complex features, e.g., high-frequency modes. The key idea of MMNNs is to view a linear combination of activation functions as randomly parameterized basis functions, called a component, as a whole to represent a smooth function. Each layer has multiple components all sharing the common basis functions with different linear combinations. The number of components, called rank, is typically much smaller than the layer’s width and increases to enhance the flexibility of decomposition when dealing with more complex functions. These components are combined and composed (through layers) in a structured and balanced way in terms of network width, rank, and depth to approximate a complicated function effectively. Another important feature we used in practice is that weights and biases inside each activation function are randomly assigned and fixed during the optimization while the linear combination weights of activation functions in each component are trained. This leads to more efficient training processes motivated by our finding that a one-hidden-layer neural network can be trained effectively to approximate a smooth function well using random basis functions. We also demonstrate interesting learning dynamics based on Adam optimizer [14], which is crucial for the successful and efficient training of MMNNs. An important remark is that a balanced and holistic approach needs to consider both representation and optimization as well as their interplay altogether.
The outline of this paper is summarized as follows. In Section 2, the design of MMNNs is proposed and explained. Then, in Section 3, a mathematical framework for smooth decomposition and transformation using the MMNN architecture is presented, demonstrating that each component can be effectively approximated by a one-hidden-layer network. Extensive numerical experiments are presented in Section 4 to verify the analysis and demonstrate the effectiveness of MMNNs in the approximation of more complicated functions. Further discussion is presented in Section 5, where more insights and implementation guidelines of MMNNs are provided. Finally, remarks and conclusions are provided in Section 6.
2 Multi-component and multi-layer neural network (MMNN)
This section begins with an overview of the main notations used in this paper, as detailed in Section 2.1. Subsequently, we introduce a novel network architecture, the Multi-component and Multi-layer Neural Network (MMNN), which is developed based on the balanced decomposition principle discussed in Section 2.2. Following this, in Section 2.3, we outline the learning strategy of MMNN and highlight its advantages over other methods. Finally, in Section 2.4, we compare the numerical performance of MMNNs and FCNNs.
2.1 Notations
The following is an overview of the basic notations used in this paper.
-
•
The symbols , , , and are used to denote the sets of natural numbers (including ), integers, rational numbers, and real numbers, respectively. The set of positive natural numbers is denoted as .
-
•
The indicator (or characteristic) function of a set , denoted by , is a function that takes the value for elements of and for elements not in .
-
•
The floor and ceiling functions of a real number can be represented as and .
-
•
Vectors are denoted by bold lowercase letters, such as . On the other hand, matrices are represented by bold uppercase letters. For example, refers to a real matrix of size , and denotes the transpose of matrix .
-
•
Slicing notation is used for a vector , where denotes a slice of from its -th to the -th entries for any with and denotes the -th entry of . For example, if , then and . A similar notation is employed for matrices. For instance, refers to the -th column of , whereas indicates the -th row of . Additionally, corresponds to , which means it extracts the entries from the -th to the -th in the -th row.
2.2 Architecture of MMNNs
In this section, we introduce our Multi-component and Multi-layer Neural Network (MMNN). Each layer of MMNN is a (shallow) neural network of the form
to approximate a vector-valued function , where , and is the width of this network. Here, represents the activation function that can be applied elementwise to vector inputs. Throughout this paper, the activation function is , unless otherwise specified. One can also write it in a more compact form,
(1) |
where
We call each element of , i.e., for , a component. Here are a few key features of :
-
1.
Each component is viewed as a linear combination of basis functions , which is a function in , as a whole.
-
2.
Different components of share the same set of basis with different coefficients .
-
3.
Only are trained while are randomly assigned and fixed.
-
4.
The output dimension and network width can be tuned according to the intrinsic dimension and complexity of the problem.
In comparison, each layer in a typical deep FCNN takes the form , and each hidden neuron is individually a function of the input or each point is mapped to , where is the layer width. All weights are training parameters. In MMNN, each layer is composed of multiple components . Each component is a linear combination of randomly parameterized hidden neurons , which can be more effectively and stably trained through as a smooth decomposition/transformation. Typically the number of components is (much) smaller than the layer width in our experiments.
A MMNN is a multi-layer composition of , i.e.,
where each is a multi-component shallow network defined in (1) of width , where
The width of this MMNN is defined as , the rank as , and the depth as . To simplify, we denote a network with width , rank , and depth using the compact notation . See Figure 1(a) for an illustration of MMNN of size . In contrast, an FCNN can be expressed in the following composition form
where is an affine linear map given by . Readers are referred to Figure 1(b) for an illustration and also a comparison with the MMNN.
For very deep MMNNs, one can borrow ideas from ResNets [8] to address the gradient vanishing issue, making training more efficient. Incorporating this idea, we propose a new architecture given by a multi-layer composition of , i.e.,
where each is a multi-component shallow network defined in (1) with width ,
and is the identity map. We call this architecture ResMMNN. See Figure 1(c) for an illustration of a ResMMNN of size (4,2,3).
The above definition of ResMMNNs requires . If this condition does not hold, we can alternatively define ResMMNN via
where is an operation defined as follows. For any functions and , the operation is given by
It is noteworthy that such an operation is both straightforward and cost-effective to implement. For example, in Python, one can use the following code:
y = f(x); z = g(x); n = min(len(y), len(z)); z[:n] = y[:n] + z[:n] |
After executing this code, z will be the result of the map at x. We remark that the definition of ResMMNN can be generalized to only adding identity maps to certain specific layers, which we still refer to as ResMMNN.
![Refer to caption](x1.png)
![Refer to caption](x2.png)
![Refer to caption](x3.png)
2.3 Learning strategy of MMNNs
Our learning strategy is motivated by the following basic principle: a function can be decomposed in a multi-component and multi-layer structure each component of which can be approximated and trained effectively using a one-hidden-layer network, which is a linear combination of random basis functions (e.g., of the form , see Section 3). Hence optimizing the linear combination weights of the random basis functions, i.e., is both efficient and adequate. On the other hand, optimizing the weights (orientations of the basis functions) ’s and biases ’s to make the basis functions more adaptive to fine features of the target function, which would require capturing high-frequency information by a single layer network, leads to not only significantly more parameters to optimize but also difficulties in training as shown in [38]. Specifically, for each layer of MMNN, we fix the activation function parameters (’s and ’s) as per PyTorch’s default setting during the training process. This entails initializing both weights and biases uniformly from the distribution , where .111It is noteworthy that this initialization approach is similar to the widely used Xavier initialization [4], which draws weights from the distribution with and sets the bias to . The whole training process optimizes all ’s and ’s simultaneously using Adam. Note that it is important to have a uniform sampling of orientations and biases for the random basis functions to be able to approximate an arbitrary smooth function well. Unless stated otherwise, parameter initialization adheres to the default settings provided by PyTorch in our experiments.
To demonstrate the advantages of our training approach (labeled S1), we conduct a comparison with the typical strategy in deep neural networks, denoted as Strategy S2, which uses the default PyTorch initialization and optimizes all parameters during training. In our tests, we select an oscillatory target function and use fairly compact networks. The tests are performed on a total of uniform samples in with a mini-batch size of and a learning rate for epoch- set at for , where denotes the floor operation. The Adam optimizer [14] is applied throughout the training process.
network | (width, rank, depth) | #parameters (trained / all) | test error (MSE) | test error (MAX) | training time |
---|---|---|---|---|---|
MMNN1 (S1) | (400, 20, 6) | 40501 / 83301 | 23.9s / 1000 epochs | ||
MMNN1 (S2) | (400, 20, 6) | 83301 / 83301 | 30.2s / 1000 epochs | ||
MMNN2 (S1) | (590, 28, 6) | 83331 / 170061 | 25.2s / 1000 epochs |
![Refer to caption](x4.png)
![Refer to caption](x5.png)
![Refer to caption](x6.png)
As illustrated in Table 1 and Figure 2, our learning strategy S is significantly more effective than strategy S with comparable accuracy. There are two main advantages of S1. First, S1 requires training only about half the number of parameters compared to S2, which results in time savings. Second, S1 converges more quickly and performs significantly better when the training is not sufficient. We would like to note that in certain specific cases, S2 may outperform S1, particularly when the network size is relatively small and S2 is well-trained. This is expected since S trains all parameters, whereas S only trains a subset. Based on our experience, S1 is more effective in practice, particularly for sufficiently large networks. Alternatively, one might consider a hybrid learning strategy.
2.4 MMNNs versus FCNNs
Previously in Section 1, we discussed the theoretical differences between MMNNs and FCNNs. Now, let’s explore and compare their numerical performance. To ensure a fair comparison, we will use networks with a similar number of parameters, ensuring that all networks have sufficient parameters to learn the target function effectively. Typically, when training an FCNN, all parameters are optimized. For a thorough comparison, we will employ two learning strategies for MMNNs as detailed in Section 2.3: S1 and S2. S1 involves training approximately half the number of parameters of the MMNN, while S2 involves training all parameters.
We choose a 1D function and a 2D function
where and
Refer to Figures 4 and 4 for illustrations of and , respectively.
![Refer to caption](x7.png)
![Refer to caption](extracted/5701051/figures/vsFCNN/vsFCNNf2D.png)
![Refer to caption](x8.png)
We select large network sizes (see Table 2) to ensure that all networks possess sufficient parameters to learn the target functions.222FCNNs perform poorly if the network size is small. For a fair comparison, we choose relatively large network sizes for FCNNs and MMNNs, where both perform reasonably well. For training the 1D function, we sample a total of 1000 data points on a uniform grid within , using a mini-batch size of 100 and a learning rate of for epochs . For training the 2D function, we sample a total of data points on a uniform grid within , using a mini-batch size of 1000 and a learning rate of for epochs .
target function | network | (width, rank, depth) | #parameters (trained / all) | test error (MSE) | test error (MAX) | training time |
---|---|---|---|---|---|---|
MMNN1 (S1) | (388, 18, 6) | 35399 / 73035 | 23.3s / 1000 epochs | |||
FCNN1-1 | (83, –, 6) | 35110 / 35110 | 19.5s / 1000 epochs | |||
MMNN1 (S2) | (388, 18, 6) | 73035 / 73035 | 27.4s / 1000 epochs | |||
FCNN1-2 | (120, –, 6) | 72961 / 72961 | 22.3s / 1000 epochs | |||
MMNN2 (S1) | (789, 36, 12) | 313630 / 637120 | 30.3s / 10 epochs | |||
FCNN2-1 | (168, –, 12) | 312985 / 312985 | 26.7s / 10 epochs | |||
MMNN2 (S2) | (789, 36, 12) | 637120 / 637120 | 35.8s / 10 epochs | |||
FCNN2-2 | (240, –, 12) | 637201 / 637201 | 29.3s / 10 epochs |
![Refer to caption](x9.png)
![Refer to caption](x10.png)
![Refer to caption](x11.png)
![Refer to caption](x12.png)
![Refer to caption](x13.png)
![Refer to caption](x14.png)
![Refer to caption](extracted/5701051/figures/vsFCNN/2DvsFCNN_Diff_w789r36d12kind3.png)
![Refer to caption](extracted/5701051/figures/vsFCNN/2DvsFCNN_Diff_w168r0d12kind0.png)
![Refer to caption](extracted/5701051/figures/vsFCNN/2DvsFCNN_Diff_w789r36d12kind2.png)
![Refer to caption](extracted/5701051/figures/vsFCNN/2DvsFCNN_Diff_w240r0d12kind1.png)
As illustrated in Table 2 and Figure 5, MMNNs outperform FCNNs when both have the same depth and a comparable number of parameters, particularly for relatively oscillatory target functions. Moreover, as indicated in Table 2, the training time for MMNN (S1) is similar to that of FCNN, while MMNN (S2) takes a bit more time. We remark that the primary advantage of MMNNs lies in capturing high-frequency components. As we can see from Figure 5, the differences between network approximations and the corresponding target functions show that FCNNs approximate high-frequency parts of the target functions poorly. In contrast, the approximation errors for MMNNs, especially with the S1 learning strategy, are more evenly distributed across the entire domain, indicating their effectiveness in capturing high-frequency components. The Adam optimizer [14] is applied throughout the training process.
3 Multi-component and multi-layer decomposition
Although a one-hidden-layer neural network is a low-pass filter that can not represent and learn high-frequency features effectively [38], we use mathematical construction to show that MMNNs, which are composed of one-hidden-layer neural networks, can overcome this difficulty by decomposition of the complexity through components and/or depth. We emphasize that the decomposition is highly non-unique. Our construction is “man-made” which can be different from the one by computer through an optimization (learning) process. Our discussion begins with one-dimensional construction in Section 3.1 and later extends to higher dimensions in Section 3.2.
3.1 One dimensional construction
We begin with a two-component decomposition in 1D as both an illustration and an example in Section 3.1.1. Later in Section 3.1.2, we introduce the general multi-component decomposition. Finally in Section 3.1.3, we use concrete examples for demonstration.
3.1.1 Two-component decomposition
We show a simple “divide and conquer” strategy for a target function (example) , a high frequency Fourier mode when is large. Define
and
Then for any we have
Through this decomposition and piecewise linear transformation, which can be approximated easily by a single layer of network, one only needs to approximate a function that is smoother than the original : is simplified, while is reduced to half of the frequency of the original target function .
We observe that this decomposition approach is universally applicable for any function . Specifically, the decomposition is defined as
where the component functions and are defined by
and
Moreover,
Hence, for any , we achieve the following reconstruction of :
demonstrating a structured decomposition that allows the function to be expressed through the composition of a smoother function with a piecewise (component-wise) transformation and rescaling.
3.1.2 General multi-component decomposition
Now we propose a general multi-component adaptive decomposition, a “divide and conquer” strategy, that can distribute the complexity of a target function evenly to multiple components.
Given a sequence where the target function is defined on the interval , we will demonstrate how our new architecture allows us to partition the complexities of the function into smaller intervals . By rescaling each subinterval, one only needs to deal with a much smoother function in each interval. This approach enables us to effectively approximate the target function over the entire interval .
Let be the linear map with
(2) |
Define
(3) |
To decompose the target function into smoother pieces, we define a piecewise linear transformation using a linear combination of two ReLU functions (or a simple single layer network),
(4) |
Here is the “slope” of , which is a local rescaling. For example, becomes a smoother function than after stretching to a larger domain . See an illustration of in Figure 6.
![Refer to caption](x15.png)
![Refer to caption](x16.png)
![Refer to caption](x17.png)
Theorem 3.1.
Given , suppose and are given in Equations (2) and (4), respectively. Then the target function has the following (smoother) decomposition () with a piecewise linear transformation (),
where is given in Equation (3).
Proof of Theorem 3.1.
By definition of in Equation (4), it is easy to check
for a fixed and any . It follows that
It follows that
Since is arbitrary, the above equation holds for all . ∎
For each smoother , one can use a shallow network component - a linear combination of random basis functions to approximate well on . Then
is a one-hidden-layer neural network approximation of the target function that can approximate a complex function better than a single layer. See Figure 7 for an illustration. In practice, one can choose repeated decomposition using a multi-component and multi-layer network structure which is the motivation for MMNN. It is well-known that neural networks can approximate smooth functions well. For localized rapid change/oscillation, our construction shows that a small network in terms of the width as well as the number of components and layers can achieve adaptive decomposition and deal with it rather easily. Hence MMNN is effective in approximating a function with localized fine features. This is an important advantage in dealing with low-dimensional structures embedded in high dimensions. The most difficult situation is approximating global highly oscillatory functions, especially with diverse frequency modes, for which wider networks with more components and layers are needed to deal with both the complexity and curse of dimensions.
3.1.3 Examples
Here we use two examples to demonstrate the complexity decomposition strategy presented in the previous section. We start with the Runge function and modify it to , which has a localized rapid change near . As an example, we use four components , choose points at , and let and for all . In practice, each component is approximated by a single-layer network - a linear combination of basis functions, and trained by an optimization method, e.g., Adam. Our examples here are just a proof of concept for the decomposition of a target function into smoother components using MMNN structure in the form
where and (piecewise tranformation/rescaling) are defined as in (3) and (4), respectively. These components are illustrated in Figure 8. Each component is relatively smooth, making it easier for approximation and learning through shallow networks. This approach essentially utilizes a divide-and-conquer principle.
![Refer to caption](x18.png)
![Refer to caption](x19.png)
![Refer to caption](x20.png)
![Refer to caption](x21.png)
![Refer to caption](x22.png)
![Refer to caption](x23.png)
![Refer to caption](x24.png)
![Refer to caption](x25.png)
![Refer to caption](x26.png)
![Refer to caption](x27.png)
The second example is a globally oscillatory function of the form
Again we illustrate using four components , selecting points at , and setting and for all . As shown in Figure 9, the target function is decomposed into components that are less oscillatory again facilitating their approximation and learning through shallow networks.
![Refer to caption](x28.png)
![Refer to caption](x29.png)
![Refer to caption](x30.png)
![Refer to caption](x31.png)
![Refer to caption](x32.png)
![Refer to caption](x33.png)
![Refer to caption](x34.png)
![Refer to caption](x35.png)
![Refer to caption](x36.png)
![Refer to caption](x37.png)
3.2 High dimensional cases
Let us now consider the extension to multi-dimension using two dimensions as an example since the simple dimension-by-dimension strategy applies to any dimension.
Given and , dividing the domain of the function into small Cartesian rectangles . Let and be the linear maps with
(5) |
For and , we define
(6) |
It is evident that with appropriate transformation and rescaling, is smooth in when is held constant, is smooth in when is fixed, and is smooth in both and . Define
(7) |
The following result provides a decomposition of that fits into the structure of MMNN.
Theorem 3.2.
Proof of Theorem 3.2.
Fixing , for any , we have
It follows that
implying
For each , using the 1D decomposition technique described in Section 3.1, we find the decompositions for and . We have
implying
Moreover,
implying
Therefore, for any ,
Since and are arbitrary, the above equation holds for all . ∎
3.3 Related work
Approximation
Extensive research has examined the approximation capabilities of neural networks, focusing on various architectures to approximate diverse target functions. Early studies concentrated on the universal approximation power of single-hidden-layer networks [3, 10, 11], which demonstrated that sufficiently large neural networks could approximate specific functions with arbitrary precision mathematically, without quantifying the error relative to network size. Subsequent research, such as [36, 35, 1, 39, 2, 6, 7, 21, 30, 31, 19, 32, 37, 33], analyzed the approximation error for different networks in terms of size characterized by width, depth, or the number of parameters. Those studies have primarily concentrated on the mathematical theory that supports the existence theory for such neural networks. However, there has been limited focus on determining the parameters within these networks computationally and the numerical errors, particularly those arising from finite precision in computer simulations. This gap motivated our current investigation, which considers practical training processes and numerical errors. Specifically, the balanced structure of MMNN, the choice of training parameters, and the associated learning strategy discussed here are intended to facilitate a smooth decomposition of the function, thereby promoting an efficient training process.
Low-rank methods
Low-rank structures in the weight matrix of a fully connected neural network have been investigated by various groups. For example, the methods proposed in [26, 12, 28] focus on accelerating training and reducing memory requirements while maintaining final performance. The concept of low-rank structures is further extended to tensor train decomposition in [22]. The MMNN proposed here differs in two key aspects. First, each layer contains two matrices: outside and inside the activation functions. Each row of represents the weights for a linear combination of a set of random basis functions, forming a component in each layer. The number of rows in , which equals the number of components, is selected based on the complexity of the function and is typically much smaller than the number of columns, corresponding to the number of basis functions. Each row of represents a random parameterization of a basis function, with the number of rows in corresponding to the number of basis functions, usually much larger than the number of columns in , which is the input dimension. Secondly, in our MMNN, only is trained while remains fixed with randomly initialized values. Theoretical studies and numerical experiments demonstrate that the architecture of MMNN, combined with the learning strategy, is effective in approximating complex functions.
Random features
Fixing of each layer and use of random basis functions in the MMNNs is inspired by a previous approach known as random features [24, 34, 16, 27, 23]. In typical random feature methods, only the linear combination parameters at the output layer are trained which also leads to the issue of ill-conditioning of the representation. While in MMNNS matrix and vector of each layer are trained. Our MMNN employs a composition architecture and learning mechanism that enhances the approximation capabilities compared to random feature methods while achieving a more effective training process than a standard fully connected network of equivalent size. Extensive experiments demonstrate that our approach can strike a satisfactory balance between approximation accuracy and training cost.
Komogolrov-Arnold (KA) representation
The KA representation theorem [15] states that any multivariate continuous function on a hypercube can be expressed as a finite composition of continuous univariate functions and the binary operation of addition. However, this elegant mathematical representation may result in non-smooth or even fractal univariate functions in general due to this very specific form of representation, a computational challenge one has to address in practice. KA representation has been explored in several studies [17, 20, 32, 13]. A recently proposed network known as the KA network (KAN) utilizes spline functions to approximate the univariate functions in the KA representation. The proposed MMNN is motivated by a multi-component and multi-layer smooth decomposition, or a “divide and conquer” approach, employing distinct network architectures, activation functions, and training strategies.
4 Numerical experiments
In this section, we perform extensive experiments to validate our analysis and demonstrate the effectiveness of MMNNs through multi-component and multi-layer decomposition studied in Section 3. In particular, our tests show its ability in 1) adaptively capturing localized high-frequency features in Section 4.1, 2) approximating highly oscillatory functions in Section 4.2, and 3) extending to higher dimensions in Section 4.3 as well as some interesting learning dynamics in Section 4.4. All our experiments involve target functions that include high-frequency components in various ways and are difficult to handle by shallow networks (no matter how wide) as shown in our previous work [38]. Moreover, our experience on these tests shows that using a fully connected deep neural network would require many more parameters and is much harder (if possible) to train to get a comparable result. This is mainly due to a balanced and structured network design of MMNN in terms of 1) the network width , which is the number of hidden neurons or random basis functions in each component, 2) the rank , which is the number of components in each layer, and 3) the network depth , which is the number of layers in the network. The use of a controllable number of collective components (through ) in each layer instead of a large number of individual neurons and the use of fixed and randomly chosen weights make the training process more effective.
In all tests, 1) data are sampled enough to resolve fine features in the target function, 2) the Adam optimizer is used in training, 3) mean squared error (MSE) is the loss function, 4) all parameters are initialized according to the PyTorch default initialization (see Section 2.3) unless otherwise specified, 5) ’s and ’s (the parameters inside the activation functions, see Section 2.2) are fixed and only ’s and ’s (the parameters outside the activation functions) are trained, 6) computations are conducted on an NVIDIA RTX 3500 Ada Generation Laptop GPU (power cap 130W), with most experiments concluding within a range from a few dozen to several thousand seconds. All our MMNN setups are specified by three parameters which depends on the function complexity. Another tuning parameter is the learning rate the choice of which is guided by the following criteria: 1) not too large initially due to stability, 2) a decreasing rate with iterations such that the learning rate becomes small near the equilibrium to achieve a good accuracy while not decreasing too fast (especially during a long training process for more difficult target functions) so that the training is stalled.
4.1 Localized rapid changes
We begin with two examples in 1D. The first is , which is smooth but features a rapid transition at zero. While demonstrated in our previous work [38], a shallow network struggles to capture such a simple local fast transition which contains high-frequencies, we show that this function can be approximated easily by a composition of a smooth function on top of a (repeated) spatial decomposition and local rescaling using MMNN structure in Section 2.2. Our test indeed verifies that our new architecture can effectively capture a localized fast transition rather easily using a very small network of size as shown in Figure 11. For this test, a total of 1000 data points are uniformly sampled in the range , with a mini-batch size of 100, a learning rate of , and the number of epochs set to 2000. Figure 11 gives the error plot.
![Refer to caption](x38.png)
![Refer to caption](x39.png)
![Refer to caption](x40.png)
![Refer to caption](x41.png)
Next, we consider a more complicated target function, , which represents a localized fast oscillation. For this example, we will conduct two tests. The first one is to show the flexibility of MMNN to automatically adapt to local features. The network has a small size as above . Each layer has a network width of 16. In other words, each component is a linear combination of 16 ReLU functions which has no way to approximate such a target function well. However, with a multi-layer and multi-component decomposition with parameters appropriately trained by Adam, MMNN can adapt to the behavior of the target function as shown in Figure 12. Figure 14 gives the error plot. Also, the test shows that this example is more difficult to train. For this test, there are a total of 1000 uniformly sampled points in with a mini-batch size of 100 and a learning rate of , where denotes floor operation and is the epoch number. It should be noted that in this test, we initialize the biases ’s to and use the PyTorch default initialization method for the weights . This approach, inspired by Xavier initialization, is chosen because the target function is locally oscillatory and the MMNN size is quite small, necessitating a setup adaptive to the target function to facilitate the training. For other experiments, both the biases and weights use the PyTorch default initialization. We then compare with least square approximation using uniform finite element method (FEM) basis with the same degrees of freedom. As shown in Figure 14, MMNN renders a better approximation due to automatic adaptation through the training process. We would like to remark that when training an extremely compact MMNN which does not have much flexibility and makes the training more subtle, the training hyperparameters, such as learning rate, min-batch size, and etc., need to be more carefully tuned. However, when there is some redundancy in MMNN, i.e., an MMNN with a slightly larger size, MMNN becomes more flexible and the training process becomes easier. On the other hand, when the network becomes too large, then training a large number of parameters and over-redundancy will lead to potential difficulties for optimization. This also shows that there is a trade-off between representation and optimization one needs to balance in practice.
![Refer to caption](x42.png)
![Refer to caption](x43.png)
![Refer to caption](x44.png)
![Refer to caption](x45.png)
![Refer to caption](x46.png)
![Refer to caption](x47.png)
![Refer to caption](x48.png)
Now we show an example in 2D shown in Figure 16 and defined in polar coordinates by
Again a rather compact MMNN of size can produce a good approximation. Figure 17 shows the error during the training process and Figure 16 shows the plot of training and testing errors in MSE. For this test there are a total of uniformly sampled points in with mini-batch size of and a learning rate of , where is the epoch number. We compare the result with piecewise linear interpolation and least square approximation using FEM basis on a uniform grid with the same number of degrees of freedom in Figure 18. As observed before, MMNN renders the best result due to its adaptivity through training.
![Refer to caption](extracted/5701051/figures/Local2D/LHF2D3.png)
![Refer to caption](x49.png)
![Refer to caption](x50.png)
![Refer to caption](extracted/5701051/figures/Local2D/epoch10_idx1_w100r10d6_f.png)
![Refer to caption](extracted/5701051/figures/Local2D/epoch50_idx1_w100r10d6_f.png)
![Refer to caption](extracted/5701051/figures/Local2D/epoch100_idx1_w100r10d6_f.png)
![Refer to caption](extracted/5701051/figures/Local2D/epoch1000_idx1_w100r10d6_f.png)
![Refer to caption](extracted/5701051/figures/Local2D/epoch1000_idx1_w100r10d6_diff.png)
![Refer to caption](extracted/5701051/figures/Local2D/LHF2D4interp.png)
![Refer to caption](extracted/5701051/figures/Local2D/LHF2D5fem.png)
![Refer to caption](extracted/5701051/figures/Local2D/epoch1000_idx1_w100r10d6_f.png)
![Refer to caption](extracted/5701051/figures/Local2D/LHF2D4interpDiff.png)
![Refer to caption](extracted/5701051/figures/Local2D/LHF2D5femDiff.png)
4.2 Highly oscillatory functions
Globally oscillatory functions with significant high-frequency components can not be approximated well by a shallow network when a global bounded activation function of the form , such as ReLU, is used. Due to almost orthogonality or high decorrelation (in terms of the inner product) between and oscillatory functions with high likelihood (in terms of a random choice of ), the set of parameters that can render a good approximation, namely the Rashomon set [29], becomes smaller and smaller (in terms of relative measure) and hence harder and harder to find as the target function becomes more and more oscillatory (see [38]). Although this difficulty can be alleviated by complexity decomposition using MMNN as shown in Section 2, it still requires a larger network in terms of width, rank, and layers and more training. Here we limit our tests to oscillatory functions in 1D and 2D due to the dramatic increase of complexity with dimensions, or the curse of dimensions, in general.
We again start with a 1D example, . A MMNN of size produces a good approximation of this highly oscillatory function, as illustrated by the error plot in Figure 20, with a smaller learning rate and a longer training process compared to previous examples with localized fine features. Due to the significant depth, we consider using ResMMNN as discussed in Section 2.2. For this test, a total of uniformly sampled points in are used with a mini-batch size of and a learning rate of , where is the epoch number. Also, an interesting learning dynamics for Adam is observed from Figure 20. In the beginning, nothing seems to happen until about epoch 3600 when learning starts from the boundary. Then more and more features are captured from the boundary to the inside gradually. Eventually, all features are captured and then fine-tuned together to improve the overall approximation.
![Refer to caption](x51.png)
![Refer to caption](x52.png)
![Refer to caption](x53.png)
![Refer to caption](x54.png)
![Refer to caption](x55.png)
![Refer to caption](x56.png)
![Refer to caption](x57.png)
Next, we consider a two-dimensional target function of the following form:
where
In our test, we choose to ensure the function exhibits significant oscillations and contains diverse Fourier modes as illustrated by Figure 22. Given the complexity of the function, we employ a MMNN with size . Again, ResMMNN is used due to the depth. For this test, a total of data are sampled on a uniform grid in with a mini-batch size of and a learning rate of , where is the epoch number. The training process is illustrated by Figure 23. Figure 22 shows -error plot.
![Refer to caption](extracted/5701051/figures/Global2D/HForiginal3.png)
![Refer to caption](x58.png)
![Refer to caption](x59.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch25_idx1_w600r30d15_diff.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch50_idx1_w600r30d15_diff.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch1000_idx1_w600r30d15_diff.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch2000_idx1_w600r30d15_diff.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch25_idx1_w600r30d15_f.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch50_idx1_w600r30d15_f.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch1000_idx1_w600r30d15_f.png)
![Refer to caption](extracted/5701051/figures/Global2D/epoch2000_idx1_w600r30d15_f.png)
We trained the same function using identical network settings, except we limited the domain of interest to a unit disc. We sampled data points uniformly distributed over the area, then filtered to retain only those points that fall within the unit disk, totaling approximately samples. As illustrated in Figure 24, our network successfully learned the target function in the disc with no adjustments or modifications. This test highlights the network’s flexibility for domain geometry, an advantage over traditional mesh or grid-based methods, especially in higher dimensions.
![Refer to caption](extracted/5701051/figures/Global2D/UnitBallepoch2000_idx1_w600r30d15_f.png)
![Refer to caption](extracted/5701051/figures/Global2D/UnitBallepoch2000_idx1_w600r30d15_diff.png)
![Refer to caption](x60.png)
4.3 Tests in three dimension and higher
In this section, we test a few examples in three and four dimensions. Even sampling an interesting function becomes challenging as the dimension becomes higher. Although our examples are limited by our computation power using a laptop, our tests show that MMNN performs well and is more effective than a fully connected network.
The first example is a 3D function a level set of which is shown in Figure 27. Using polar coordinates , , , the target function is defined as:
where
![Refer to caption](x61.png)
![Refer to caption](x61.png)
![Refer to caption](x62.png)
Our MMNN is of a compact size . For this test, a total of data are sampled on a uniform grid in with a mini-batch size of and a learning rate of for epochs . Figure 27 gives the error plot. As shown in Figures 27 and 27, the levelsets corresponding to the target function and the learned MMNN approximation are nearly identical. To visually demonstrate the quality of the approximation and complex structure of the 3D function, we present several slices of the target function and the MMNN approximation by fixing either , , or in Figure 28.
![Refer to caption](extracted/5701051/figures/HighDim/3DsliceZ0.png)
![Refer to caption](extracted/5701051/figures/HighDim/3DsliceZ1.png)
![Refer to caption](extracted/5701051/figures/HighDim/3DsliceY0.png)
![Refer to caption](extracted/5701051/figures/HighDim/3DsliceY1.png)
![Refer to caption](extracted/5701051/figures/HighDim/3DsliceX1.png)
![Refer to caption](extracted/5701051/figures/HighDim/3DsliceX0.png)
Next, we consider the probability density function (PDF) of a Gaussian (normal) distribution in 4D,
where is the covariance matrix. We set and We remark that the eigenvalues of are which means that the distribution is quite anisotropic and concentrated near the center.
A compact MMNN with size of produces a good approximation as shown in the error plot Figure 30. Figure 30 compares the true function and the MMNN approximation with . For this test a total of data are sampled on a uniform grid in with a mini-batch size of and a learning rate at for epochs .
![Refer to caption](extracted/5701051/figures/HighDim/4DsliceZU0.png)
![Refer to caption](x63.png)
4.4 Learning dynamics
In this section, we show some interesting learning dynamics observed during the training process. As the first example in Section 4.2 and the following examples show, the training process not just learns from low frequency first but can also learn feature by feature, i.e., can be localized in both frequency domain and spatial domain. We believe this is due to the combination of MMNN’s “divide and conquer” ability and the Adam optimizer which utilizes momentum. More understanding is needed and will be studied in our future research.
We again start with a 1D example, . A MMNN of size produces a good approximation of this highly oscillatory function, as illustrated by the error plot in Figure 32. For this test, a total of uniformly sampled points in are used with a mini-batch size of and a learning rate of , where is the epoch number. As illustrated in Figure 32, the function is less oscillatory near 0. Therefore, we might anticipate that the network will initially learn the part near 0 and then feature by feature from the middle to the boundary. The experimental results presented in Figure 33 agree with our expectations.
![Refer to caption](x64.png)
![Refer to caption](x65.png)
![Refer to caption](x66.png)
![Refer to caption](x67.png)
![Refer to caption](x68.png)
![Refer to caption](x69.png)
![Refer to caption](x70.png)
![Refer to caption](x71.png)
![Refer to caption](x72.png)
![Refer to caption](x73.png)
Now we show an example of 2D function (see Figure 35) defined in polar coordinates as
Our MMNN is of a compact size . For this test, a total of data are sampled on a uniform grid in with a mini-batch size of and a learning rate of for epochs . Figure 35 gives the error plot. The training process shown in Figure 36 illustrates that an overall coarse scale or low-frequency component of the shape is learned first and then localized features are learned one by one from coarse to fine.
![Refer to caption](extracted/5701051/figures/Dynamics/LDfun2D.png)
![Refer to caption](x74.png)
![Refer to caption](x75.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch1_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch3_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch5_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch7_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch14_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch22_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch30_idx1_w500r20d8_f.png)
![Refer to caption](extracted/5701051/figures/Dynamics/epoch300_idx1_w500r20d8_f.png)
5 Further discussion
In this section, we provide several general insights about MMNNs. First, in Section 5.1, we explore the advantages of MMNNs compared to fully connected networks (FCNNs) or multi-layer perceptrons (MLPs). Next, in Section 5.2, we offer practical guidelines for determining the appropriate MMNN size based on our theoretical understanding and extensive numerical experiments. Finally in Section 5.3, we discuss the use of alternative activation functions beyond in MMNNs.
5.1 Advantages compared to FCNNs or MLPs
-
•
The two key differences between a standard FCNN or MLP and a MMNN are 1) the introduction of the weights for different linear combinations of hidden neurons (or perceptrons) as the multi-components in each layer, and 2) the training strategy that fixes those randomly initialized (random features) in the hidden neurons. Hence it is extremely easy to modify a FCNN or MLP to a MMNN.
-
•
MMNNs are much more effective than FCNNs in terms of representation, training, and accuracy especially for complex functions. In comparison, as shown in those experiments in Section 2.4, MMNNs 1) have much fewer training parameters, 2) converge much faster in training, 3) achieve much better accuracy. Moreover, experiments show that training process of MMNNs converges not only faster but also with a steady rate while FCNNs saturates pretty early to a quite low accuracy, as commonly observed in practices. These nice behaviors of MMNNs are due to their balanced structure for smooth decomposition as well as the training strategy. In practice, the introduction of in MMNNs provides an important balance between the network width, which is the number of hidden neurons (basis functions) and can be very large, and the dimension of the input space, which is the number of components from the previous layer and can be much smaller than the network width. In other words, using a few linear combinations of the basis functions can capture smooth structures in the input space well. On the other hand, for FCNNs the two are the same and no balance is exerted.
5.2 Practical guidelines for MMNN
There are three hyperparameters for the configuration of MMNN sizes, the network width, the number of components (rank), and the number of layers (depth). Here are the general guidelines based on our mathematical construction and extensive experiments:
-
1.
The network width should provide enough resolution to capture fine details of the target function. This means that the width should be at least comparable to the size of an adaptive mesh that can approximate the target function well.
-
2.
The number of components (rank) is related to the overall complexity of the target function which depends on its spatial domain and Fourier domain representation as well as the input dimension. As indicated by our mathematical multi-component construction, it is related to the “divide and conquer” strategy.
-
3.
The number of layers (depth) is also related to the overall complexity of the target function as for the number of components. Rank and depth are complementary but work together effectively for a smooth decomposition of the target function. The rule of thumb for depth is similar to that for the rank.
Here we use more concrete examples to illustrate the guidelines. For simplicity we fix the input dimension and domain of interest. As the domain size and dimension increases, the network size needs to increase correspondingly. For a smooth target function, a compact MMNN in terms of width, rank, and depth is enough and easy training process can render accurate results. Larger MMNNs are needed for target functions with localized rapid changes. Even with a relative compact size, the training process can allocate resources adaptive to the target function and render good approximation. The most difficult situation is to approximate globally highly oscillatory functions with diverse Fourier modes for which large MMNNs are needed. For instance, if the oscillation frequency doubles, the network width should increase by where is the dimension. In general the network width needs to deal with the curse of dimensionality just like a mesh based method. However, the growth of the number of components and layers with the increase of complexity seems to be relative mild (maybe polylogarithmic suggested by our mathematical construction).
-
•
Overall, for a given target function, MMNNs can work well with quite a large range of configuration with a trade-off between the network size and training process. For example, the training process for a network more on the compact size with respect to the complexity of a given target function may become more subtle and challenging, e.g., choosing the appropriate learning rate and batch size, due to the lack of flexibility (or redundancy) of the representation. On the other hand, a network of too large size (or redundancy) with respect to the complexity of a given target function requires unnecessarily expensive training cost. An interesting and important question for future research is how to develop a posteriori strategy to automatically adjust the network size in practice.
-
•
The most advantageous situation for using MMNNs is when approximating a function in relative high dimension which is mostly smooth except for localized fine features, e.g., a distribution in high dimensions concentrated on a low dimensional manifold. Through training, MMNNs can provide an automatic adaptive approximation of the underlying structure which can be challenging for a traditional mesh based method.
-
•
We would like to remark that learning rate scheduler can be a subtle and important issue for all training process in practice. For all our training process, the Step Learning Rate suffices. However, one could consider using other learning rate schedulers, such as the Cosine Scheduler [18] or the gradual warm-up strategy [5]. Exploring and designing a more efficient learning rate scheduler with some automatic restart mechanism is a potential interesting topic for future work.
5.3 Beyond ReLU to other activation functions
We also tried using different activation functions for MMNNs, e.g., [9], [25], , and . In general, ReLU provides the overall best results for various target functions. However, in situations where a smooth (e.g., or real analytic) approximation is needed, one might consider using smooth alternatives to such as or , which generally yield results comparable to :
-
•
For target functions that are or even real analytic (and can be highly oscillatory), such as , (or ) tends to perform slightly better than .
-
•
For target functions with non-differentiable points, such as , (or ) generally performs slightly worse than .
-
•
The use of (or ) typically results in slightly longer training time compared to .
Additionally, other popular S-shaped activation functions like and have demonstrated poor performance in our tests, possibly due to the vanishing gradient problem. For highly oscillatory target functions, when using or training errors did not even decrease during the training process.
6 Conclusion
In this work, we introduced the Multi-component and Multi-layer Neural Network (MMNN) and demonstrated its effectiveness in approximating complex functions. By incorporating the principles of structured and balanced decomposition, the MMNN architecture addresses the limitations of shallow networks, particularly in capturing high-frequency components and localized fine features. Our proposed network structure as confirmed by extensive numerical experiments can approximate highly oscillatory functions and functions with rapid transitions efficiently and accurately. Additionally, we highlight the advantages of our training strategy, which optimizes only the linear combination weights of basis functions for each component while kee** the parameters within the activation (basis) functions fixed, leading to a more efficient and stable training process.
The theoretical underpinnings and practical implementations presented in this paper suggest that MMNNs offer a promising direction for constructing neural networks capable of handling complex tasks with fewer parameters and reduced computational overhead. Future research can explore further generalizations and applications of MMNNs, as well as investigate the interplay between representation and optimization in more depth.
Acknowledgments
H. Zhao was partially supported by NSF grants DMS-2309551, and DMS-2012860. Y. Zhong was partially supported by NSF grant DMS-2309530, H. Zhou was partially supported by NSF grant DMS-2307465.
References
- [1] Helmut. Bölcskei, Philipp. Grohs, Gitta. Kutyniok, and Philipp. Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1(1):8–45, 2019.
- [2] Charles K. Chui, Shao-Bo Lin, and Ding-Xuan Zhou. Construction of neural networks for realization of localized deep learning. Frontiers in Applied Mathematics and Statistics, 4:14, 2018.
- [3] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2:303–314, 1989.
- [4] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
- [5] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv e-prints, page arXiv:1706.02677, June 2017.
- [6] Rémi Gribonval, Gitta Kutyniok, Morten Nielsen, and Felix Voigtlaender. Approximation spaces of deep neural networks. Constructive Approximation, 55:259–367, 2022.
- [7] Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep ReLU neural networks in norms. Analysis and Applications, 18(05):803–859, 2020.
- [8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016.
- [9] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv e-prints, page arXiv:1606.08415, June 2016.
- [10] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991.
- [11] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989.
- [12] Yerlan Idelbayev and Miguel Á. Carreira-Perpiñán. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8046–8056, 2020.
- [13] Aysu Ismayilova and Vugar E. Ismailov. On the kolmogorov neural networks. Neural Networks, 176:106333, 2024.
- [14] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
- [15] A. N. Kolmogorov. On the representation of continuous functions of several variables by superposition of continuous functions of one variable and addition. Dokl. Akad. Nauk SSSR, pages 953–956, 1957.
- [16] Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan A. K. Suykens. Random features for kernel approximation: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7128–7148, 2022.
- [17] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-Arnold networks. arXiv e-prints, page arXiv:2404.19756, April 2024.
- [18] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
- [19] Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465–5506, 2021.
- [20] Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by MLP neural networks. Neurocomputing, 25(1):81–91, 1999.
- [21] Hadrien Montanelli and Haizhao Yang. Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem. Neural Networks, 129:1–6, 2020.
- [22] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
- [23] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2021.
- [24] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
- [25] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. arXiv e-prints, page arXiv:1710.05941, October 2017.
- [26] Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N. Gomez. Exploring low rank training of deep neural networks. arXiv e-prints, page arXiv:2209.13569, September 2022.
- [27] Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- [28] Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6655–6659, 2013.
- [29] Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1827–1858, 2022.
- [30] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Nonlinear approximation via compositions. Neural Networks, 119:74–84, 2019.
- [31] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation characterized by number of neurons. Communications in Computational Physics, 28(5):1768–1811, 2020.
- [32] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation: Achieving arbitrary accuracy with fixed number of neurons. Journal of Machine Learning Research, 23(276):1–60, 2022.
- [33] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation in terms of intrinsic parameters. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19909–19934. PMLR, 17–23 Jul 2022.
- [34] Aman Sinha and John C Duchi. Learning kernels with random features. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
- [35] Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.
- [36] Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 639–649. PMLR, 06–09 Jul 2018.
- [37] Shijun Zhang. Deep neural network approximation via function compositions. PhD Thesis, National University of Singapore, 2020.
- [38] Shijun Zhang, Hongkai Zhao, Yimin Zhong, and Haomin Zhou. Why shallow networks struggle with approximating and learning high frequency: A numerical study. arXiv e-prints, page arXiv:2306.17301, June 2023.
- [39] Ding-Xuan Zhou. Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2):787–794, 2020.