No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks

Abstract

In the past decade, many successful networks are on novel architectures, which almost exclusively use the same type of neurons. Recently, more and more deep learning studies have been inspired by the idea of NeuroAI and the neuronal diversity observed in human brains, leading to the proposal of novel artificial neuron designs. Designing well-performing neurons represents a new dimension relative to designing well-performing neural architectures. Biologically, the brain does not rely on a single type of neuron that universally functions in all aspects. Instead, it acts as a sophisticated designer of task-based neurons. In this study, we address the following question: since the human brain is a task-based neuron user, can the artificial network design go from the task-based architecture design to the task-based neuron design? Since methodologically there are no one-size-fits-all neurons, given the same structure, task-based neurons can enhance the feature representation ability relative to the existing universal neurons due to the intrinsic inductive bias for the task. Specifically, we propose a two-step framework for prototy** task-based neurons. First, symbolic regression is used to identify optimal formulas that fit input data by utilizing base functions such as logarithmic, trigonometric, and exponential functions. We introduce vectorized symbolic regression that stacks all variables in a vector and regularizes each input variable to perform the same computation, which can expedite the regression speed, facilitate parallel computation, and avoid overfitting. Second, we parameterize the acquired elementary formula to make parameters learnable, which serves as the aggregation function of the neuron. The activation functions such as ReLU and the sigmoidal functions remain the same because they have proven to be good. As the initial step, we evaluate the proposed framework via systematic experiments on tabular data and using polynomials as base functions. Empirically, experimental results on synthetic data, classic benchmarks, and real-world applications show that the proposed task-based neuron design is not only feasible but also delivers competitive performance over other state-of-the-art models. We have shared our code in https://github.com/NewT123-WM/Task_based_neurons.

1 Introduction

In the past decade, a majority of deep learning research is on designing outstanding architectures, such as the bottleneck in autoencoders [1], shortcuts [2, 3], and neural architecture search (NAS) [4]. Almost exclusively, these works employ neurons of the same type that use an inner product and a nonlinear activation. We refer to such a neuron as a linear neuron, and a network made of these neurons as a linear network (LN) hereafter. Recently, the field “NeuroAI” emerged [5] to advocate that a large amount of neuroscience knowledge can help catalyze the next generation of AI. This idea is well-motivated, as the brain remains the most intelligent system to date, and an artificial network can be regarded as a miniature of the brain. Following the advocacy of “NeuroAI”, it is noted that our brain is made up of many functionally and morphologically different neurons, while the existing mainstream artificial networks are homogeneous at the neuronal level. Thus, why not introduce neuronal diversity into artificial networks and examine associated merits?

Our overarching opinion is that the neuron type and architecture are two complementary dimensions of an artificial network. Designing well-performing neurons represents a new dimension relative to designing well-performing architectures. Therefore, the neuronal type should be given full attention to harness the full potential of connectionism. In recent years, a plethora of studies have introduced new neurons into deep learning [6, 7, 8, 9, 10, 11] such as polynomial neurons [6] and quadratic neurons [7, 8, 9, 10, 11]. Despite focusing only on a specific type of neuron, this thread of studies reasonably verifies the feasibility and potential of develo** deep learning with new neurons. However, the performance of these neurons is not universally satisfactory, and the improvement is minor on some tasks. Biological neuronal diversity, both in terms of morphology and functionality, arises from the brain’s needs to perform complex tasks [12]. The brain does not rely on a single type of neuron to universally function in all aspects. Instead, it acts as a sophisticated designer of task-based neurons. Hence, in the realm of deep learning, we think that promoting neuronal diversity should also not be limited to specific neuron types like linear or quadratic neurons. Instead, it should take into account the specific context of the tasks at hand.

Can we design different neurons for different tasks (task-based neurons)? Computationally, the philosophy of task-based architectures and task-based neurons is quite distinct. The former is “one-for-all”, which implicitly assumes that stacking a universal and basic type of neurons into different structures can solve a wide class of complicated nonlinear problems. This philosophy is well underpinned by the universal approximation theorem [13]. The latter is “one-for-one”, which assumes that there are no one-size-fits-all neuron types, and it is better to solve a specific problem by prototy** customized neurons. Because task-based neurons are imparted with the implicit bias for the task, the network of task-based neurons can integrate the task-driven forces of all these neurons, which given the same structure should exhibit stronger performance than the network of generic neurons. The key difference between task-based neurons and preset neurons is that the mathematical expression in the task-based neurons is adaptive according to the preference of the task, while in the preset neurons, the mathematical expression is preset.

Figure 1: Two steps to establish a task-based neuron. a) The symbolic regression constructs an elementary neuronal model, which stacks all variables in a vector and regularizes each input variable to perform the same computation. b) The acquired elementary formula is parameterized to be learnable, serving as the aggregation function of a neuron.

Along this direction, three main challenges face us in prototy** task-based neurons: 1) How to efficiently design task-based neurons? 2) How to make the resultant neurons transferable to a network? 3) How to transfer the superiority at the neuronal level to the network level? Here, we propose a two-step framework to address these challenges: First, we introduce vectorized symbolic regression (VSR) to construct an elementary neuronal model, as depicted in Figure 1. Symbolic regression (SR) draws inspiration from scientific discoveries in physics [14, 15], aiming to identify optimal formulas that fit input data by utilizing base functions such as logarithmic, trigonometric, and exponential functions. The vectorized symbolic regression stacks all variables in a vector and regularizes each input variable to perform the same computation. Given the complexity and unclear nonlinearity of the tasks, formulas learned from vectorized symbolic regression can capture the underlying patterns in the data, and these patterns are different in different contexts. Thus, fixed formulas used in pre-designed neurons are disadvantageous. Second, we parameterize the acquired elementary formula to make parameters learnable, which serves as the aggregation function of the neuron. The role of the vectorized symbolic regression is to identify the basic patterns behind data, the parameterization allows the task-based neurons to adapt and interact with each other within a network. The activation functions such as ReLU and the sigmoidal functions remain the same when connected to a network, as these activation functions are widely tested as well-performed.

The vectorized symbolic regression in our framework greatly expedites the search process by avoiding learning highly complex and disordered formulas, particularly for high-dimensional inputs (challenge 1). It also facilitates parallel computing and ensures the feasibility of building a deep network with the designed neurons (challenge 2). The formulas learned by vectorized symbolic regression capture basic patterns in the data and are not sufficiently complex to solve highly intricate problems due to the ablated search space. By connecting task-based neurons into a network, we tap into the power of connectionism, enabling further amplification of the advantages without concerns about overfitting (challenge 3). We refer to a network made of task-based neurons as a task-based network (TN) hereafter.

As the initial step, we evaluate the feasibility and superiority of the proposed framework via systematic experiments on tabular data. Tabular data is one of the most common and important types of data in various domains, including finance, medicine, e-commerce, and many others [16]. Moreover, it encompasses various types of information, such as electronic health records, financial records, sensor readings, and more. Analyzing and understanding tabular data is crucial for making informed decisions and extracting valuable insights. Motivated by the success of quadratic and polynomial neurons, we mainly use polynomials as the base functions for symbolic regression. System experiments show that task-based neurons and associated networks can outperform networks of preset neurons and other state-of-the-art models. To summarize, our contributions are threefold:

• Towards NeuroAI, we propose a framework to design task-based neurons, which is a new dimension compared to task-based architectures and can greatly expand the armory of deep learning models.

• We propose the vectorized symbolic regression to solve the computational challenges in prototy** new neurons. Methodologically, our work is the first to introduce symbolic regression into the design of neurons in deep learning.

• With systematic experiments over synthetic data, public data, and real-world applications, we confirm the effectiveness of the task-based neurons.

2 Related Work

Neuronal diversity. There has been a growing interest in recent years to prototype new neurons and introduce neuronal diversity into artificial networks [7]. It is important to clarify that modifying the activation function should not be considered as creating new neurons. This is because the decision boundary of a neuron is solely determined by the aggregation function, as long as the activation function is monotonic. Therefore, compared to modifying the aggregation function, changing the activation function has a relatively weaker influence on the behavior of a neuron. Currently, excluding spiking neurons [17] featuring the spatiotemporal processing ability, the exploration of neuronal diversity primarily revolves around polynomial or quadratic neurons [7], which replace the inner product with a polynomial or a specially-engineered quadratic function, expanding the range of computations by individual neurons.

In recent years, polynomial neurons were revisited in hope of enhancing the expressive ability of a single neuron. The key issue in designing polynomial neurons is to reduce the complexity such that they can be deployed into a deep network. [18, 6] decreased the complexity of polynomial neurons via tensor decomposition and factor sharing, while a majority of studies directly used quadratic neurons to save parameters to express high-order terms. Table I summarizes the recently-proposed quadratic neurons. As seen, given an $n$-dimensional input, the complexity of neurons in [19, 8, 9] is of $\mathcal{O}(n^{2})$, which is still not bearable for deep networks, while neurons from [10, 20, 21, 22] enjoy the linear parametric complexity. Notably, neurons in [10, 20, 21, 22] are the special cases of [22]. Polynomial and quadratic neurons have demonstrated competitive performance in many tasks such as medical imaging [23], bearing fault diagnosis [11], inverse problems in partial differential equations (PDEs) [20], and signal processing [24].

TABLE I: A summary of the recently-proposed quadratic neurons. $\sigma(\cdot)$ is the nonlinear activation function. $\odot$ denotes Hadamard product. $\mathbf{W}\in\mathbb{R}^{n\times n}$, $\textbf{w}_{i}\in\mathbb{R}^{n\times 1}$, and the bias terms in these neurons are omitted for simplicity.

Authors	Formulations
Zoumpourlis et al.(2017) [19]	$\mathbf{y}=\sigma(\bm{x}^{\top}\mathbf{W}\bm{x}+\textbf{w}^{\top}\bm{x})$
Jiang et al.(2019) [8]	$\mathbf{y}=\sigma(\bm{x}^{\top}\mathbf{W}\bm{x}$)
Mantini&Shah(2021) [9]
Goyal et al.(2020) [10]	$\mathbf{y}=\sigma(\textbf{w}^{\top}(\bm{x}\odot\bm{x}))$
Bu&Karpatne(2021) [20]	$\mathbf{y}=\sigma((\textbf{w}_{1}^{\top}\bm{x})(\textbf{w}_{2}^{\top}\bm{x}))$
Xu et al.(2022) [21]	$\mathbf{y}=\sigma((\textbf{w}_{1}^{\top}\bm{x})(\textbf{w}_{2}^{\top}\bm{x})+\textbf{w}_{3}^{\top}\bm{x})$
Fan et al.(2018) [22]	$\mathbf{y}=\sigma((\textbf{w}_{1}^{\top}\bm{x})(\textbf{w}_{2}^{\top}\bm{x})+\textbf{w}_{3}^{\top}(\bm{x}\odot\bm{x}))$

AutoML and Neural Architecture search (NAS). AutoML, short for Automated Machine Learning, attempts to automate and accelerate these steps to make machine learning more accessible to individuals and organizations that may not have extensive domain expertise [25]. With AutoML, users can focus on higher-level decisions and insights rather than getting swamped in the technical details of machine learning. Regarding selecting a neural network, neural architecture search (NAS) [26, 4] that automatically navigates the best-performing architecture from a set of possible architectures for a given dataset has gained lots of traction in the field of AutoML. NAS has been successful in discovering novel and efficient network architectures, leading to state-of-the-art results in various domains [27].

Although the neuron type and the architecture are the two most important elements of an artificial network, the neuronal type is much less explored. Our methodology for task-based neurons is parallel to NAS, which is a valuable addition to AutoML by extending its scope from architectures to neuronal types.

Symbolic Regression in Deep Learning Recent advances in symbolic regression include learning underlying PDEs from data [28], differentiable symbolic regression [29], and so on. Some works use neural networks to improve symbolic regression. For example, [30] uses a neural network architecture to span the hypothesis space of symbolic regression such that the formulas can be learned in an end-to-end manner. [31] extended symbolic regression to solve parametric systems whose coefficients may vary but the intrinsic structure of the underlying equation keeps intact. [32] proposed a Symbolic Network-based Rectifiable Learning Framework (SNR) that can correct errors generated in the learning-with-experience model.

However, the role of symbolic regression in deep learning is much less explored. To the best of our knowledge, our work is the first time that symbolic regression has been used for task-based neuronal designs.

3 Method

In this section, we first provide a detailed description of how to create task-based neurons using the proposed vectorized symbolic regression. We will explain step-by-step and highlight the benefits of using this approach in terms of efficiency, parallelism, and generalizability.

3.1 Vectorized Symbolic Regression

Unlike traditional regression algorithms that fit numerical coefficients, symbolic regression first encodes a formula into a tree structure and then uses a genetic algorithm to explore the space of possible mathematical expressions to identify the best formula. Because no gradients with respect to the mathematical formula can be computed, the most common technique for solving symbolic regression problems is genetic programming (GP) [33]. GP is a powerful population-based evolutionary algorithm, which mainly uses crossover (Figure 2) and mutation (Figure 3) to generate new formulas.

Figure 2: A schematic diagram of crossover operation.

Crossover is a genetic programming operation to generate new individuals by means of subtree crossover among the selected individuals, and then explore the symbolic expression space. The specific method is to randomly select subtrees of the winner candidates and exchange them (Figure 2). This operation promotes diversity in the population and can lead to the discovery of new and more effective mathematical formulas.

Figure 3: A schematic diagram of mutation operation.

Mutation is a genetic programming operation to randomly select a position of an individual, and generate a new individual through single-point mutation. Due to the randomness of mutation, it can re-join some functions and variables that were eliminated before, thereby potentially leading to the discovery of novel and effective expressions. By injecting variability into the population, mutation plays a crucial role in exploring the solution space and preventing premature convergence to suboptimal solutions (Figure 3).

In prototy** task-based neurons, we consider three important aspects: 1) How to efficiently design task-based neurons? 2) How to make the resultant neurons transferable to a network? 3) How to transfer the superiority at the neuronal level to the network level? We find that the traditional symbolic regression cannot fulfill these needs, particularly for high-dimensional inputs, due to three problems: First, the regression process of traditional symbolic regression becomes slow and computationally expensive for high-dimensional inputs. The search space becomes vast for high-dimensional inputs, as it requires checking an arbitrary form of interactions among two or more input variables. Second, the formulas learned by the traditional symbolic regression are heterogeneous, which suffers from the parametric explosion for high-dimensional inputs and does not support parallel computing and GPU acceleration. Thus, such formulas cannot serve as the aggregation function of a neuron because the resultant neuron cannot be easily integrated into deep networks. Third, the traditional symbolic regression may learn overly complex formulas, subjected to the risk of overfitting when connecting those neurons into a network.

To address these problems, we propose a solution called vectorized symbolic regression. This approach regularizes every variable to learn the same formula, allowing us to organize all variables into a vector. The formulas are then based on vector computation, as illustrated in Figure 4. Unlike traditional symbolic regression, which tends to identify a heterogeneous formula. The vectorized symbolic regression is simple yet mighty, which has valuable characteristics suitable to this task:

Figure 4: The proposed vectorized symbolic regression.

Regression Speed: The vectorized symbolic regression decreases the computational complexity of the regression process, making it much faster than traditional symbolic regression, especially for high-dimensional inputs. This is because the search space is significantly reduced when all variables are regularized to learn the same formula.

Low Complexity and Parallel Computing: Due to the homogeneity, the proposed vectorized symbolic regression leads to mathematical formulas with much fewer parameters. Given $d$-dimensional inputs, the number of parameters is $\mathcal{O}(d)$, which is at the same level as the linear neuron. Moreover, because each variable conducts the same operation, formulas obtained from the proposed vectorized symbolic regression can be organized into the vector or matrix computation, which can facilitate parallel computation aided by GPUs. The low complexity and parallel computing allow for faster and more efficient training of deep networks composed of task-based neurons.

Generalization: The proposed vectorized symbolic regression has a significantly restricted search space. It is unlikely that a homogeneous formula can perfectly fit or overfit data all the time. Therefore, the learned formula tends to underfit data. The power of a neural network is not solely determined by neurons. We can introduce additional flexibility and adaptability to the network structure, enabling it to better handle complex problems and achieve the optimal generalization performance.

Remark. One may ask since linear neurons can already represent any function based on universal approximation [13], why are task-based neurons necessary? While it is true that there is no task that can only be done by task-based neurons but not by linear neurons, the key issue is effectiveness and efficiency. It was reported that a linear network needs an exponential number of parameters to learn the multiplication operation [34]. Task-based neurons search the suitable formulas from a broad function space, which can automatically integrate task-related priors, thereby leveraging the specific strengths of these neurons to tackle complex tasks effectively. Furthermore, task-based neurons can be optimized for a specific task, which can improve the efficiency of the network.

TABLE II: The comparison between SR and the proposed VSR.

	traditional SR	vectorized SR
Regression Speed	✗	✓
Parametric Complexity	✗	✓
Parallel Computing	✗	✓
Generalization	✗	✓

3.2 Parameterization

We expect that the vectorized symbolic regression can identify hidden patterns behind data collected from different tasks. Leveraging these patterns to prototype new neurons would be useful. These patterns are basic and not necessarily specific functions. For instance, we refer to a cell as circular that is characterized by an elliptical equation, but we don’t need to specify the radius of the circle. To take advantage of these patterns, we reparameterize the learned formula by making the fixed constants trainable. Such neurons will perform better than preset neurons since considering the complexity of tasks, there should be no one-size-fits-all neurons. By reparameterizing the learned formula, we can fine-tune the neuron’s behavior to better fit the task at hand. As mentioned earlier, the task-based neurons established through vectorized symbolic regression have limited expressive ability and cannot effectively scale to handle complex tasks on their own. Given a network, the trainable parameters allow for a more efficient and effective search for the optimal solution.

TABLE III: Comparison of the network built by neurons using randomly generated polynomials and expressions generated by symbolic regression. The number $h_{1}$-$h_{2}$-$\cdots$-$h_{k}$-$y(z)$ means that this network has $k$ hidden layers, each with $h_{k}$ neurons, and $z$ is the number of parameters used in such a network. For example, 5-3-1 (145) means this network has two hidden layers with 5 and 3 neurons respectively in each layer, and 145 parameters. RP denotes random polynomials.

Datasets	Random Polynomial (RP)	Structure		Test results
		TN	RP	TN	RP
california housing	$(\bm{x}\odot^{3}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}+1$	6-3-1 (148)	6-3-1 (148)	0.0720 (0.0024)	0.0770 (0.0051)
house sales	$(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}+\bm{x}$	6-4-1 (483)	8-5-1 (509)	0.0079 (0.0008)	0.0133 (0.0150)
airfoil self noise	$(\bm{x}\odot\bm{x})^{\top}$	4-1 (53)	8-1 (57)	0.0438 (0.0065)	0.0840 (0.0102)
wine quality	$(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}$	6-1 (295)	9-5-1 (313)	0.0545 (0.0026)	0.0567 (0.0034)
fifa	$(\bm{x}\odot^{5}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}+\bm{x}$	3-1 (94)	5-1 (96)	0.0611 (0.0032)	0.0616 (0.0048)
diamonds	$(\bm{x}\odot^{7}\bm{x})^{\top}+(\bm{x}\odot^{3}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}$	4-1 (205)	7-1 (218)	0.0109 (0.0036)	0.0123 (0.0052)
abalone	$(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{x}+1$	5-1 (141)	8-1 (153)	0.0239 (0.0024)	0.0240 (0.0030)
Bike Sharing Demand	$(\bm{x}\odot\bm{x})^{\top}$	8-1 (217)	10-8-1 (227)	0.0184 (0.0026)	0.0747 (0.0025)
space ga	$(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot^{3}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}$	2-1 (59)	3-1 (67)	0.0057 (0.0029)	0.0078 (0.0041)
Airlines DepDelay	$(\bm{x}\odot\bm{x})^{\top}$	4-1 (53)	8-1 (57)	0.1645 (0.0055)	0.1664 (0.0058)
credit	$(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{x}+1$	6-2 (152)	6-2 (224)	0.7441 (0.0092)	0.7427 (0.0118)
heloc	$(\bm{x}\odot^{6}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}+\bm{x}$	18-2 (1316)	18-2 (1316)	0.7077 (0.0145)	0.6944 (0.0157)
electricity	$(\bm{x}\odot\bm{x})^{\top}$	5-2 (107)	10-2 (112)	0.7862 (0.0075)	0.7647 (0.0083)
phoneme	$(\bm{x}\odot^{5}\bm{x})^{\top}+(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{x}+1$	5-3 (89)	5-3 (89)	0.8242 (0.0207)	0.8198 (0.0252)
bank-marketing	$(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}+\bm{x}+1$	5-2 (187)	7-2 (198)	0.7938 (0.0099)	0.7919 (0.0099)
MagicTelescope	$(\bm{x}\odot^{4}\bm{x})^{\top}+(\bm{x}\odot\bm{x})^{\top}+\bm{x}+1$	6-2 (296)	8-2 (298)	0.8449 (0.0092)	0.8417 (0.0092)
vehicle	$(\bm{x}\odot^{4}\bm{x})^{\top}+1$	4-4 (360)	13-7-4 (377)	0.8176 (0.0362)	0.6894 (0.0401)
Oranges-vs.-Grapefruit	$(\bm{x}\odot\bm{x})^{\top}$	3-2 (47)	6-2 (50)	0.9305 (0.0037)	0.8952 (0.0160)
eye movements	$(\bm{x}\odot^{3}\bm{x})^{\top}+1$	10-2 (452)	15-8-2 (461)	0.5849 (0.0125)	0.5823 (0.0146)
Contaminant	$(\bm{x}\odot^{5}\bm{x})^{\top}+(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{x}$	10-2 (1612)	15-6-2 (1679)	0.9208 (0.0192)	0.9142 (0.0203)

TABLE IV: Test results of the symbolic regression (SR), the vectorized symbolic regression (VSR), and task-based networks (TN).

Dataset	#Instances	#Features	SR	VSR	TN (Strcture)
google stock price	12,454	4	0.0362	0.5505	0.0319 (4-1)
concrete strength	10,308	8	0.4617	0.7914	0.1144 (4-1)
health insurance	13,386	6	0.3072	0.7009	0.1849 (3-1)
white wine quality	395,611	11	0.7430	0.9288	0.6174 (4-1)
song popularity	1,492,613	13	1.0137	1.0075	0.9366 (13-1)

4 Analysis Experiments

In this section, we present a series of experiments designed to analyze the feasibility, necessity, and superiority of the proposed task-based neurons. For all experiments, we prescribe that the function space of symbolic regression is polynomial. We first validate via the synthetic data the feasibility of the proposed framework for the task-based neurons (Section 1.1 in Supplementary Materials, SMs). It is shown that the vectorized symbolic regression can capture correct hidden formulas from heavily-noised data. Then, we show the necessity of the proposed methodology relative to the symbolic regression. We find that the vectorized symbolic regression is inferior to the symbolic regression in fitting data, while the resultant task-based networks can outperform the symbolic regression. Next, we compare task-based neurons with conventional neurons and different quadratic neurons (Section 1.3 in SMs) to show the superiority of task-based neurons. Furthermore, we compare task-based neurons with neurons using random polynomials to confirm that the polynomials learned from the symbolic regression are reasonable. Lastly, we extend the search space from polynomial bases to trigonometric functions (Section 1.3 in SMs). By expanding the repertoire of functions that task-based neurons can search and utilize, their adaptability and effectiveness in handling more diverse and complex tasks are investigated.

4.1 Necessity of Task-based Neurons

One may argue that instead of using the vectorized symbolic regression to construct an elementary neuron, why not directly use symbolic regression to fit data? The reason is that symbolic regression is not good at handling high-dimensional data. Constructing a neuron is to tap into connectionism, which has been proven to be a powerful approach by numerous successes of deep learning.

To validate these points, we perform TN, traditional symbolic regression (SR), and vectorized symbolic regression (VSR) on five publicly available tabular datasets and compare their performance. For SR and VSR, prediction on the test set is made directly using the expression obtained by SR and VSR, respectively. For TN, we no longer perform the vectorized symbolic regression; instead, we use the mathematical expression obtained from VSR to build a neuron. The network structure is the fully connected network.

TABLE V: The formulas learned by the vectorized symbolic regression over 20 public data.

Datasets	Instances	Features	Classes	Predicted Function
california housing	20640	8	continuous	$\bm{0.068}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.15}\bm{x}^{\top}+0.76$
house sales	21613	15	continuous	$-\bm{0.062}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.025}(\bm{x}\odot^{3}\bm{x})^{\top}-\bm{0.010}(\bm{x}\odot\bm{x})^{\top}+\bm{0.067}\bm{x}^{\top}+0.74$
airfoil self noise	1503	5	continuous	$\bm{0.064}(\bm{x}\odot\bm{x})^{\top}-\bm{0.038}\bm{x}^{\top}-0.087$
wine quality	6497	11	continuous	$\bm{0.0076}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.055}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.10}(\bm{x}\odot\bm{x})^{\top}+\bm{0.055}\bm{x}^{\top}-0.00034$
fifa	18063	5	continuous	$\bm{0.30}(\bm{x}\odot^{5}\bm{x})^{\top}-\bm{0.63}(\bm{x}\odot^{4}\bm{x})^{\top}-\bm{0.10}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.38}(\bm{x}\odot\bm{x})^{\top}+\bm{0.13}\bm{x}^{\top}+0.010$
diamonds	53940	9	continuous	$-\bm{0.075}(\bm{x}\odot^{7}\bm{x})^{\top}+\bm{0.16}(\bm{x}\odot^{6}\bm{x})^{\top}+\bm{0.10}(\bm{x}\odot^{5}\bm{x})^{\top}-\bm{0.27}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.090}(\bm{x}\odot^{3}\bm{x})^{\top}$
abalone	4177	8	continuous	$-\bm{0.088}(\bm{x}\odot^{3}\bm{x})^{\top}-\bm{0.12}(\bm{x}\odot\bm{x})^{\top}+\bm{0.046}\bm{x}^{\top}$
Bike Sharing Demand	17379	12	continuous	$-\bm{0.081}(\bm{x}\odot\bm{x})^{\top}+\bm{0.054}\bm{x}^{\top}$
space ga	3107	6	continuous	$\bm{0.052}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.12}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.025}(\bm{x}\odot\bm{x})^{\top}-\bm{0.073}\bm{x}^{\top}+0.54$
Airlines DepDelay	8000	5	continuous	$\bm{0.010}(\bm{x}\odot\bm{x})^{\top}+\bm{0.042}\bm{x}^{\top}-0.27$
credit	16714	10	2	$-\bm{0.43}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.37}(\bm{x}\odot\bm{x})^{\top}+0.21$
heloc	10000	22	2	$\bm{0.031}(\bm{x}\odot^{6}\bm{x})^{\top}-\bm{0.026}(\bm{x}\odot^{5}\bm{x})^{\top}+\bm{0.055}(\bm{x}\odot^{3}\bm{x})^{\top}$
electricity	38474	8	2	$-\bm{0.21}(\bm{x}\odot\bm{x})^{\top}+\bm{0.21}\bm{x}^{\top}+1.18$
phoneme	3172	5	2	$\bm{1.36}(\bm{x}\odot^{5}\bm{x})^{\top}-\bm{2.91}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.60}(\bm{x}\odot\bm{x})^{\top}+\bm{1.22}\bm{x}^{\top}$
bank-marketing	10578	7	2	$-\bm{1.04}(\bm{x}\odot^{4}\bm{x})^{\top}-\bm{0.14}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.81}(\bm{x}\odot\bm{x})^{\top}+\bm{0.043}\bm{x}^{\top}-0.068$
MagicTelescope	13376	10	2	$-\bm{0.30}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{1.13}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.46}(\bm{x}\odot\bm{x})^{\top}-\bm{0.060}\bm{x}^{\top}+1.02$
vehicle	846	18	4	$-\bm{0.074}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.068}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.072}(\bm{x}\odot\bm{x})^{\top}+\bm{0.0015}\bm{x}^{\top}$
Oranges-vs.-Grapefruit	10000	5	2	$-\bm{0.52}(\bm{x}\odot\bm{x})^{\top}+\bm{0.70}\bm{x}^{\top}+0.89$
eye movements	7608	20	2	$-\bm{0.017}(\bm{x}\odot^{3}\bm{x})^{\top}-\bm{0.011}(\bm{x}\odot\bm{x})^{\top}$
contaminant	2400	30	2	$-\bm{0.75}(\bm{x}\odot^{5}\bm{x})^{\top}-\bm{0.67}(\bm{x}\odot^{4}\bm{x})^{\top}+\bm{0.38}(\bm{x}\odot^{3}\bm{x})^{\top}+\bm{0.24}(\bm{x}\odot\bm{x})^{\top}+\bm{0.13}\bm{x}^{\top}$

To eliminate the effect of large differences in the order of magnitude of features in the data, we do the normalization for features. The training and test sets are divided according to the ratio of $8:2$, and MSE is the evaluation index. We use machine learning library gplearn to implement symbolic regression, gplearn is a mature symbolic regression library based on Python, with good stability and superior performance. Traditional symbolic regression deals with each input variable individually. To implement the vectorized symbolic regression, we just need to feed the input vector into the symbolic regression in gplearn.

We set the fixed random seed to ensure that the symbolic regression can be repeated. For important hyperparameters, we finetune them to achieve the optimal performance for SR and VSR. Finally, in the traditional SR, the population size is set to 5,000; the max generation is 20; the crossover probability is set at 55%; the mutation probability at 40%; the reproduction probability is set at 5%. In VSR, the random seed is set to 100; the population size is set to 5,000; the max generation is set to 30; the crossover probability is set at 30%; the mutation probability at 60%; the reproduction probability at 10%; the tournament size is 3% of the population size. The range of numeric symbols returned is set to $[-1,1]$, following a uniform distribution.

The test results are shown in Table IV, from which we can draw two highlights: First, overall, the performance of the VSR is inferior to SR. While VSR slightly outcompetes SR on the dataset of song popularity, SR leads VSR by a large margin on the rest of datasets. This is because the search space of VSR is ablated. SR is more likely to find a better formula. Second, TN outperforms SR on five datasets, which suggests the power of connectionism and the efficacy of VSR to build a neuron. When a basic pattern regarding data is captured by a single neuron, the corresponding network can leverage these basic patterns to form an effective representation.

4.2 Superiority of Task-based Neurons over Neurons Using Random Polynomials

To further illustrate the necessity and effectiveness of using symbolic regression to generate neurons for different tasks, we perform experiments on 20 public datasets: 10 for classification and 10 for regression. MSE and classification accuracy are evaluation metrics for regression and classification, respectively. Datasets are collected from the scikit-learn package and the official website of OpenML. We first normalize the original data to $[-1,1]$. Then we perform the vectorized symbolic regression on the normalized dataset.

The relevant information regarding these 20 datasets, and the regression results are shown in Table V. After learning formulas from different datasets, we use them to build a neuron and connect neurons into a network to conduct the training and test. We randomly generate a polynomial with the highest number of expressions as the highest number of expressions obtained by symbolic regression, as shown in Table III, and then use the randomly generated expressions to build a neuron and the associated network and compare it with the task-based network.

It should be noted that 1) on the dataset of electricity, Airlines DepDelay, and Oranges-vs.-Grapefruit, because the expression obtained by symbolic regression is of degree 2, we can only set the randomly generated polynomial to a polynomial with only one term and degree 2 (which is $x^{2}$) to make it different from the expression obtained by the symbolic regression. 2) When using neurons to build the network, since the weight parameters and bias will be randomly initialized again, we set the coefficients and constant terms (if any) of the randomly generated neuron expression to 1. In fact, the coefficients and constants of the neuronal expressions do not affect the performance of the network, because they are randomly initialized.

It can be seen that the fitting ability of the network built by randomly generated neurons is weaker than that of the neural network built by neurons obtained by symbolic regression. On both regression and classification, task-based networks scale better and use fewer parameters. These results indicate that symbolic regression plays an important role in identifying the appropriate polynomials.

4.3 Superiority of Task-based Neurons over Linear Neurons

Here, we test the superiority of task-based neurons relative to linear ones. We use the same 20 datasets in the last subsection: 10 for regression and 10 for classification. We don’t need to repeat the process of the vectorized symbolic regression. Instead, we directly use polynomials learned in Table V. The training and test sets are divided according to the ratio of $8:2$. For TN and LN, the data division and the batch size are the same. We select 5 different network structures for each dataset for a comprehensive comparison. When designing the network structures of TN, we ensure that the number of parameters of TN is fewer than the LN to show the superiority of task-based neurons in efficiency. The specific network structure and corresponding number of parameters are shown in SMs. Each dataset is tested 10 times for reliability of results. The MSE and classification accuracy are presented in the form of $\mathrm{mean}~{}(\mathrm{std})$ in Table VI.

TABLE VI: Test results of linear networks and task-based networks of different structures on the public data

Datasets	LN(S1)	TN(S1)	LN(S2)	TN(S2)	LN(S3)	TN(S3)	LN(S4)	TN(S4)	LN(S5)	TN(S5)
california housing	0.0760 (0.0057)	0.0702 (0.0019)	0.0844 (0.0486)	0.0685 (0.0047)	0.0988 (0.0664)	0.0667 (0.0057)	0.0861 (0.0452)	0.0670 (0.0044)	0.1013 (0.0614)	0.0540 (0.0036)
house sales	0.0113 (0.0046)	0.0109 (0.0041)	0.0111 (0.0037)	0.0100 (0.0027)	0.0139 (0.0130)	0.0098 (0.0017)	0.0139 (0.0105)	0.0098 (0.0017)	0.0158 (0.0128)	0.0112 (0.0047)
airfoil self noise	0.0428 (0.0181)	0.0402 (0.0050)	0.0329 (0.0286)	0.0270 (0.0042)	0.0669 (0.0547)	0.0287 (0.0091)	0.0250 (0.0070)	0.0233 (0.0049)	0.0327 (0.0358)	0.0134 (0.0024)
wine quality	0.0636 (0.0084)	0.0625 (0.0083)	0.0619 (0.0041)	0.0602 (0.0021)	0.0634 (0.0098)	0.0592 (0.0051)	0.0613 (0.0076)	0.0592 (0.0051)	0.0604 (0.0094)	0.0597 (0.0066)
fifa	0.0895 (0.0283)	0.0637 (0.0055)	0.0728 (0.0085)	0.0608 (0.0023)	0.1138 (0.0477)	0.0622 (0.0032)	0.1093 (0.0425)	0.0634 (0.0051)	0.0946 (0.0420)	0.0597 (0.0018)
diamonds	0.0128 (0.0050)	0.0097 (0.0023)	0.0108 (0.0066)	0.0078 (0.0020)	0.0084 (0.0032)	0.0081 (0.0027)	0.0103 (0.0073)	0.0080 (0.0018)	0.0145 (0.0169)	0.0084 (0.0024)
abalone	0.0253 (0.0026)	0.0250 (0.0025)	0.0289 (0.0109)	0.0250 (0.0029)	0.0296 (0.0087)	0.0269 (0.0074)	0.0282 (0.0098)	0.0258 (0.0032)	0.0300 (0.0102)	0.0268 (0.0037)
Bike Sharing Demand	0.0144 (0.0040)	0.0132 (0.0022)	0.0133 (0.0047)	0.0110 (0.0022)	0.0103 (0.0015)	0.0092 (0.0010)	0.0137 (0.0043)	0.0101 (0.0021)	0.0104 (0.0018)	0.0083 (0.0011)
space ga	0.0077 (0.0021)	0.0069 (0.0015)	0.0063 (0.0020)	0.0054 (0.0007)	0.0120 (0.0050)	0.0054 (0.0015)	0.0108 (0.0053)	0.0061 (0.0036)	0.0103 (0.0046)	0.0049 (0.0006)
Airlines DepDelay	0.1631 (0.0055)	0.1617 (0.0049)	0.1643 (0.0057)	0.1616 (0.0046)	0.1662 (0.0062)	0.1649 (0.0051)	0.1636 (0.0060)	0.1634 (0.0047)	0.1651 (0.0063)	0.1635 (0.0059)
credit	0.7123 (0.0398)	0.7433 (0.0077)	0.7302 (0.0108)	0.7392 (0.0087)	0.7358 (0.0089)	0.7447 (0.0055)	0.7267 (0.0260)	0.7447 (0.0055)	0.7108 (0.0316)	0.7372 (0.0083)
heloc	0.7030 (0.0091)	0.7080 (0.0077)	0.6980 (0.0112)	0.7040 (0.0063)	0.6950 (0.0145)	0.6950 (0.0098)	0.6890 (0.0073)	0.6900 (0.0073)	0.6910 (0.0129)	0.6930 (0.0100)
electricity	0.7770 (0.0062)	0.7860 (0.0045)	0.7880 (0.0068)	0.7950 (0.0046)	0.7910 (0.0046)	0.8000 (0.0047)	0.7900 (0.0055)	0.7990 (0.0051)	0.7630 (0.0882)	0.8100 (0.0035)
phoneme	0.8170 (0.0321)	0.8420 (0.0099)	0.8020 (0.0994)	0.8510 (0.0116)	0.7790 (0.1460)	0.8560 (0.0107)	0.8470 (0.0131)	0.8550 (0.0079)	0.8120 (0.1080)	0.8540 (0.0090)
bank-marketing	0.7780 (0.0140)	0.7840 (0.0078)	0.7870 (0.0102)	0.7940 (0.0067)	0.7600 (0.0839)	0.7930 (0.0066)	0.7560 (0.0893)	0.7920 (0.0073)	0.7840 (0.0086)	0.7920 (0.0083)
MagicTelescope	0.8452 (0.0098)	0.8531 (0.0071)	0.8430 (0.0078)	0.8557 (0.0062)	0.8456 (0.0047)	0.8580 (0.0040)	0.8462 (0.0073)	0.8573 (0.0063)	0.8433 (0.0090)	0.8580 (0.0066)
vehicle	0.8090 (0.0335)	0.8180 (0.0277)	0.8090 (0.0258)	0.8240 (0.0237)	0.7980 (0.0312)	0.8150 (0.0241)	0.8100 (0.0377)	0.8110 (0.0264)	0.8090 (0.0349)	0.8140 (0.0214)
Oranges-vs.-Grapefruit	0.9288 (0.1381)	0.9429 (0.0049)	0.9430 (0.1489)	0.9740 (0.0020)	0.9429 (0.1490)	0.9751 (0.0020)	0.9426 (0.1457)	0.9751 (0.0020)	0.8927 (0.1944)	0.9758 (0.0076)
eye movements	0.5790 (0.0117)	0.5840 (0.0116)	0.5800 (0.0188)	0.5910 (0.0107)	0.5720 (0.0267)	0.5880 (0.0179)	0.5680 (0.0293)	0.5840 (0.0103)	0.5790 (0.0288)	0.5800 (0.0113)
Contaminant	0.9220 (0.0108)	0.9300 (0.0095)	0.9290 (0.0134)	0.9300 (0.0120)	0.9250 (0.0092)	0.9310 (0.0083)	0.9260 (0.0126)	0.9300 (0.0109)	0.9010 (0.0638)	0.9340 (0.0096)

Regression. The designs of LN and TN all adopt a fully connected network. The activation function is ReLU for LN and Sigmoid for TN. Both networks use MSELoss as the loss function and RMSProp as the optimizer. The details of the specific network structures and the corresponding number of parameters are shown in SMs.

In Table VI, in every dataset, the mean and standard deviation of fitting errors (MSE) of LN are larger than those of TN, indicating that TN has a stronger fitting ability and better generalization than LN. In some datasets such as fifa and airfoil self noise, TN leads LN by a large margin.

Classification. For this task, the network design uses a fully connected network both for LN and TN. The activation function uses ReLU for LN and Sigmoid for TN. Both networks use CrossEntropyLoss as the loss function and Adam as the optimizer. The network is trained and tested in the same way as the regression tasks.

The classification results are shown in Table VI. It is observed that TN has higher training accuracy and test accuracy, indicating that TN has stronger fitting ability. For every dataset, the accuracy of TN over test sets is higher than LN, and the standard deviation is smaller than LN. For datasets like phoneme, electricity, and Orange-vs-Grapefruit, the improvement by TN is at least 3%, which is significant. Moreover, TN achieves better classification accuracy with fewer parameters.

5 Comparative Experiments

In this section, we compare the task-based networks with other state-of-the-art models over two real-world tasks. To highlight the superiority of the network using task-based neurons, we select advanced machine learning models for comparison, namely XGBoost[35], LightGBM[36], CatBoost[37], TabNet[38], TabTransformer[39], FT-Transformer[16] and DANETs[40]. All these models are either classic models or recent models that were published in prestigious venues of machine learning.

High-energy Particle Collision Prediction. High-energy particle collision experiments are an important means to help people understand the fundamental composition and evolution of our universe. In the collision, the medium quickly becomes a soup of deconfined quarks, gluons, and partons within the first few microseconds. Quark-Gluon Plasma (QGP) is the name of this blazing and dense fireball. To investigate the distinctive qualities and evolution of the QGP, particles created at each phase of the medium developed in high-energy collision tests are employed as a probe. The important QGP phase probe is the $J/\psi$ meson, which is the bound state of the charm quark ($c$) and its antiparticle ($\bar{c}$). The invariant mass spectrum of the particle is very useful for selecting the distribution region of the $J/\psi$ signal. In high-energy physics, predicting the invariant mass spectrum of the particle is a critical research topic.

TABLE VII: The test results (MSE errors) of different models on particle collision dataset.

Method	particle collision	asteroid prediction
XGBoost	$0.0094\pm 0.0006$	$0.0646\pm 0.1031$
LightGBM	$0.0056\pm 0.0004$	$0.1391\pm 0.1676$
CatBoost	$0.0028\pm 0.0002$	$0.0817\pm 0.0846$
TabNet	$0.0040\pm 0.0006$	$0.0627\pm 0.0939$
TabTransformer	$0.0038\pm 0.0008$	$0.4219\pm 0.2776$
FT-Transformer	$0.0050\pm 0.0020$	$0.2136\pm 0.2189$
DANETs	$0.0076\pm 0.0009$	$0.1709\pm 0.1859$
Task-based Network	$\mathbf{0.0016\pm 0.0005}$	$\mathbf{0.0513\pm 0.0551}$

CERN collected the collision data and disclosed a dataset on Kaggle . This data set is used to predict the invariant mass of the two electrons in the collision experiment by the properties of electrons such as energy, charge, transverse momentum, and pseudorapidity. It consists of 99,915 observations, and each observation has 16 features.

Asteroid Prediction. Asteroids are petite celestial bodies composed of rocks that revolve around the Sun in elliptical orbits. They are remnants left over from the early stages of solar system formation. Two categories of asteroids, namely Near-Earth Asteroids (NEAs) and Potentially Hazardous Asteroids (PHAs), attract particular interest among researchers. NEAs are characterized by their proximity to Earth’s orbit, indicating the potential risk of collision with our planet. PHAs, in particular, raise concerns due to their potential to approach Earth’s orbit even closer, coupled with their substantial size, which could result in significant damage upon collision with Earth. The prediction of diameter holds crucial significance in the identification of NEAs and PHAs, as the size of these celestial bodies profoundly influences the manner in which they interact with Earth. Smaller asteroids are more prone to be burned out within the Earth’s atmosphere prior to reaching the terrestrial surface, while larger asteroids possess substantial potential for catastrophic destruction upon impact. By accurately predicting the diameters of NEAs and PHAs, researchers can strengthen their assessment of the potential threats these celestial entities pose to our planet, thereby facilitating the formulation of strategies to mitigate any potential hazards.

To predict the diameters of NEAs and PHAs, physicists from the Jet Propulsion Laboratory of CalTech collected a dataset that consists of 137,636 observations and each observation has 19 features, available on Kaggle .

For both datasets, the training set, validation set, and test set are randomly divided at a ratio of 8:1:1. We choose the MSE as the evaluation metric. To ensure a reliable comparison and mitigate the influence of randomness, we conduct 10 times tests for each model, and the final results are presented in the form of $\mathrm{mean}\pm(\mathrm{std})$. The detailed test results are shown in Table VII.

It can be seen that while TabTransformer has good performance on the particle collision dataset but unsatisfactory performance on the asteroid dataset, CatBoost and TabNet have the consistent performance on both datasets. The highlight of Table VII is that the task-based network is the best performer on both datasets. It leads TabTransformer, FT-transformer, and DANETs by a large margin.

6 Conclusion and Future Work

In this paper, towards NeuroAI, we have proposed the roadmap for task-based neurons via symbolic regression, which is a new frontier of neural network research compared to the architecture design. Systematic experiments over 10 synthetic datasets (Supplementary Materials), 25 public datasets, and 3 real-world applications, have confirmed the potential of the task-based neuronal designs.

In the future, on the one hand, the process of the vectorized symbolic regression should be assiduously optimized. We can further investigate how to select suitable base functions for different scenarios to replace the simple symbolic regression algorithm in this paper. Also, from the perspective of algorithmic efficiency, the proposed vectorized symbolic regression has not been facilitated by parallel acceleration and GPU acceleration, so the regression speed has room for improvement. On the other hand, we will explore synergizing task-based neurons and task-based architectures to build a more powerful network model.