-
Interpretable Symbolic Regression for Data Science: Analysis of the 2022 Competition
Authors:
F. O. de Franca,
M. Virgolin,
M. Kommenda,
M. S. Majumder,
M. Cranmer,
G. Espada,
L. Ingelse,
A. Fonseca,
M. Landajuela,
B. Petersen,
R. Glatt,
N. Mundhenk,
C. S. Lee,
J. D. Hochhalter,
D. L. Randall,
P. Kamienny,
H. Zhang,
G. Dick,
A. Simon,
B. Burlacu,
Jaan Kasak,
Meera Machado,
Casper Wilstrup,
W. G. La Cava
Abstract:
Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main attraction of this approach is that it returns an interpretable model that can be insightful to users. Historically, the majority of algorithms for symbolic regression have been based on evolutionary algorithms. However, there has been a recent surge of new proposals that instead utilize appr…
▽ More
Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main attraction of this approach is that it returns an interpretable model that can be insightful to users. Historically, the majority of algorithms for symbolic regression have been based on evolutionary algorithms. However, there has been a recent surge of new proposals that instead utilize approaches such as enumeration algorithms, mixed linear integer programming, neural networks, and Bayesian optimization. In order to assess how well these new approaches behave on a set of common challenges often faced in real-world data, we hosted a competition at the 2022 Genetic and Evolutionary Computation Conference consisting of different synthetic and real-world datasets which were blind to entrants. For the real-world track, we assessed interpretability in a realistic way by using a domain expert to judge the trustworthiness of candidate models.We present an in-depth analysis of the results obtained in this competition, discuss current challenges of symbolic regression algorithms and highlight possible improvements for future competitions.
△ Less
Submitted 3 July, 2023; v1 submitted 3 April, 2023;
originally announced April 2023.
-
Symbolic Regression via Neural-Guided Genetic Programming Population Seeding
Authors:
T. Nathan Mundhenk,
Mikel Landajuela,
Ruben Glatt,
Claudio P. Santiago,
Daniel M. Faissol,
Brenden K. Petersen
Abstract:
Symbolic regression is the process of identifying mathematical expressions that fit observed output from a black-box process. It is a discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic programming. In this work, we introduce a hybrid neural-guided/genetic programming ap…
▽ More
Symbolic regression is the process of identifying mathematical expressions that fit observed output from a black-box process. It is a discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic programming. In this work, we introduce a hybrid neural-guided/genetic programming approach to symbolic regression and other combinatorial optimization problems. We propose a neural-guided component used to seed the starting population of a random restart genetic programming component, gradually learning better starting populations. On a number of common benchmark tasks to recover underlying expressions from a dataset, our method recovers 65% more expressions than a recently published top-performing model using the same experimental setup. We demonstrate that running many genetic programming generations without interdependence on the neural-guided component performs better for symbolic regression than alternative formulations where the two are more strongly coupled. Finally, we introduce a new set of 22 symbolic regression benchmark problems with increased difficulty over existing benchmarks. Source code is provided at www.github.com/brendenpetersen/deep-symbolic-optimization.
△ Less
Submitted 17 November, 2021; v1 submitted 29 October, 2021;
originally announced November 2021.
-
Incorporating domain knowledge into neural-guided search
Authors:
Brenden K. Petersen,
Claudio P. Santiago,
Mikel Landajuela
Abstract:
Many AutoML problems involve optimizing discrete objects under a black-box reward. Neural-guided search provides a flexible means of searching these combinatorial spaces using an autoregressive recurrent neural network. A major benefit of this approach is that builds up objects sequentially--this provides an opportunity to incorporate domain knowledge into the search by directly modifying the logi…
▽ More
Many AutoML problems involve optimizing discrete objects under a black-box reward. Neural-guided search provides a flexible means of searching these combinatorial spaces using an autoregressive recurrent neural network. A major benefit of this approach is that builds up objects sequentially--this provides an opportunity to incorporate domain knowledge into the search by directly modifying the logits emitted during sampling. In this work, we formalize a framework for incorporating such in situ priors and constraints into neural-guided search, and provide sufficient conditions for enforcing constraints. We integrate several priors and constraints from existing works into this framework, propose several new ones, and demonstrate their efficacy in informing the task of symbolic regression.
△ Less
Submitted 19 July, 2021;
originally announced July 2021.
-
Improving exploration in policy gradient search: Application to symbolic optimization
Authors:
Mikel Landajuela,
Brenden K. Petersen,
Soo K. Kim,
Claudio P. Santiago,
Ruben Glatt,
T. Nathan Mundhenk,
Jacob F. Pettit,
Daniel M. Faissol
Abstract:
Many machine learning strategies designed to automate mathematical tasks leverage neural networks to search large combinatorial spaces of mathematical symbols. In contrast to traditional evolutionary approaches, using a neural network at the core of the search allows learning higher-level symbolic patterns, providing an informed direction to guide the search. When no labeled data is available, suc…
▽ More
Many machine learning strategies designed to automate mathematical tasks leverage neural networks to search large combinatorial spaces of mathematical symbols. In contrast to traditional evolutionary approaches, using a neural network at the core of the search allows learning higher-level symbolic patterns, providing an informed direction to guide the search. When no labeled data is available, such networks can still be trained using reinforcement learning. However, we demonstrate that this approach can suffer from an early commitment phenomenon and from initialization bias, both of which limit exploration. We present two exploration methods to tackle these issues, building upon ideas of entropy regularization and distribution initialization. We show that these techniques can improve the performance, increase sample efficiency, and lower the complexity of solutions for the task of symbolic regression.
△ Less
Submitted 19 July, 2021;
originally announced July 2021.
-
Distilling Wikipedia mathematical knowledge into neural network models
Authors:
Joanne T. Kim,
Mikel Landajuela,
Brenden K. Petersen
Abstract:
Machine learning applications to symbolic mathematics are becoming increasingly popular, yet there lacks a centralized source of real-world symbolic expressions to be used as training data. In contrast, the field of natural language processing leverages resources like Wikipedia that provide enormous amounts of real-world textual data. Adopting the philosophy of "mathematics as language," we bridge…
▽ More
Machine learning applications to symbolic mathematics are becoming increasingly popular, yet there lacks a centralized source of real-world symbolic expressions to be used as training data. In contrast, the field of natural language processing leverages resources like Wikipedia that provide enormous amounts of real-world textual data. Adopting the philosophy of "mathematics as language," we bridge this gap by introducing a pipeline for distilling mathematical expressions embedded in Wikipedia into symbolic encodings to be used in downstream machine learning tasks. We demonstrate that a $\textit{mathematical}$ $\textit{language}$ $\textit{model}$ trained on this "corpus" of expressions can be used as a prior to improve the performance of neural-guided search for the task of symbolic regression.
△ Less
Submitted 13 April, 2021;
originally announced April 2021.
-
Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients
Authors:
Brenden K. Petersen,
Mikel Landajuela,
T. Nathan Mundhenk,
Claudio P. Santiago,
Soo K. Kim,
Joanne T. Kim
Abstract:
Discovering the underlying mathematical expressions describing a dataset is a core challenge for artificial intelligence. This is the problem of $\textit{symbolic regression}$. Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are underexplored. We propose a framework that leverages deep learning for symbolic regression via…
▽ More
Discovering the underlying mathematical expressions describing a dataset is a core challenge for artificial intelligence. This is the problem of $\textit{symbolic regression}$. Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are underexplored. We propose a framework that leverages deep learning for symbolic regression via a simple idea: use a large model to search the space of small models. Specifically, we use a recurrent neural network to emit a distribution over tractable mathematical expressions and employ a novel risk-seeking policy gradient to train the network to generate better-fitting expressions. Our algorithm outperforms several baseline methods (including Eureqa, the gold standard for symbolic regression) in its ability to exactly recover symbolic expressions on a series of benchmark problems, both with and without added noise. More broadly, our contributions include a framework that can be applied to optimize hierarchical, variable-length objects under a black-box performance metric, with the ability to incorporate constraints in situ, and a risk-seeking policy gradient formulation that optimizes for best-case performance instead of expected performance.
△ Less
Submitted 5 April, 2021; v1 submitted 10 December, 2019;
originally announced December 2019.
