SDGym: Low-Code Reinforcement Learning Environments using System Dynamics Models
Authors:
Emmanuel Klu,
Sameer Sethi,
DJ Passey,
Donald Martin Jr
Abstract:
Understanding the long-term impact of algorithmic interventions on society is vital to achieving responsible AI. Traditional evaluation strategies often fall short due to the complex, adaptive and dynamic nature of society. While reinforcement learning (RL) can be a powerful approach for optimizing decisions in dynamic settings, the difficulty of realistic environment design remains a barrier to b…
▽ More
Understanding the long-term impact of algorithmic interventions on society is vital to achieving responsible AI. Traditional evaluation strategies often fall short due to the complex, adaptive and dynamic nature of society. While reinforcement learning (RL) can be a powerful approach for optimizing decisions in dynamic settings, the difficulty of realistic environment design remains a barrier to building robust agents that perform well in practical settings. To address this issue we tap into the field of system dynamics (SD) as a complementary method that incorporates collaborative simulation model specification practices. We introduce SDGym, a low-code library built on the OpenAI Gym framework which enables the generation of custom RL environments based on SD simulation models. Through a feasibility study we validate that well specified, rich RL environments can be generated from preexisting SD models and a few lines of configuration code. We demonstrate the capabilities of the SDGym environment using an SD model of the electric vehicle adoption problem. We compare two SD simulators, PySD and BPTK-Py for parity, and train a D4PG agent using the Acme framework to showcase learning and environment interaction. Our preliminary findings underscore the dual potential of SD to improve RL environment design and for RL to improve dynamic policy discovery within SD models. By open-sourcing SDGym, the intent is to galvanize further research and promote adoption across the SD and RL communities, thereby catalyzing collaboration in this emerging interdisciplinary space.
△ Less
Submitted 19 October, 2023;
originally announced October 2023.
Spectral and Dynamic Consequences of Network Specialization
Authors:
Leonid Bunimovich,
DJ Passey,
Dallas Smith,
Benjamin Webb
Abstract:
One of the hallmarks of real networks is their ability to perform increasingly complex tasks as their topology evolves. To explain this, it has been observed that as a network grows certain subsets of the network begin to specialize the function(s) they perform. A recent model of network growth based on this notion of specialization has been able to reproduce some of the most well-known topologica…
▽ More
One of the hallmarks of real networks is their ability to perform increasingly complex tasks as their topology evolves. To explain this, it has been observed that as a network grows certain subsets of the network begin to specialize the function(s) they perform. A recent model of network growth based on this notion of specialization has been able to reproduce some of the most well-known topological features found in real-world networks including right-skewed degree distributions, the small world property, modular as well as hierarchical topology, etc. Here we describe how specialization under this model also effects the spectral properties of a network. This allows us to give conditions under which a network is able to maintain its dynamics as its topology evolves. Specifically, we show that if a network is intrinsically stable, which is a stronger version of the standard notion of global stability, then the network maintains this type of dynamics as the network evolves. This is one of the first steps toward unifying the rigorous study of the two types of dynamics exhibited by networks. These are the \emph{dynamics of} a network, which is the study of the topological evolution of the network's structure, modeled here by the process of network specialization, and the \emph{dynamics on} a network, which is the changing state of the network elements, where the type of dynamics we consider is global stability. The main examples we apply our results to are recurrent neural networks, which are the basis of certain types of machine learning algorithms.
△ Less
Submitted 12 August, 2019;
originally announced August 2019.
