-
Exponential time differencing for matrix-valued dynamical systems
Authors:
Nayef Shkeir,
Tobias Schäfer,
Tobias Grafke
Abstract:
Matrix evolution equations occur in many applications, such as dynamical Lyapunov/Sylvester systems or Riccati equations in optimization and stochastic control, machine learning or data assimilation. In many cases, their tightest stability condition is coming from a linear term. Exponential time differencing (ETD) is known to produce highly stable numerical schemes by treating the linear term in a…
▽ More
Matrix evolution equations occur in many applications, such as dynamical Lyapunov/Sylvester systems or Riccati equations in optimization and stochastic control, machine learning or data assimilation. In many cases, their tightest stability condition is coming from a linear term. Exponential time differencing (ETD) is known to produce highly stable numerical schemes by treating the linear term in an exact fashion. In particular, for stiff problems, ETD methods are a method of choice. We propose an extension of the class of ETD algorithms to matrix-valued dynamical equations. This allows us to produce highly efficient and stable integration schemes. We show their efficiency and applicability for a variety of real-world problems, from geophysical applications to dynamical problems in machine learning.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning
Authors:
Mani Hamidi,
Sina Khajehabdollahi,
Emmanouil Giannakakis,
Tim Schäfer,
Anna Levina,
Charley M. Wu
Abstract:
Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth c…
▽ More
Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth curriculum. We find that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics, including training time, generalizability, and robustness to some perturbations. We further examine how different aspects of a modular network's connectivity contribute to its computational capability. We then demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed and only the connections between modules are trained. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales, and help build more scalable and compressible artificial networks.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
High-speed odour sensing using miniaturised electronic nose
Authors:
Nik Dennler,
Damien Drix,
Tom P. A. Warner,
Shavika Rastogi,
Cecilia Della Casa,
Tobias Ackels,
Andreas T. Schaefer,
André van Schaik,
Michael Schmuker
Abstract:
Animals have evolved to rapidly detect and recognise brief and intermittent encounters with odour packages, exhibiting recognition capabilities within milliseconds. Artificial olfaction has faced challenges in achieving comparable results -- existing solutions are either slow; or bulky, expensive, and power-intensive -- limiting applicability in real-world scenarios for mobile robotics. Here we in…
▽ More
Animals have evolved to rapidly detect and recognise brief and intermittent encounters with odour packages, exhibiting recognition capabilities within milliseconds. Artificial olfaction has faced challenges in achieving comparable results -- existing solutions are either slow; or bulky, expensive, and power-intensive -- limiting applicability in real-world scenarios for mobile robotics. Here we introduce a miniaturised high-speed electronic nose; characterised by high-bandwidth sensor readouts, tightly controlled sensing parameters and powerful algorithms. The system is evaluated on a high-fidelity odour delivery benchmark. We showcase successful classification of tens-of-millisecond odour pulses, and demonstrate temporal pattern encoding of stimuli switching with up to 60 Hz. Those timescales are unprecedented in miniaturised low-power settings, and demonstrably exceed the performance observed in mice. For the first time, it is possible to match the temporal resolution of animal olfaction in robotic systems. This will allow for addressing challenges in environmental and industrial monitoring, security, neuroscience, and beyond.
△ Less
Submitted 5 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks
Authors:
Sina Khajehabdollahi,
Roxana Zeraati,
Emmanouil Giannakakis,
Tim Jakob Schäfer,
Georg Martius,
Anna Levina
Abstract:
Recurrent neural networks (RNNs) in the brain and in silico excel at solving tasks with intricate temporal dependencies. Long timescales required for solving such tasks can arise from properties of individual neurons (single-neuron timescale, $τ$, e.g., membrane time constant in biological neurons) or recurrent interactions among them (network-mediated timescale). However, the contribution of each…
▽ More
Recurrent neural networks (RNNs) in the brain and in silico excel at solving tasks with intricate temporal dependencies. Long timescales required for solving such tasks can arise from properties of individual neurons (single-neuron timescale, $τ$, e.g., membrane time constant in biological neurons) or recurrent interactions among them (network-mediated timescale). However, the contribution of each mechanism for optimally solving memory-dependent tasks remains poorly understood. Here, we train RNNs to solve $N$-parity and $N$-delayed match-to-sample tasks with increasing memory requirements controlled by $N$ by simultaneously optimizing recurrent weights and $τ$s. We find that for both tasks RNNs develop longer timescales with increasing $N$, but depending on the learning objective, they use different mechanisms. Two distinct curricula define learning objectives: sequential learning of a single-$N$ (single-head) or simultaneous learning of multiple $N$s (multi-head). Single-head networks increase their $τ$ with $N$ and are able to solve tasks for large $N$, but they suffer from catastrophic forgetting. However, multi-head networks, which are explicitly required to hold multiple concurrent memories, keep $τ$ constant and develop longer timescales through recurrent connectivity. Moreover, we show that the multi-head curriculum increases training speed and network stability to ablations and perturbations, and allows RNNs to generalize better to tasks beyond their training regime. This curriculum also significantly improves training GRUs and LSTMs for large-$N$ tasks. Our results suggest that adapting timescales to task requirements via recurrent interactions allows learning more complex objectives and improves the RNN's performance.
△ Less
Submitted 22 September, 2023;
originally announced September 2023.
-
Your Face Mirrors Your Deepest Beliefs-Predicting Personality and Morals through Facial Emotion Recognition
Authors:
P. A. Gloor,
A. Fronzetti Colladon,
E. Altuntas,
C. Cetinkaya,
M. F. Kaiser,
L. Ripperger,
T. Schaefer
Abstract:
Can we really "read the mind in the eyes"? Moreover, can AI assist us in this task? This paper answers these two questions by introducing a machine learning system that predicts personality characteristics of individuals on the basis of their face. It does so by tracking the emotional response of the individual's face through facial emotion recognition (FER) while watching a series of 15 short vid…
▽ More
Can we really "read the mind in the eyes"? Moreover, can AI assist us in this task? This paper answers these two questions by introducing a machine learning system that predicts personality characteristics of individuals on the basis of their face. It does so by tracking the emotional response of the individual's face through facial emotion recognition (FER) while watching a series of 15 short videos of different genres. To calibrate the system, we invited 85 people to watch the videos, while their emotional responses were analyzed through their facial expression. At the same time, these individuals also took four well-validated surveys of personality characteristics and moral values: the revised NEO FFI personality inventory, the Haidt moral foundations test, the Schwartz personal value system, and the domain-specific risk-taking scale (DOSPERT). We found that personality characteristics and moral values of an individual can be predicted through their emotional response to the videos as shown in their face, with an accuracy of up to 86% using gradient-boosted trees. We also found that different personality characteristics are better predicted by different videos, in other words, there is no single video that will provide accurate predictions for all personality characteristics, but it is the response to the mix of different videos that allows for accurate prediction.
△ Less
Submitted 23 December, 2021;
originally announced December 2021.
-
Bayesian Inverse Reinforcement Learning for Collective Animal Movement
Authors:
Toryn L. J. Schafer,
Christopher K. Wikle,
Mevin B. Hooten
Abstract:
Agent-based methods allow for defining simple rules that generate complex group behaviors. The governing rules of such models are typically set a priori and parameters are tuned from observed behavior trajectories. Instead of making simplifying assumptions across all anticipated scenarios, inverse reinforcement learning provides inference on the short-term (local) rules governing long term behavio…
▽ More
Agent-based methods allow for defining simple rules that generate complex group behaviors. The governing rules of such models are typically set a priori and parameters are tuned from observed behavior trajectories. Instead of making simplifying assumptions across all anticipated scenarios, inverse reinforcement learning provides inference on the short-term (local) rules governing long term behavior policies by using properties of a Markov decision process. We use the computationally efficient linearly-solvable Markov decision process to learn the local rules governing collective movement for a simulation of the self propelled-particle (SPP) model and a data application for a captive guppy population. The estimation of the behavioral decision costs is done in a Bayesian framework with basis function smoothing. We recover the true costs in the SPP simulation and find the guppies value collective movement more than targeted movement toward shelter.
△ Less
Submitted 11 June, 2022; v1 submitted 8 September, 2020;
originally announced September 2020.
-
CLS-SMT: Bringing Together Combinatory Logic Synthesis and Satisfiability Modulo Theories
Authors:
Fadil Kallat,
Tristan Schäfer,
Anna Vasileva
Abstract:
We introduce an approach that aims to combine the usage of satisfiability modulo theories (SMT) solvers with the Combinatory Logic Synthesizer (CL)S framework. (CL)S is a tool for the automatic composition of software components from a user-specified repository. The framework yields a tree grammar that contains all composed terms that comply with a target type. Type specifications for (CL)S are b…
▽ More
We introduce an approach that aims to combine the usage of satisfiability modulo theories (SMT) solvers with the Combinatory Logic Synthesizer (CL)S framework. (CL)S is a tool for the automatic composition of software components from a user-specified repository. The framework yields a tree grammar that contains all composed terms that comply with a target type. Type specifications for (CL)S are based on combinatory logic with intersection types. Our approach translates the tree grammar into SMT functions, which allows the consideration of additional domain-specific constraints. We demonstrate the usefulness of our approach in several experiments.
△ Less
Submitted 26 August, 2019;
originally announced August 2019.
-
StruClus: Structural Clustering of Large-Scale Graph Databases
Authors:
Till Schäfer,
Petra Mutzel
Abstract:
We present a structural clustering algorithm for large-scale datasets of small labeled graphs, utilizing a frequent subgraph sampling strategy. A set of representatives provides an intuitive description of each cluster, supports the clustering process, and helps to interpret the clustering results. The projection-based nature of the clustering approach allows us to bypass dimensionality and featur…
▽ More
We present a structural clustering algorithm for large-scale datasets of small labeled graphs, utilizing a frequent subgraph sampling strategy. A set of representatives provides an intuitive description of each cluster, supports the clustering process, and helps to interpret the clustering results. The projection-based nature of the clustering approach allows us to bypass dimensionality and feature extraction problems that arise in the context of graph datasets reduced to pairwise distances or feature vectors. While achieving high quality and (human) interpretable clusterings, the runtime of the algorithm only grows linearly with the number of graphs. Furthermore, the approach is easy to parallelize and therefore suitable for very large datasets. Our extensive experimental evaluation on synthetic and real world datasets demonstrates the superiority of our approach over existing structural and subspace clustering algorithms, both, from a runtime and quality point of view.
△ Less
Submitted 28 September, 2016;
originally announced September 2016.
-
Quickest detection in coupled systems
Authors:
Hongzhong Zhang,
Olympia Hadjiliadis,
Tobias Schäfer,
H. Vincent Poor
Abstract:
This work considers the problem of quickest detection of signals in a coupled system of $N$ sensors, which receive continuous sequential observations from the environment. It is assumed that the signals, which are modeled by general Itô processes, are coupled across sensors, but that their onset times may differ from sensor to sensor. Two main cases are considered; in the first one signal strength…
▽ More
This work considers the problem of quickest detection of signals in a coupled system of $N$ sensors, which receive continuous sequential observations from the environment. It is assumed that the signals, which are modeled by general Itô processes, are coupled across sensors, but that their onset times may differ from sensor to sensor. Two main cases are considered; in the first one signal strengths are the same across sensors while in the second one they differ by a constant. The objective is the optimal detection of the first time at which any sensor in the system receives a signal. The problem is formulated as a stochastic optimization problem in which an extended minimal Kullback-Leibler divergence criterion is used as a measure of detection delay, with a constraint on the mean time to the first false alarm. The case in which the sensors employ cumulative sum (CUSUM) strategies is considered, and it is proved that the minimum of $N$ CUSUMs is asymptotically optimal as the mean time to the first false alarm increases without bound. In particular, in the case of equal signal strengths across sensors, it is seen that the difference in detection delay of the $N$-CUSUM stop** rule and the unknown optimal stop** scheme tends to a constant related to the number of sensors as the mean time to the first false alarm increases without bound. Alternatively, in the case of unequal signal strengths, it is seen that this difference tends to zero.
△ Less
Submitted 16 March, 2014;
originally announced March 2014.
-
A state vector algebra for algorithmic implementation of second-order logic
Authors:
Dmitry Lesnik,
Tobias Schaefer
Abstract:
We present a mathematical framework for map** second-order logic relations onto a simple state vector algebra. Using this algebra, basic theorems of set theory can be proven in an algorithmic way, hence by an expert system. We illustrate the use of the algebra with simple examples and show that, in principle, all theorems of basic set theory can be recovered in an elementary way. The developed t…
▽ More
We present a mathematical framework for map** second-order logic relations onto a simple state vector algebra. Using this algebra, basic theorems of set theory can be proven in an algorithmic way, hence by an expert system. We illustrate the use of the algebra with simple examples and show that, in principle, all theorems of basic set theory can be recovered in an elementary way. The developed technique can be used for an automated theorem proving in the 1st and 2nd order logic.
△ Less
Submitted 17 November, 2015; v1 submitted 9 December, 2013;
originally announced December 2013.
-
Quickest detection in coupled systems
Authors:
Olympia Hadjiliadis,
Tobias Schaefer,
H. Vincent Poor
Abstract:
This work considers the problem of quickest detection of signals in a coupled system of N sensors, which receive continuous sequential observations from the environment. It is assumed that the signals, which are modeled a general Ito processes, are coupled across sensors, but that their onset times may differ from sensor to sensor. The objective is the optimal detection of the first time at whic…
▽ More
This work considers the problem of quickest detection of signals in a coupled system of N sensors, which receive continuous sequential observations from the environment. It is assumed that the signals, which are modeled a general Ito processes, are coupled across sensors, but that their onset times may differ from sensor to sensor. The objective is the optimal detection of the first time at which any sensor in the system receives a signal. The problem is formulated as a stochastic optimization problem in which an extended average Kullback- Leibler divergence criterion is used as a measure of detection delay, with a constraint on the mean time between false alarms. The case in which the sensors employ cumulative sum (CUSUM) strategies is considered, and it is proved that the minimum of N CUSUMs is asymptotically optimal as the mean time between false alarms increases without bound.
△ Less
Submitted 24 October, 2009;
originally announced October 2009.
