-
Augmenting Replay in World Models for Continual Reinforcement Learning
Authors:
Luke Yang,
Levin Kuhlmann,
Gideon Kowadlo
Abstract:
Continual RL is a challenging problem where the agent is exposed to a sequence of tasks; it should learn new tasks without forgetting old ones, and learning the new task should improve performance on previous and future tasks. The most common approaches use model-free RL algorithms as a base, and replay buffers have been used to overcome catastrophic forgetting. However, the buffers are often very…
▽ More
Continual RL is a challenging problem where the agent is exposed to a sequence of tasks; it should learn new tasks without forgetting old ones, and learning the new task should improve performance on previous and future tasks. The most common approaches use model-free RL algorithms as a base, and replay buffers have been used to overcome catastrophic forgetting. However, the buffers are often very large making scalability difficult. Also, the concept of replay comes from biological inspiration, where evidence suggests that replay is applied to a world model, which implies model-based RL -- and model-based RL should have benefits for continual RL, where it is possible to exploit knowledge independent of the policy. We present WMAR, World Models with Augmented Replay, a model-based RL algorithm with a world model and memory efficient distribution matching replay buffer. It is based on the well-known DreamerV3 algorithm, which has a simple FIFO buffer and was not tested in a continual RL setting. We evaluated WMAR vs WMAR (FIFO only) on tasks with and without shared structure from OpenAI ProcGen and Atari respectively, and without a task oracle. We found that WMAR has favourable properties on continual RL with significantly reduced computational overhead compared to WMAR (FIFO only). WMAR had small benefits over DreamerV3 on tasks with shared structure and substantially better forgetting characteristics on tasks without shared structure; but at the cost of lower plasticity seen in a lower maximum on new tasks. The results suggest that model-based RL using a world model with a memory efficient replay buffer can be an effective and practical approach to continual RL, justifying future work.
△ Less
Submitted 4 March, 2024; v1 submitted 29 January, 2024;
originally announced January 2024.
-
MachineLearnAthon: An Action-Oriented Machine Learning Didactic Concept
Authors:
Michal Tkáč,
Jakub Sieber,
Lara Kuhlmann,
Matthias Brueggenolte,
Alexandru Rinciog,
Michael Henke,
Artur M. Schweidtmann,
Qinghe Gao,
Maximilian F. Theisen,
Radwa El Shawi
Abstract:
Machine Learning (ML) techniques are encountered nowadays across disciplines, from social sciences, through natural sciences to engineering. The broad application of ML and the accelerated pace of its evolution lead to an increasing need for dedicated teaching concepts aimed at making the application of this technology more reliable and responsible. However, teaching ML is a daunting task. Aside f…
▽ More
Machine Learning (ML) techniques are encountered nowadays across disciplines, from social sciences, through natural sciences to engineering. The broad application of ML and the accelerated pace of its evolution lead to an increasing need for dedicated teaching concepts aimed at making the application of this technology more reliable and responsible. However, teaching ML is a daunting task. Aside from the methodological complexity of ML algorithms, both with respect to theory and implementation, the interdisciplinary and empirical nature of the field need to be taken into consideration. This paper introduces the MachineLearnAthon format, an innovative didactic concept designed to be inclusive for students of different disciplines with heterogeneous levels of mathematics, programming and domain expertise. At the heart of the concept lie ML challenges, which make use of industrial data sets to solve real-world problems. These cover the entire ML pipeline, promoting data literacy and practical skills, from data preparation, through deployment, to evaluation.
△ Less
Submitted 29 January, 2024;
originally announced January 2024.
-
Left/Right Brain, human motor control and the implications for robotics
Authors:
Jarrad Rinaldo,
Levin Kuhlmann,
Jason Friedman,
Gideon Kowadlo
Abstract:
Neural Network movement controllers promise a variety of advantages over conventional control methods, however, they are not widely adopted due to their inability to produce reliably precise movements. This research explores a bilateral neural network architecture as a control system for motor tasks. We aimed to achieve hemispheric specialisation similar to what is observed in humans across differ…
▽ More
Neural Network movement controllers promise a variety of advantages over conventional control methods, however, they are not widely adopted due to their inability to produce reliably precise movements. This research explores a bilateral neural network architecture as a control system for motor tasks. We aimed to achieve hemispheric specialisation similar to what is observed in humans across different tasks; the dominant system (usually the right hand, left hemisphere) excels at tasks involving coordination and efficiency of movement, and the non-dominant system performs better at tasks requiring positional stability. Specialisation was achieved by training the hemispheres with different loss functions tailored to the expected behaviour of the respective hemispheres. We compared bilateral models with and without specialised hemispheres, with and without inter-hemispheric connectivity (representing the biological Corpus Callosum), and unilateral models with and without specialisation. The models were trained and tested on two tasks common in the human motor control literature: the random reach task, suited to the dominant system, a model with better coordination, and the hold position task, suited to the non-dominant system, a model with more stable movement. Each system outperformed the non-preferred system in its preferred task. For both tasks, a bilateral model outperformed the non-preferred hand and was as good or better than the preferred hand. The results suggest that the hemispheres could collaborate on tasks or work independently to their strengths. This study provides ideas for how a biologically inspired bilateral architecture could be exploited for industrial motor control.
△ Less
Submitted 10 July, 2024; v1 submitted 25 January, 2024;
originally announced January 2024.
-
RODD: Robust Outlier Detection in Data Cubes
Authors:
Lara Kuhlmann,
Daniel Wilmes,
Emmanuel Müller,
Markus Pauly,
Daniel Horn
Abstract:
Data cubes are multidimensional databases, often built from several separate databases, that serve as flexible basis for data analysis. Surprisingly, outlier detection on data cubes has not yet been treated extensively. In this work, we provide the first framework to evaluate robust outlier detection methods in data cubes (RODD). We introduce a novel random forest-based outlier detection approach…
▽ More
Data cubes are multidimensional databases, often built from several separate databases, that serve as flexible basis for data analysis. Surprisingly, outlier detection on data cubes has not yet been treated extensively. In this work, we provide the first framework to evaluate robust outlier detection methods in data cubes (RODD). We introduce a novel random forest-based outlier detection approach (RODD-RF) and compare it with more traditional methods based on robust location estimators. We propose a general type of test data and examine all methods in a simulation study. Moreover, we apply ROOD-RF to real world data. The results show that RODD-RF can lead to improved outlier detection.
△ Less
Submitted 14 March, 2023;
originally announced March 2023.
-
Coherent False Seizure Prediction in Epilepsy, Coincidence or Providence?
Authors:
Jens Müller,
Hongliu Yang,
Matthias Eberlein,
Georg Leonhardt,
Ortrud Uckermann,
Levin Kuhlmann,
Ronald Tetzlaff
Abstract:
Seizure forecasting using machine learning is possible, but the performance is far from ideal, as indicated by many false predictions and low specificity. Here, we examine false and missing alarms of two algorithms on long-term datasets to show that the limitations are less related to classifiers or features, but rather to intrinsic changes in the data. We evaluated two algorithms on three dataset…
▽ More
Seizure forecasting using machine learning is possible, but the performance is far from ideal, as indicated by many false predictions and low specificity. Here, we examine false and missing alarms of two algorithms on long-term datasets to show that the limitations are less related to classifiers or features, but rather to intrinsic changes in the data. We evaluated two algorithms on three datasets by computing the correlation of false predictions and estimating the information transfer between both classification methods. For 9 out of 12 individuals both methods showed a performance better than chance. For all individuals we observed a positive correlation in predictions. For individuals with strong correlation in false predictions we were able to boost the performance of one method by excluding test samples based on the results of the second method. Substantially different algorithms exhibit a highly consistent performance and a strong coherency in false and missing alarms. Hence, changing the underlying hypothesis of a preictal state of fixed time length prior to each seizure to a proictal state is more helpful than further optimizing classifiers. The outcome is significant for the evaluation of seizure prediction algorithms on continuous data.
△ Less
Submitted 26 October, 2021;
originally announced October 2021.
-
A Generative Model to Synthesize EEG Data for Epileptic Seizure Prediction
Authors:
Khansa Rasheed,
Junaid Qadir,
Terence J. O'Brien,
Levin Kuhlmann,
Adeel Razi
Abstract:
Prediction of seizure before they occur is vital for bringing normalcy to the lives of patients. Researchers employed machine learning methods using hand-crafted features for seizure prediction. However, ML methods are too complicated to select the best ML model or best features. Deep Learning methods are beneficial in the sense of automatic feature extraction. One of the roadblocks for accurate s…
▽ More
Prediction of seizure before they occur is vital for bringing normalcy to the lives of patients. Researchers employed machine learning methods using hand-crafted features for seizure prediction. However, ML methods are too complicated to select the best ML model or best features. Deep Learning methods are beneficial in the sense of automatic feature extraction. One of the roadblocks for accurate seizure prediction is scarcity of epileptic seizure data. This paper addresses this problem by proposing a deep convolutional generative adversarial network to generate synthetic EEG samples. We use two methods to validate synthesized data namely, one-class SVM and a new proposal which we refer to as convolutional epileptic seizure predictor (CESP). Another objective of our study is to evaluate performance of well-known deep learning models (e.g., VGG16, VGG19, ResNet50, and Inceptionv3) by training models on augmented data using transfer learning with average time of 10 min between true prediction and seizure onset. Our results show that CESP model achieves sensitivity of 78.11% and 88.21%, and FPR of 0.27/h and 0.14/h for training on synthesized and testing on real Epilepsyecosystem and CHB-MIT datasets, respectively. Effective results of CESP trained on synthesized data shows that synthetic data acquired the correlation between features and labels very well. We also show that employment of idea of transfer learning and data augmentation in patient-specific manner provides highest accuracy with sensitivity of 90.03% and 0.03 FPR/h which was achieved using Inceptionv3, and that augmenting data with samples generated from DCGAN increased prediction results of our CESP model and Inceptionv3 by 4-5% as compared to state-of-the-art traditional augmentation techniques. Finally, we note that prediction results of CESP achieved by using augmented data are better than chance level for both datasets.
△ Less
Submitted 1 December, 2020;
originally announced December 2020.
-
Machine Learning for Predicting Epileptic Seizures Using EEG Signals: A Review
Authors:
Khansa Rasheed,
Adnan Qayyum,
Junaid Qadir,
Shobi Sivathamboo,
Patrick Kwan,
Levin Kuhlmann,
Terence O'Brien,
Adeel Razi
Abstract:
With the advancement in artificial intelligence (AI) and machine learning (ML) techniques, researchers are striving towards employing these techniques for advancing clinical practice. One of the key objectives in healthcare is the early detection and prediction of disease to timely provide preventive interventions. This is especially the case for epilepsy, which is characterized by recurrent and u…
▽ More
With the advancement in artificial intelligence (AI) and machine learning (ML) techniques, researchers are striving towards employing these techniques for advancing clinical practice. One of the key objectives in healthcare is the early detection and prediction of disease to timely provide preventive interventions. This is especially the case for epilepsy, which is characterized by recurrent and unpredictable seizures. Patients can be relieved from the adverse consequences of epileptic seizures if it could somehow be predicted in advance. Despite decades of research, seizure prediction remains an unsolved problem. This is likely to remain at least partly because of the inadequate amount of data to resolve the problem. There have been exciting new developments in ML-based algorithms that have the potential to deliver a paradigm shift in the early and accurate prediction of epileptic seizures. Here we provide a comprehensive review of state-of-the-art ML techniques in early prediction of seizures using EEG signals. We will identify the gaps, challenges, and pitfalls in the current research and recommend future directions.
△ Less
Submitted 4 February, 2020;
originally announced February 2020.
-
Human Intracranial EEG Quantitative Analysis and Automatic Feature Learning for Epileptic Seizure Prediction
Authors:
Ramy Hussein,
Mohamed Osama Ahmed,
Rabab Ward,
Z. Jane Wang,
Levin Kuhlmann,
Yi Guo
Abstract:
Objective: The aim of this study is to develop an efficient and reliable epileptic seizure prediction system using intracranial EEG (iEEG) data, especially for people with drug-resistant epilepsy. The prediction procedure should yield accurate results in a fast enough fashion to alert patients of impending seizures. Methods: We quantitatively analyze the human iEEG data to obtain insights into how…
▽ More
Objective: The aim of this study is to develop an efficient and reliable epileptic seizure prediction system using intracranial EEG (iEEG) data, especially for people with drug-resistant epilepsy. The prediction procedure should yield accurate results in a fast enough fashion to alert patients of impending seizures. Methods: We quantitatively analyze the human iEEG data to obtain insights into how the human brain behaves before and between epileptic seizures. We then introduce an efficient pre-processing method for reducing the data size and converting the time-series iEEG data into an image-like format that can be used as inputs to convolutional neural networks (CNNs). Further, we propose a seizure prediction algorithm that uses cooperative multi-scale CNNs for automatic feature learning of iEEG data. Results: 1) iEEG channels contain complementary information and excluding individual channels is not advisable to retain the spatial information needed for accurate prediction of epileptic seizures. 2) The traditional PCA is not a reliable method for iEEG data reduction in seizure prediction. 3) Hand-crafted iEEG features may not be suitable for reliable seizure prediction performance as the iEEG data varies between patients and over time for the same patient. 4) Seizure prediction results show that our algorithm outperforms existing methods by achieving an average sensitivity of 87.85% and AUC score of 0.84. Conclusion: Understanding how the human brain behaves before seizure attacks and far from them facilitates better designs of epileptic seizure predictors. Significance: Accurate seizure prediction algorithms can warn patients about the next seizure attack so they could avoid dangerous activities. Medications could then be administered to abort the impending seizure and minimize the risk of injury.
△ Less
Submitted 7 April, 2019;
originally announced April 2019.
-
Convolutional Neural Networks for Epileptic Seizure Prediction
Authors:
Matthias Eberlein,
Raphael Hildebrand,
Ronald Tetzlaff,
Nico Hoffmann,
Levin Kuhlmann,
Benjamin Brinkmann,
Jens Müller
Abstract:
Epilepsy is the most common neurological disorder and an accurate forecast of seizures would help to overcome the patient's uncertainty and helplessness. In this contribution, we present and discuss a novel methodology for the classification of intracranial electroencephalography (iEEG) for seizure prediction. Contrary to previous approaches, we categorically refrain from an extraction of hand-cra…
▽ More
Epilepsy is the most common neurological disorder and an accurate forecast of seizures would help to overcome the patient's uncertainty and helplessness. In this contribution, we present and discuss a novel methodology for the classification of intracranial electroencephalography (iEEG) for seizure prediction. Contrary to previous approaches, we categorically refrain from an extraction of hand-crafted features and use a convolutional neural network (CNN) topology instead for both the determination of suitable signal characteristics and the binary classification of preictal and interictal segments. Three different models have been evaluated on public datasets with long-term recordings from four dogs and three patients. Overall, our findings demonstrate the general applicability. In this work we discuss the strengths and limitations of our methodology.
△ Less
Submitted 11 April, 2023; v1 submitted 2 November, 2018;
originally announced November 2018.
-
Semi-supervised Seizure Prediction with Generative Adversarial Networks
Authors:
Nhan Duy Truong,
Levin Kuhlmann,
Mohammad Reza Bonyadi,
Omid Kavehei
Abstract:
In this article, we propose an approach that can make use of not only labeled EEG signals but also the unlabeled ones which is more accessible. We also suggest the use of data fusion to further improve the seizure prediction accuracy. Data fusion in our vision includes EEG signals, cardiogram signals, body temperature and time. We use the short-time Fourier transform on 28-s EEG windows as a pre-p…
▽ More
In this article, we propose an approach that can make use of not only labeled EEG signals but also the unlabeled ones which is more accessible. We also suggest the use of data fusion to further improve the seizure prediction accuracy. Data fusion in our vision includes EEG signals, cardiogram signals, body temperature and time. We use the short-time Fourier transform on 28-s EEG windows as a pre-processing step. A generative adversarial network (GAN) is trained in an unsupervised manner where information of seizure onset is disregarded. The trained Discriminator of the GAN is then used as feature extractor. Features generated by the feature extractor are classified by two fully-connected layers (can be replaced by any classifier) for the labeled EEG signals. This semi-supervised seizure prediction method achieves area under the operating characteristic curve (AUC) of 77.68% and 75.47% for the CHBMIT scalp EEG dataset and the Freiburg Hospital intracranial EEG dataset, respectively. Unsupervised training without the need of labeling is important because not only it can be performed in real-time during EEG signal recording, but also it does not require feature engineering effort for each patient.
△ Less
Submitted 20 June, 2018;
originally announced June 2018.
-
A Generalised Seizure Prediction with Convolutional Neural Networks for Intracranial and Scalp Electroencephalogram Data Analysis
Authors:
Nhan Duy Truong,
Anh Duy Nguyen,
Levin Kuhlmann,
Mohammad Reza Bonyadi,
Jiawei Yang,
Omid Kavehei
Abstract:
Seizure prediction has attracted a growing attention as one of the most challenging predictive data analysis efforts in order to improve the life of patients living with drug-resistant epilepsy and tonic seizures. Many outstanding works have been reporting great results in providing a sensible indirect (warning systems) or direct (interactive neural-stimulation) control over refractory seizures, s…
▽ More
Seizure prediction has attracted a growing attention as one of the most challenging predictive data analysis efforts in order to improve the life of patients living with drug-resistant epilepsy and tonic seizures. Many outstanding works have been reporting great results in providing a sensible indirect (warning systems) or direct (interactive neural-stimulation) control over refractory seizures, some of which achieved high performance. However, many works put heavily handcraft feature extraction and/or carefully tailored feature engineering to each patient to achieve very high sensitivity and low false prediction rate for a particular dataset. This limits the benefit of their approaches if a different dataset is used. In this paper we apply Convolutional Neural Networks (CNNs) on different intracranial and scalp electroencephalogram (EEG) datasets and proposed a generalized retrospective and patient-specific seizure prediction method. We use Short-Time Fourier Transform (STFT) on 30-second EEG windows with 50% overlap** to extract information in both frequency and time domains. A standardization step is then applied on STFT components across the whole frequency range to prevent high frequencies features being influenced by those at lower frequencies. A convolutional neural network model is used for both feature extraction and classification to separate preictal segments from interictal ones. The proposed approach achieves sensitivity of 81.4%, 81.2%, 82.3% and false prediction rate (FPR) of 0.06/h, 0.16/h, 0.22/h on Freiburg Hospital intracranial EEG (iEEG) dataset, Children's Hospital of Boston-MIT scalp EEG (sEEG) dataset, and Kaggle American Epilepsy Society Seizure Prediction Challenge's dataset, respectively. Our prediction method is also statistically better than an unspecific random predictor for most of patients in all three datasets.
△ Less
Submitted 6 December, 2017; v1 submitted 6 July, 2017;
originally announced July 2017.
-
Supervised Learning in Automatic Channel Selection for Epileptic Seizure Detection
Authors:
Nhan Truong,
Levin Kuhlmann,
Mohammad Reza Bonyadi,
Jiawei Yang,
Andrew Faulks,
Omid Kavehei
Abstract:
Detecting seizure using brain neuroactivations recorded by intracranial electroencephalogram (iEEG) has been widely used for monitoring, diagnosing, and closed-loop therapy of epileptic patients, however, computational efficiency gains are needed if state-of-the-art methods are to be implemented in implanted devices. We present a novel method for automatic seizure detection based on iEEG data that…
▽ More
Detecting seizure using brain neuroactivations recorded by intracranial electroencephalogram (iEEG) has been widely used for monitoring, diagnosing, and closed-loop therapy of epileptic patients, however, computational efficiency gains are needed if state-of-the-art methods are to be implemented in implanted devices. We present a novel method for automatic seizure detection based on iEEG data that outperforms current state-of-the-art seizure detection methods in terms of computational efficiency while maintaining the accuracy. The proposed algorithm incorporates an automatic channel selection (ACS) engine as a pre-processing stage to the seizure detection procedure. The ACS engine consists of supervised classifiers which aim to find iEEGchannelswhich contribute the most to a seizure. Seizure detection stage involves feature extraction and classification. Feature extraction is performed in both frequency and time domains where spectral power and correlation between channel pairs are calculated. Random Forest is used in classification of interictal, ictal and early ictal periods of iEEG signals. Seizure detection in this paper is retrospective and patient-specific. iEEG data is accessed via Kaggle, provided by International Epilepsy Electro-physiology Portal. The dataset includes a training set of 6.5 hours of interictal data and 41 minin ictal data and a test set of 9.14 hours. Compared to the state-of-the-art on the same dataset, we achieve 49.4% increase in computational efficiency and 400 mins better in average for detection delay. The proposed model is able to detect a seizure onset at 91.95% sensitivity and 94.05% specificity with a mean detection delay of 2.77 s. The area under the curve (AUC) is 96.44%, that is comparable to the current state-of-the-art with AUC of 96.29%.
△ Less
Submitted 31 January, 2017;
originally announced January 2017.
-
Parameter and state estimation of nonlinear systems using a multi-observer under the supervisory framework
Authors:
Michelle S. Chong,
Dragan Nešić,
Romain Postoyan,
Levin Kuhlmann
Abstract:
We present a hybrid scheme for the parameter and state estimation of nonlinear continuous-time systems, which is inspired by the supervisory setup used for control. State observers are synthesized for some nominal parameter values and a criterion is designed to select one of these observers at any given time instant, which provides state and parameter estimates. Assuming that a persistency of exci…
▽ More
We present a hybrid scheme for the parameter and state estimation of nonlinear continuous-time systems, which is inspired by the supervisory setup used for control. State observers are synthesized for some nominal parameter values and a criterion is designed to select one of these observers at any given time instant, which provides state and parameter estimates. Assuming that a persistency of excitation condition holds, the convergence of the parameter and state estimation errors to zero is ensured up to a margin, which can be made as small as desired by increasing the number of observers. To reduce the potential computational complexity of the scheme, we explain how the sampling of the parameter set can be dynamically updated using a zoom-in procedure. This strategy typically requires a fewer number of observers for a given estimation error margin compared to the static sampling policy. The results are shown to be applicable to linear systems and to a class of nonlinear systems. We illustrate the applicability of the approach by estimating the synaptic gains and the mean membrane potentials of a neural mass model.
△ Less
Submitted 18 March, 2014;
originally announced March 2014.
