-
A Study on Optimization Techniques for Variational Quantum Circuits in Reinforcement Learning
Authors:
Michael Kölle,
Timo Witter,
Tobias Rohe,
Gerhard Stenzel,
Philipp Altmann,
Thomas Gabor
Abstract:
Quantum Computing aims to streamline machine learning, making it more effective with fewer trainable parameters. This reduction of parameters can speed up the learning process and reduce the use of computational resources. However, in the current phase of quantum computing development, known as the noisy intermediate-scale quantum era (NISQ), learning is difficult due to a limited number of qubits…
▽ More
Quantum Computing aims to streamline machine learning, making it more effective with fewer trainable parameters. This reduction of parameters can speed up the learning process and reduce the use of computational resources. However, in the current phase of quantum computing development, known as the noisy intermediate-scale quantum era (NISQ), learning is difficult due to a limited number of qubits and widespread quantum noise. To overcome these challenges, researchers are focusing on variational quantum circuits (VQCs). VQCs are hybrid algorithms that merge a quantum circuit, which can be adjusted through parameters, with traditional classical optimization techniques. These circuits require only few qubits for effective learning. Recent studies have presented new ways of applying VQCs to reinforcement learning, showing promising results that warrant further exploration. This study investigates the effects of various techniques -- data re-uploading, input scaling, output scaling -- and introduces exponential learning rate decay in the quantum proximal policy optimization algorithm's actor-VQC. We assess these methods in the popular Frozen Lake and Cart Pole environments. Our focus is on their ability to reduce the number of parameters in the VQC without losing effectiveness. Our findings indicate that data re-uploading and an exponential learning rate decay significantly enhance hyperparameter stability and overall performance. While input scaling does not improve parameter efficiency, output scaling effectively manages greediness, leading to increased learning speed and robustness.
△ Less
Submitted 20 May, 2024;
originally announced May 2024.
-
Qandle: Accelerating State Vector Simulation Using Gate-Matrix Caching and Circuit Splitting
Authors:
Gerhard Stenzel,
Sebastian Zielinski,
Michael Kölle,
Philipp Altmann,
Jonas Nüßlein,
Thomas Gabor
Abstract:
To address the computational complexity associated with state-vector simulation for quantum circuits, we propose a combination of advanced techniques to accelerate circuit execution. Quantum gate matrix caching reduces the overhead of repeated applications of the Kronecker product when applying a gate matrix to the state vector by storing decomposed partial matrices for each gate. Circuit splittin…
▽ More
To address the computational complexity associated with state-vector simulation for quantum circuits, we propose a combination of advanced techniques to accelerate circuit execution. Quantum gate matrix caching reduces the overhead of repeated applications of the Kronecker product when applying a gate matrix to the state vector by storing decomposed partial matrices for each gate. Circuit splitting divides the circuit into sub-circuits with fewer gates by constructing a dependency graph, enabling parallel or sequential execution on disjoint subsets of the state vector. These techniques are implemented using the PyTorch machine learning framework. We demonstrate the performance of our approach by comparing it to other PyTorch-compatible quantum state-vector simulators. Our implementation, named Qandle, is designed to seamlessly integrate with existing machine learning workflows, providing a user-friendly API and compatibility with the OpenQASM format. Qandle is an open-source project hosted on GitHub https://github.com/gstenzel/qandle and PyPI https://pypi.org/project/qandle/ .
△ Less
Submitted 14 April, 2024;
originally announced April 2024.
-
MEDIATE: Mutually Endorsed Distributed Incentive Acknowledgment Token Exchange
Authors:
Philipp Altmann,
Katharina Winter,
Michael Kölle,
Maximilian Zorn,
Thomy Phan,
Claudia Linnhoff-Popien
Abstract:
Recent advances in multi-agent systems (MAS) have shown that incorporating peer incentivization (PI) mechanisms vastly improves cooperation. Especially in social dilemmas, communication between the agents helps to overcome sub-optimal Nash equilibria. However, incentivization tokens need to be carefully selected. Furthermore, real-world applications might yield increased privacy requirements and l…
▽ More
Recent advances in multi-agent systems (MAS) have shown that incorporating peer incentivization (PI) mechanisms vastly improves cooperation. Especially in social dilemmas, communication between the agents helps to overcome sub-optimal Nash equilibria. However, incentivization tokens need to be carefully selected. Furthermore, real-world applications might yield increased privacy requirements and limited exchange. Therefore, we extend the PI protocol for mutual acknowledgment token exchange (MATE) and provide additional analysis on the impact of the chosen tokens. Building upon those insights, we propose mutually endorsed distributed incentive acknowledgment token exchange (MEDIATE), an extended PI architecture employing automatic token derivation via decentralized consensus. Empirical results show the stable agreement on appropriate tokens yielding superior performance compared to static tokens and state-of-the-art approaches in different social dilemma environments with various reward distributions.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learning
Authors:
Philipp Altmann,
Céline Davignon,
Maximilian Zorn,
Fabian Ritz,
Claudia Linnhoff-Popien,
Thomas Gabor
Abstract:
To enhance the interpretability of Reinforcement Learning (RL), we propose Revealing Evolutionary Action Consequence Trajectories (REACT). In contrast to the prevalent practice of validating RL models based on their optimal behavior learned during training, we posit that considering a range of edge-case trajectories provides a more comprehensive understanding of their inherent behavior. To induce…
▽ More
To enhance the interpretability of Reinforcement Learning (RL), we propose Revealing Evolutionary Action Consequence Trajectories (REACT). In contrast to the prevalent practice of validating RL models based on their optimal behavior learned during training, we posit that considering a range of edge-case trajectories provides a more comprehensive understanding of their inherent behavior. To induce such scenarios, we introduce a disturbance to the initial state, optimizing it through an evolutionary algorithm to generate a diverse population of demonstrations. To evaluate the fitness of trajectories, REACT incorporates a joint fitness function that encourages both local and global diversity in the encountered states and chosen actions. Through assessments with policies trained for varying durations in discrete and continuous environments, we demonstrate the descriptive power of REACT. Our results highlight its effectiveness in revealing nuanced aspects of RL models' behavior beyond optimal performance, thereby contributing to improved interpretability.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
A Reinforcement Learning Environment for Directed Quantum Circuit Synthesis
Authors:
Michael Kölle,
Tom Schubert,
Philipp Altmann,
Maximilian Zorn,
Jonas Stein,
Claudia Linnhoff-Popien
Abstract:
With recent advancements in quantum computing technology, optimizing quantum circuits and ensuring reliable quantum state preparation have become increasingly vital. Traditional methods often demand extensive expertise and manual calculations, posing challenges as quantum circuits grow in qubit- and gate-count. Therefore, harnessing machine learning techniques to handle the growing variety of gate…
▽ More
With recent advancements in quantum computing technology, optimizing quantum circuits and ensuring reliable quantum state preparation have become increasingly vital. Traditional methods often demand extensive expertise and manual calculations, posing challenges as quantum circuits grow in qubit- and gate-count. Therefore, harnessing machine learning techniques to handle the growing variety of gate-to-qubit combinations is a promising approach. In this work, we introduce a comprehensive reinforcement learning environment for quantum circuit synthesis, where circuits are constructed utilizing gates from the the Clifford+T gate set to prepare specific target states. Our experiments focus on exploring the relationship between the depth of synthesized quantum circuits and the circuit depths used for target initialization, as well as qubit count. We organize the environment configurations into multiple evaluation levels and include a range of well-known quantum states for benchmarking purposes. We also lay baselines for evaluating the environment using Proximal Policy Optimization. By applying the trained agents to benchmark tests, we demonstrated their ability to reliably design minimal quantum circuits for a selection of 2-qubit Bell states.
△ Less
Submitted 13 January, 2024;
originally announced January 2024.
-
Quantum Advantage Actor-Critic for Reinforcement Learning
Authors:
Michael Kölle,
Mohamad Hgog,
Fabian Ritz,
Philipp Altmann,
Maximilian Zorn,
Jonas Stein,
Claudia Linnhoff-Popien
Abstract:
Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empir…
▽ More
Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks.
△ Less
Submitted 13 January, 2024;
originally announced January 2024.
-
Challenges for Reinforcement Learning in Quantum Circuit Design
Authors:
Philipp Altmann,
Jonas Stein,
Michael Kölle,
Adelina Bärligea,
Thomas Gabor,
Thomy Phan,
Sebastian Feld,
Claudia Linnhoff-Popien
Abstract:
Quantum computing (QC) in the current NISQ era is still limited in size and precision. Hybrid applications mitigating those shortcomings are prevalent to gain early insight and advantages. Hybrid quantum machine learning (QML) comprises both the application of QC to improve machine learning (ML) and ML to improve QC architectures. This work considers the latter, leveraging reinforcement learning (…
▽ More
Quantum computing (QC) in the current NISQ era is still limited in size and precision. Hybrid applications mitigating those shortcomings are prevalent to gain early insight and advantages. Hybrid quantum machine learning (QML) comprises both the application of QC to improve machine learning (ML) and ML to improve QC architectures. This work considers the latter, leveraging reinforcement learning (RL) to improve the search for viable quantum architectures, which we formalize by a set of generic challenges. Furthermore, we propose a concrete framework, formalized as a Markov decision process, to enable learning policies capable of controlling a universal set of continuously parameterized quantum gates. Finally, we provide benchmark comparisons to assess the shortcomings and strengths of current state-of-the-art RL algorithms.
△ Less
Submitted 4 April, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
Improving Parameter Training for VQEs by Sequential Hamiltonian Assembly
Authors:
Jonas Stein,
Navid Roshani,
Maximilian Zorn,
Philipp Altmann,
Michael Kölle,
Claudia Linnhoff-Popien
Abstract:
A central challenge in quantum machine learning is the design and training of parameterized quantum circuits (PQCs). Similar to deep learning, vanishing gradients pose immense problems in the trainability of PQCs, which have been shown to arise from a multitude of sources. One such cause are non-local loss functions, that demand the measurement of a large subset of involved qubits. To facilitate t…
▽ More
A central challenge in quantum machine learning is the design and training of parameterized quantum circuits (PQCs). Similar to deep learning, vanishing gradients pose immense problems in the trainability of PQCs, which have been shown to arise from a multitude of sources. One such cause are non-local loss functions, that demand the measurement of a large subset of involved qubits. To facilitate the parameter training for quantum applications using global loss functions, we propose a Sequential Hamiltonian Assembly, which iteratively approximates the loss function using local components. Aiming for a prove of principle, we evaluate our approach using Graph Coloring problem with a Varational Quantum Eigensolver (VQE). Simulation results show, that our approach outperforms conventional parameter training by 29.99% and the empirical state of the art, Layerwise Learning, by 5.12% in the mean accuracy. This paves the way towards locality-aware learning techniques, allowing to evade vanishing gradients for a large class of practically relevant problems.
△ Less
Submitted 9 December, 2023;
originally announced December 2023.
-
Towards Transfer Learning for Large-Scale Image Classification Using Annealing-based Quantum Boltzmann Machines
Authors:
Daniëlle Schuman,
Leo Sünkel,
Philipp Altmann,
Jonas Stein,
Christoph Roch,
Thomas Gabor,
Claudia Linnhoff-Popien
Abstract:
Quantum Transfer Learning (QTL) recently gained popularity as a hybrid quantum-classical approach for image classification tasks by efficiently combining the feature extraction capabilities of large Convolutional Neural Networks with the potential benefits of Quantum Machine Learning (QML). Existing approaches, however, only utilize gate-based Variational Quantum Circuits for the quantum part of t…
▽ More
Quantum Transfer Learning (QTL) recently gained popularity as a hybrid quantum-classical approach for image classification tasks by efficiently combining the feature extraction capabilities of large Convolutional Neural Networks with the potential benefits of Quantum Machine Learning (QML). Existing approaches, however, only utilize gate-based Variational Quantum Circuits for the quantum part of these procedures. In this work we present an approach to employ Quantum Annealing (QA) in QTL-based image classification. Specifically, we propose using annealing-based Quantum Boltzmann Machines as part of a hybrid quantum-classical pipeline to learn the classification of real-world, large-scale data such as medical images through supervised training. We demonstrate our approach by applying it to the three-class COVID-CT-MD dataset, a collection of lung Computed Tomography (CT) scan slices. Using Simulated Annealing as a stand-in for actual QA, we compare our method to classical transfer learning, using a neural network of the same order of magnitude, to display its improved classification performance. We find that our approach consistently outperforms its classical baseline in terms of test accuracy and AUC-ROC-Score and needs less training epochs to do this.
△ Less
Submitted 27 November, 2023;
originally announced November 2023.
-
Disentangling Quantum and Classical Contributions in Hybrid Quantum Machine Learning Architectures
Authors:
Michael Kölle,
Jonas Maurer,
Philipp Altmann,
Leo Sünkel,
Jonas Stein,
Claudia Linnhoff-Popien
Abstract:
Quantum computing offers the potential for superior computational capabilities, particularly for data-intensive tasks. However, the current state of quantum hardware puts heavy restrictions on input size. To address this, hybrid transfer learning solutions have been developed, merging pre-trained classical models, capable of handling extensive inputs, with variational quantum circuits. Yet, it rem…
▽ More
Quantum computing offers the potential for superior computational capabilities, particularly for data-intensive tasks. However, the current state of quantum hardware puts heavy restrictions on input size. To address this, hybrid transfer learning solutions have been developed, merging pre-trained classical models, capable of handling extensive inputs, with variational quantum circuits. Yet, it remains unclear how much each component -- classical and quantum -- contributes to the model's results. We propose a novel hybrid architecture: instead of utilizing a pre-trained network for compression, we employ an autoencoder to derive a compressed version of the input data. This compressed data is then channeled through the encoder part of the autoencoder to the quantum component. We assess our model's classification capabilities against two state-of-the-art hybrid transfer learning architectures, two purely classical architectures and one quantum architecture. Their accuracy is compared across four datasets: Banknote Authentication, Breast Cancer Wisconsin, MNIST digits, and AudioMNIST. Our research suggests that classical components significantly influence classification in hybrid transfer learning, a contribution often mistakenly ascribed to the quantum element. The performance of our model aligns with that of a variational quantum circuit using amplitude embedding, positioning it as a feasible alternative.
△ Less
Submitted 13 January, 2024; v1 submitted 9 November, 2023;
originally announced November 2023.
-
Multi-Agent Quantum Reinforcement Learning using Evolutionary Optimization
Authors:
Michael Kölle,
Felix Topp,
Thomy Phan,
Philipp Altmann,
Jonas Nüßlein,
Claudia Linnhoff-Popien
Abstract:
Multi-Agent Reinforcement Learning is becoming increasingly more important in times of autonomous driving and other smart industrial applications. Simultaneously a promising new approach to Reinforcement Learning arises using the inherent properties of quantum mechanics, reducing the trainable parameters of a model significantly. However, gradient-based Multi-Agent Quantum Reinforcement Learning m…
▽ More
Multi-Agent Reinforcement Learning is becoming increasingly more important in times of autonomous driving and other smart industrial applications. Simultaneously a promising new approach to Reinforcement Learning arises using the inherent properties of quantum mechanics, reducing the trainable parameters of a model significantly. However, gradient-based Multi-Agent Quantum Reinforcement Learning methods often have to struggle with barren plateaus, holding them back from matching the performance of classical approaches. We build upon an existing approach for gradient free Quantum Reinforcement Learning and propose three genetic variations with Variational Quantum Circuits for Multi-Agent Reinforcement Learning using evolutionary optimization. We evaluate our genetic variations in the Coin Game environment and also compare them to classical approaches. We showed that our Variational Quantum Circuit approaches perform significantly better compared to a neural network with a similar amount of trainable parameters. Compared to the larger neural network, our approaches archive similar results using $97.88\%$ less parameters.
△ Less
Submitted 13 January, 2024; v1 submitted 9 November, 2023;
originally announced November 2023.
-
Benchmarking Quantum Surrogate Models on Scarce and Noisy Data
Authors:
Jonas Stein,
Michael Poppel,
Philip Adamczyk,
Ramona Fabry,
Zixin Wu,
Michael Kölle,
Jonas Nüßlein,
Daniëlle Schuman,
Philipp Altmann,
Thomas Ehmer,
Vijay Narasimhan,
Claudia Linnhoff-Popien
Abstract:
Surrogate models are ubiquitously used in industry and academia to efficiently approximate given black box functions. As state-of-the-art methods from classical machine learning frequently struggle to solve this problem accurately for the often scarce and noisy data sets in practical applications, investigating novel approaches is of great interest. Motivated by recent theoretical results indicati…
▽ More
Surrogate models are ubiquitously used in industry and academia to efficiently approximate given black box functions. As state-of-the-art methods from classical machine learning frequently struggle to solve this problem accurately for the often scarce and noisy data sets in practical applications, investigating novel approaches is of great interest. Motivated by recent theoretical results indicating that quantum neural networks (QNNs) have the potential to outperform their classical analogs in the presence of scarce and noisy data, we benchmark their qualitative performance for this scenario empirically. Our contribution displays the first application-centered approach of using QNNs as surrogate models on higher dimensional, real world data. When compared to a classical artificial neural network with a similar number of parameters, our QNN demonstrates significantly better results for noisy and scarce data, and thus motivates future work to explore this potential quantum advantage in surrogate modelling. Finally, we demonstrate the performance of current NISQ hardware experimentally and estimate the gate fidelities necessary to replicate our simulation results.
△ Less
Submitted 9 December, 2023; v1 submitted 8 June, 2023;
originally announced June 2023.
-
Combining the QAOA and HHL Algorithm to achieve a Substantial Quantum Speedup for the Unit Commitment Problem
Authors:
Jonas Stein,
Jezer Jojo,
Afrah Farea,
David Bucher,
Philipp Altmann,
M. Serdar Çelebi,
Claudia Linnhoff-Popien
Abstract:
In this paper, we propose a quantum algorithm to solve the unit commitment (UC) problem at least cubically faster than existing classical approaches. This is accomplished by calculating the energy transmission costs using the HHL algorithm inside a QAOA routine. We verify our findings experimentally using quantum circuit simulators in a small case study. Further, we postulate the applicability of…
▽ More
In this paper, we propose a quantum algorithm to solve the unit commitment (UC) problem at least cubically faster than existing classical approaches. This is accomplished by calculating the energy transmission costs using the HHL algorithm inside a QAOA routine. We verify our findings experimentally using quantum circuit simulators in a small case study. Further, we postulate the applicability of the concepts developed for this algorithm to be used for a large class of optimization problems that demand solving a linear system of equations in order to calculate the cost function for a given solution.
△ Less
Submitted 15 June, 2023; v1 submitted 15 May, 2023;
originally announced May 2023.
-
CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing
Authors:
Philipp Altmann,
Fabian Ritz,
Leonard Feuchtinger,
Jonas Nüßlein,
Claudia Linnhoff-Popien,
Thomy Phan
Abstract:
The safe application of reinforcement learning (RL) requires generalization from limited training data to unseen scenarios. Yet, fulfilling tasks under changing circumstances is a key challenge in RL. Current state-of-the-art approaches for generalization apply data augmentation techniques to increase the diversity of training data. Even though this prevents overfitting to the training environment…
▽ More
The safe application of reinforcement learning (RL) requires generalization from limited training data to unseen scenarios. Yet, fulfilling tasks under changing circumstances is a key challenge in RL. Current state-of-the-art approaches for generalization apply data augmentation techniques to increase the diversity of training data. Even though this prevents overfitting to the training environment(s), it hinders policy optimization. Crafting a suitable observation, only containing crucial information, has been shown to be a challenging task itself. To improve data efficiency and generalization capabilities, we propose Compact Reshaped Observation Processing (CROP) to reduce the state information used for policy optimization. By providing only relevant information, overfitting to a specific training layout is precluded and generalization to unseen environments is improved. We formulate three CROPs that can be applied to fully observable observation- and action-spaces and provide methodical foundation. We empirically show the improvements of CROP in a distributionally shifted safety gridworld. We furthermore provide benchmark comparisons to full observability and data-augmentation in two different-sized procedurally generated mazes.
△ Less
Submitted 5 December, 2023; v1 submitted 26 April, 2023;
originally announced April 2023.
-
DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training
Authors:
Philipp Altmann,
Thomy Phan,
Fabian Ritz,
Thomas Gabor,
Claudia Linnhoff-Popien
Abstract:
We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated b…
▽ More
We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator's verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a surrogate, steering policy optimization towards more valuable regions of the reward landscape thus learning an optimal policy. Our results show that DIRECT outperforms state-of-the-art algorithms in sparse- and shifting-reward environments being able to provide a surrogate reward to the policy and direct the optimization towards valuable areas.
△ Less
Submitted 18 January, 2023;
originally announced January 2023.
-
Learning to Participate through Trading of Reward Shares
Authors:
Michael Kölle,
Tim Matheis,
Philipp Altmann,
Kyrill Schmid
Abstract:
Enabling autonomous agents to act cooperatively is an important step to integrate artificial intelligence in our daily lives. While some methods seek to stimulate cooperation by letting agents give rewards to others, in this paper we propose a method inspired by the stock market, where agents have the opportunity to participate in other agents' returns by acquiring reward shares. Intuitively, an a…
▽ More
Enabling autonomous agents to act cooperatively is an important step to integrate artificial intelligence in our daily lives. While some methods seek to stimulate cooperation by letting agents give rewards to others, in this paper we propose a method inspired by the stock market, where agents have the opportunity to participate in other agents' returns by acquiring reward shares. Intuitively, an agent may learn to act according to the common interest when being directly affected by the other agents' rewards. The empirical results of the tested general-sum Markov games show that this mechanism promotes cooperative policies among independently trained agents in social dilemma situations. Moreover, as demonstrated in a temporally and spatially extended domain, participation can lead to the development of roles and the division of subtasks between the agents.
△ Less
Submitted 18 January, 2023;
originally announced January 2023.
-
SEQUENT: Towards Traceable Quantum Machine Learning using Sequential Quantum Enhanced Training
Authors:
Philipp Altmann,
Leo Sünkel,
Jonas Stein,
Tobias Müller,
Christoph Roch,
Claudia Linnhoff-Popien
Abstract:
Applying new computing paradigms like quantum computing to the field of machine learning has recently gained attention. However, as high-dimensional real-world applications are not yet feasible to be solved using purely quantum hardware, hybrid methods using both classical and quantum machine learning paradigms have been proposed. For instance, transfer learning methods have been shown to be succe…
▽ More
Applying new computing paradigms like quantum computing to the field of machine learning has recently gained attention. However, as high-dimensional real-world applications are not yet feasible to be solved using purely quantum hardware, hybrid methods using both classical and quantum machine learning paradigms have been proposed. For instance, transfer learning methods have been shown to be successfully applicable to hybrid image classification tasks. Nevertheless, beneficial circuit architectures still need to be explored. Therefore, tracing the impact of the chosen circuit architecture and parameterization is crucial for the development of beneficially applicable hybrid methods. However, current methods include processes where both parts are trained concurrently, therefore not allowing for a strict separability of classical and quantum impact. Thus, those architectures might produce models that yield a superior prediction accuracy whilst employing the least possible quantum impact. To tackle this issue, we propose Sequential Quantum Enhanced Training (SEQUENT) an improved architecture and training process for the traceable application of quantum computing methods to hybrid machine learning. Furthermore, we provide formal evidence for the disadvantage of current methods and preliminary experimental results as a proof-of-concept for the applicability of SEQUENT.
△ Less
Submitted 26 April, 2023; v1 submitted 6 January, 2023;
originally announced January 2023.
-
Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability
Authors:
Thomy Phan,
Fabian Ritz,
Philipp Altmann,
Maximilian Zorn,
Jonas Nüßlein,
Michael Kölle,
Thomas Gabor,
Claudia Linnhoff-Popien
Abstract:
Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attent…
▽ More
Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attention-based Embeddings of Recurrence In multi-Agent Learning (AERIAL) to approximate value functions under stochastic partial observability. AERIAL replaces the true state with a learned representation of multi-agent recurrence, considering more accurate information about decentralized agent decisions than state-based CTDE. We then introduce MessySMAC, a modified version of SMAC with stochastic observations and higher variance in initial states, to provide a more general and configurable benchmark regarding stochastic partial observability. We evaluate AERIAL in Dec-Tiger as well as in a variety of SMAC and MessySMAC maps, and compare the results with state-based CTDE. Furthermore, we evaluate the robustness of AERIAL and state-based CTDE against various stochasticity configurations in MessySMAC.
△ Less
Submitted 27 December, 2023; v1 submitted 4 January, 2023;
originally announced January 2023.
-
Capturing Dependencies within Machine Learning via a Formal Process Model
Authors:
Fabian Ritz,
Thomy Phan,
Andreas Sedlmeier,
Philipp Altmann,
Jan Wieghardt,
Reiner Schmid,
Horst Sauer,
Cornel Klein,
Claudia Linnhoff-Popien,
Thomas Gabor
Abstract:
The development of Machine Learning (ML) models is more than just a special case of software development (SD): ML models acquire properties and fulfill requirements even without direct human interaction in a seemingly uncontrollable manner. Nonetheless, the underlying processes can be described in a formal way. We define a comprehensive SD process model for ML that encompasses most tasks and artif…
▽ More
The development of Machine Learning (ML) models is more than just a special case of software development (SD): ML models acquire properties and fulfill requirements even without direct human interaction in a seemingly uncontrollable manner. Nonetheless, the underlying processes can be described in a formal way. We define a comprehensive SD process model for ML that encompasses most tasks and artifacts described in the literature in a consistent way. In addition to the production of the necessary artifacts, we also focus on generating and validating fitting descriptions in the form of specifications. We stress the importance of further evolving the ML model throughout its life-cycle even after initial training and testing. Thus, we provide various interaction points with standard SD processes in which ML often is an encapsulated task. Further, our SD process model allows to formulate ML as a (meta-) optimization problem. If automated rigorously, it can be used to realize self-adaptive autonomous systems. Finally, our SD process model features a description of time that allows to reason about the progress within ML development processes. This might lead to further applications of formal methods within the field of ML.
△ Less
Submitted 10 August, 2022;
originally announced August 2022.
-
Towards Multi-Agent Reinforcement Learning using Quantum Boltzmann Machines
Authors:
Tobias Müller,
Christoph Roch,
Kyrill Schmid,
Philipp Altmann
Abstract:
Reinforcement learning has driven impressive advances in machine learning. Simultaneously, quantum-enhanced machine learning algorithms using quantum annealing underlie heavy developments. Recently, a multi-agent reinforcement learning (MARL) architecture combining both paradigms has been proposed. This novel algorithm, which utilizes Quantum Boltzmann Machines (QBMs) for Q-value approximation has…
▽ More
Reinforcement learning has driven impressive advances in machine learning. Simultaneously, quantum-enhanced machine learning algorithms using quantum annealing underlie heavy developments. Recently, a multi-agent reinforcement learning (MARL) architecture combining both paradigms has been proposed. This novel algorithm, which utilizes Quantum Boltzmann Machines (QBMs) for Q-value approximation has outperformed regular deep reinforcement learning in terms of time-steps needed to converge. However, this algorithm was restricted to single-agent and small 2x2 multi-agent grid domains. In this work, we propose an extension to the original concept in order to solve more challenging problems. Similar to classic DQNs, we add an experience replay buffer and use different networks for approximating the target and policy values. The experimental results show that learning becomes more stable and enables agents to find optimal policies in grid-domains with higher complexity. Additionally, we assess how parameter sharing influences the agents behavior in multi-agent domains. Quantum sampling proves to be a promising method for reinforcement learning tasks, but is currently limited by the QPU size and therefore by the size of the input and Boltzmann machine.
△ Less
Submitted 22 November, 2021; v1 submitted 22 September, 2021;
originally announced September 2021.
-
Benchmarking Surrogate-Assisted Genetic Recommender Systems
Authors:
Thomas Gabor,
Philipp Altmann
Abstract:
We propose a new approach for building recommender systems by adapting surrogate-assisted interactive genetic algorithms. A pool of user-evaluated items is used to construct an approximative model which serves as a surrogate fitness function in a genetic algorithm for optimizing new suggestions. The surrogate is used to recommend new items to the user, which are then evaluated according to the use…
▽ More
We propose a new approach for building recommender systems by adapting surrogate-assisted interactive genetic algorithms. A pool of user-evaluated items is used to construct an approximative model which serves as a surrogate fitness function in a genetic algorithm for optimizing new suggestions. The surrogate is used to recommend new items to the user, which are then evaluated according to the user's liking and subsequently removed from the search space. By updating the surrogate model after new recommendations have been evaluated by the user, we enable the model itself to evolve towards the user's preferences. In order to precisely evaluate the performance of that approach, the human's subjective evaluation is replaced by common continuous objective benchmark functions for evolutionary algorithms. The system's performance is compared to a conventional genetic algorithm and random search. We show that given a very limited amount of allowed evaluations on the true objective, our approach outperforms these baseline methods.
△ Less
Submitted 7 August, 2019;
originally announced August 2019.
