Showing 1–50 of 78 results for author: Linnhoff-Popien, C

Evaluating Quantum Optimization for Dynamic Self-Reliant Community Detection

Authors: David Bucher, Daniel Porawski, Benedikt Wimmer, Jonas Nüßlein, Corey O'Meara, Naeimeh Mohseni, Giorgio Cortiana, Claudia Linnhoff-Popien

Abstract: Power grid partitioning is an important requirement for resilient distribution grids. Since electricity production is progressively shifted to the distribution side, dynamic identification of self-reliant grid subsets becomes crucial for operation. This problem can be represented as a modification to the well-known NP-hard Community Detection (CD) problem. We formulate it as a Quadratic Unconstrai… ▽ More Power grid partitioning is an important requirement for resilient distribution grids. Since electricity production is progressively shifted to the distribution side, dynamic identification of self-reliant grid subsets becomes crucial for operation. This problem can be represented as a modification to the well-known NP-hard Community Detection (CD) problem. We formulate it as a Quadratic Unconstrained Binary Optimization (QUBO) problem suitable for solving using quantum computation{\color{blue}, which is expected to find better-quality partitions faster. The formulation aims to find communities with maximal self-sufficiency and minimal power flowing between them}. To assess quantum optimization for sizeable problems, we develop a hierarchical divisive method that solves sub-problem QUBOs to perform grid bisections. Furthermore, we propose a customization of the Louvain heuristic that includes self-reliance. In the evaluation, we first demonstrate that this problem examines exponential runtime scaling classically. Then, using different IEEE power system test cases, we benchmark the solution quality for multiple approaches: D-Wave's hybrid quantum-classical solvers, classical heuristics, and a branch-and-bound solver. As a result, we observe that the hybrid solvers provide very promising results, both with and without the divisive algorithm, regarding solution quality achieved within a given time frame. Directly utilizing D-Wave's Quantum Annealing (QA) hardware shows inferior partitioning. △ Less

Submitted 9 July, 2024; originally announced July 2024.

From Problem to Solution: A general Pipeline to Solve Optimisation Problems on Quantum Hardware

Authors: Tobias Rohe, Simon Grätz, Michael Kölle, Sebastian Zielinski, Jonas Stein, Claudia Linnhoff-Popien

Abstract: With constant improvements of quantum hardware and quantum algorithms, quantum advantage comes within reach. Parallel to the development of the computer at the end of the twentieth century, quantum software development will now also rapidly gain in importance and scale. On account of the inherent complexity and novelty of quantum computing (QC), as well as the expected lack of expertise of many of… ▽ More With constant improvements of quantum hardware and quantum algorithms, quantum advantage comes within reach. Parallel to the development of the computer at the end of the twentieth century, quantum software development will now also rapidly gain in importance and scale. On account of the inherent complexity and novelty of quantum computing (QC), as well as the expected lack of expertise of many of the stakeholders involved in its development, QC software development projects are exposed to the risk of being conducted in a crowded and unstructured way, lacking clear guidance and understanding. This paper presents a comprehensive quantum optimisation development pipeline, novel in its depth of 22 activities across multiple stages, coupled with project management insights, uniquely targeted to the late noisy intermediate-scale quantum (NISQ) [1] and early post-NISQ eras. We have extensively screened literature and use-cases, interviewed experts, and brought in our own expertise to develop this general quantum pipeline. The proposed solution pipeline is divided into five stages: Use-case Identification, Solution Draft, Pre-Processing, Execution and Post-Processing. Additionally, the pipeline contains two review points to address the project management view, the inherent risk of the project and the current technical maturity of QC technology. This work is intended as an orientation aid for all stakeholders involved in the development of QC applications and should therefore increase the chances of success of quantum software projects. We encourage researchers to adapt and extend the model where appropriate, as technological development also continues. △ Less

Submitted 28 June, 2024; originally announced June 2024.

Scaling of symmetry-restricted quantum circuits

Authors: Maximilian Balthasar Mansky, Miguel Armayor Martinez, Alejandro Bravo de la Serna, Santiago Londoño Castillo, Dimitra Nikoladou, Gautham Sathish, Zhihao Wang, Sebastian Wölckert, Claudia Linnhoff-Popien

Abstract: The intrinsic symmetries of physical systems have been employed to reduce the number of degrees of freedom of systems, thereby simplifying computations. In this work, we investigate the properties of $\mathcal{M}SU(2^N)$, $\mathcal{M}$-invariant subspaces of the special unitary Lie group $SU(2^N)$ acting on $N$ qubits, for some $\mathcal{M}\subseteq M_{2^N}(\mathbb{C})$. We demonstrate that for ce… ▽ More The intrinsic symmetries of physical systems have been employed to reduce the number of degrees of freedom of systems, thereby simplifying computations. In this work, we investigate the properties of $\mathcal{M}SU(2^N)$, $\mathcal{M}$-invariant subspaces of the special unitary Lie group $SU(2^N)$ acting on $N$ qubits, for some $\mathcal{M}\subseteq M_{2^N}(\mathbb{C})$. We demonstrate that for certain choices of $\mathcal{M}$, the subset $\mathcal{M}SU(2^N)$ inherits many topological and group properties from $SU(2^N)$. We then present a combinatorial method for computing the dimension of such subspaces when $\mathcal{M}$ is a representation of a permutation group acting on qubits $(GSU(2^N))$, or a Hamiltonian $(H^{(N)}SU(2^N))$. The Kronecker product of $\mathfrak{su}(2)$ matrices is employed to construct the Lie algebras associated with different permutation-invariant groups $GSU(2^N)$. Numerical results on the number of dimensions support the the developed theory. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: 27 pages, 6 figures, 1 table

Using an Evolutionary Algorithm to Create (MAX)-3SAT QUBOs

Authors: Sebastian Zielinski, Maximilian Zorn, Thomas Gabor, Sebastian Feld, Claudia Linnhoff-Popien

Abstract: A common way of solving satisfiability instances with quantum methods is to transform these instances into instances of QUBO, which in itself is a potentially difficult and expensive task. State-of-the-art transformations from MAX-3SAT to QUBO currently work by map** clauses of a 3SAT formula associated with the MAX-3SAT instance to an instance of QUBO and combining the resulting QUBOs into a si… ▽ More A common way of solving satisfiability instances with quantum methods is to transform these instances into instances of QUBO, which in itself is a potentially difficult and expensive task. State-of-the-art transformations from MAX-3SAT to QUBO currently work by map** clauses of a 3SAT formula associated with the MAX-3SAT instance to an instance of QUBO and combining the resulting QUBOs into a single QUBO instance representing the whole MAX-3SAT instance. As creating these transformations is currently done manually or via exhaustive search methods and, therefore, algorithmically inefficient, we see potential for including search-based optimization. In this paper, we propose two methods of using evolutionary algorithms to automatically create QUBO representations of MAX-3SAT problems. We evaluate our created QUBOs on 500 and 1000-clause 3SAT formulae and find competitive performance to state-of-the-art baselines when using both classical and quantum annealing solvers. △ Less

Submitted 15 May, 2024; originally announced May 2024.

Towards Robust Benchmarking of Quantum Optimization Algorithms

Authors: David Bucher, Nico Kraus, Jonas Blenninger, Michael Lachner, Jonas Stein, Claudia Linnhoff-Popien

Abstract: Benchmarking the performance of quantum optimization algorithms is crucial for identifying utility for industry-relevant use cases. Benchmarking processes vary between optimization applications and depend on user-specified goals. The heuristic nature of quantum algorithms poses challenges, especially when comparing to classical counterparts. A key problem in existing benchmarking frameworks is the… ▽ More Benchmarking the performance of quantum optimization algorithms is crucial for identifying utility for industry-relevant use cases. Benchmarking processes vary between optimization applications and depend on user-specified goals. The heuristic nature of quantum algorithms poses challenges, especially when comparing to classical counterparts. A key problem in existing benchmarking frameworks is the lack of equal effort in optimizing for the best quantum and, respectively, classical approaches. This paper presents a comprehensive set of guidelines comprising universal steps towards fair benchmarks. We discuss (1) application-specific algorithm choice, ensuring every solver is provided with the most fitting mathematical formulation of a problem; (2) the selection of benchmark data, including hard instances and real-world samples; (3) the choice of a suitable holistic figure of merit, like time-to-solution or solution quality within time constraints; and (4) equitable hyperparameter training to eliminate bias towards a particular method. The proposed guidelines are tested across three benchmarking scenarios, utilizing the Max-Cut (MC) and Travelling Salesperson Problem (TSP). The benchmarks employ classical mathematical algorithms, such as Branch-and-Cut (BNC) solvers, classical heuristics, Quantum Annealing (QA), and the Quantum Approximate Optimization Algorithm (QAOA). △ Less

Submitted 13 May, 2024; originally announced May 2024.

MEDIATE: Mutually Endorsed Distributed Incentive Acknowledgment Token Exchange

Authors: Philipp Altmann, Katharina Winter, Michael Kölle, Maximilian Zorn, Thomy Phan, Claudia Linnhoff-Popien

Abstract: Recent advances in multi-agent systems (MAS) have shown that incorporating peer incentivization (PI) mechanisms vastly improves cooperation. Especially in social dilemmas, communication between the agents helps to overcome sub-optimal Nash equilibria. However, incentivization tokens need to be carefully selected. Furthermore, real-world applications might yield increased privacy requirements and l… ▽ More Recent advances in multi-agent systems (MAS) have shown that incorporating peer incentivization (PI) mechanisms vastly improves cooperation. Especially in social dilemmas, communication between the agents helps to overcome sub-optimal Nash equilibria. However, incentivization tokens need to be carefully selected. Furthermore, real-world applications might yield increased privacy requirements and limited exchange. Therefore, we extend the PI protocol for mutual acknowledgment token exchange (MATE) and provide additional analysis on the impact of the chosen tokens. Building upon those insights, we propose mutually endorsed distributed incentive acknowledgment token exchange (MEDIATE), an extended PI architecture employing automatic token derivation via decentralized consensus. Empirical results show the stable agreement on appropriate tokens yielding superior performance compared to static tokens and state-of-the-art approaches in different social dilemma environments with various reward distributions. △ Less

Submitted 4 April, 2024; originally announced April 2024.

Comments: 12 pages, 5 figures

REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learning

Authors: Philipp Altmann, Céline Davignon, Maximilian Zorn, Fabian Ritz, Claudia Linnhoff-Popien, Thomas Gabor

Abstract: To enhance the interpretability of Reinforcement Learning (RL), we propose Revealing Evolutionary Action Consequence Trajectories (REACT). In contrast to the prevalent practice of validating RL models based on their optimal behavior learned during training, we posit that considering a range of edge-case trajectories provides a more comprehensive understanding of their inherent behavior. To induce… ▽ More To enhance the interpretability of Reinforcement Learning (RL), we propose Revealing Evolutionary Action Consequence Trajectories (REACT). In contrast to the prevalent practice of validating RL models based on their optimal behavior learned during training, we posit that considering a range of edge-case trajectories provides a more comprehensive understanding of their inherent behavior. To induce such scenarios, we introduce a disturbance to the initial state, optimizing it through an evolutionary algorithm to generate a diverse population of demonstrations. To evaluate the fitness of trajectories, REACT incorporates a joint fitness function that encourages both local and global diversity in the encountered states and chosen actions. Through assessments with policies trained for varying durations in discrete and continuous environments, we demonstrate the descriptive power of REACT. Our results highlight its effectiveness in revealing nuanced aspects of RL models' behavior beyond optimal performance, thereby contributing to improved interpretability. △ Less

Submitted 4 April, 2024; originally announced April 2024.

Comments: 12 pages, 12 figures

Sampling Problems on a Quantum Computer

Authors: Maximilian Balthasar Mansky, Jonas Nüßlein, David Bucher, Daniëlle Schuman, Sebastian Zielinski, Claudia Linnhoff-Popien

Abstract: Due to the advances in the manufacturing of quantum hardware in the recent years, significant research efforts have been directed towards employing quantum methods to solving problems in various areas of interest. Thus a plethora of novel quantum methods have been developed in recent years. In this paper, we provide a survey of quantum sampling methods alongside needed theory and applications of t… ▽ More Due to the advances in the manufacturing of quantum hardware in the recent years, significant research efforts have been directed towards employing quantum methods to solving problems in various areas of interest. Thus a plethora of novel quantum methods have been developed in recent years. In this paper, we provide a survey of quantum sampling methods alongside needed theory and applications of those sampling methods as a starting point for research in this area. This work focuses in particular on Gaussian Boson sampling, quantum Monte Carlo methods, quantum variational Monte Carlo, quantum Boltzmann Machines and quantum Bayesian networks. We strive to provide a self-contained overview over the mathematical background, technical feasibility, applicability for other problems and point out potential areas of future research. △ Less

Submitted 26 February, 2024; originally announced February 2024.

Comments: 11 pages, 4 figures. Accepted at QCE 2023

Journal ref: 2023 IEEE International Conference on Quantum Computing and Engineering (QCE)

Symmetry-restricted quantum circuits are still well-behaved

Authors: Maximilian Balthasar Mansky, Santiago Londoño Castillo, Miguel Armayor-Martínez, Alejandro Bravo de la Serna, Gautham Sathish, Zhihao Wang, Sebastian Wölckerlt, Claudia Linnhoff-Popien

Abstract: We show that quantum circuits restricted by a symmetry inherit the properties of the whole special unitary group $SU(2^n)$, in particular composition, algebraic and topological closedness and connectedness. It extends prior work on symmetric states to the operators and shows that the operator space follows the same structure as the state space. The well-behavedness is independent of the symmetry r… ▽ More We show that quantum circuits restricted by a symmetry inherit the properties of the whole special unitary group $SU(2^n)$, in particular composition, algebraic and topological closedness and connectedness. It extends prior work on symmetric states to the operators and shows that the operator space follows the same structure as the state space. The well-behavedness is independent of the symmetry requirement imposed on the subgroup. We provide an example of a permutation invariance across all qubits. △ Less

Submitted 26 February, 2024; originally announced February 2024.

Comments: 10 pages, 2 figures

Aquarium: A Comprehensive Framework for Exploring Predator-Prey Dynamics through Multi-Agent Reinforcement Learning Algorithms

Authors: Michael Kölle, Yannick Erpelding, Fabian Ritz, Thomy Phan, Steffen Illium, Claudia Linnhoff-Popien

Abstract: Recent advances in Multi-Agent Reinforcement Learning have prompted the modeling of intricate interactions between agents in simulated environments. In particular, the predator-prey dynamics have captured substantial interest and various simulations been tailored to unique requirements. To prevent further time-intensive developments, we introduce Aquarium, a comprehensive Multi-Agent Reinforcement… ▽ More Recent advances in Multi-Agent Reinforcement Learning have prompted the modeling of intricate interactions between agents in simulated environments. In particular, the predator-prey dynamics have captured substantial interest and various simulations been tailored to unique requirements. To prevent further time-intensive developments, we introduce Aquarium, a comprehensive Multi-Agent Reinforcement Learning environment for predator-prey interaction, enabling the study of emergent behavior. Aquarium is open source and offers a seamless integration of the PettingZoo framework, allowing a quick start with proven algorithm implementations. It features physics-based agent movement on a two-dimensional, edge-wrap** plane. The agent-environment interaction (observations, actions, rewards) and the environment settings (agent speed, prey reproduction, predator starvation, and others) are fully customizable. Besides a resource-efficient visualization, Aquarium supports to record video files, providing a visual comprehension of agent behavior. To demonstrate the environment's capabilities, we conduct preliminary studies which use PPO to train multiple prey agents to evade a predator. In accordance to the literature, we find Individual Learning to result in worse performance than Parameter Sharing, which significantly improves coordination and sample-efficiency. △ Less

Submitted 13 January, 2024; originally announced January 2024.

Comments: Accepted at ICAART

A Reinforcement Learning Environment for Directed Quantum Circuit Synthesis

Authors: Michael Kölle, Tom Schubert, Philipp Altmann, Maximilian Zorn, Jonas Stein, Claudia Linnhoff-Popien

Abstract: With recent advancements in quantum computing technology, optimizing quantum circuits and ensuring reliable quantum state preparation have become increasingly vital. Traditional methods often demand extensive expertise and manual calculations, posing challenges as quantum circuits grow in qubit- and gate-count. Therefore, harnessing machine learning techniques to handle the growing variety of gate… ▽ More With recent advancements in quantum computing technology, optimizing quantum circuits and ensuring reliable quantum state preparation have become increasingly vital. Traditional methods often demand extensive expertise and manual calculations, posing challenges as quantum circuits grow in qubit- and gate-count. Therefore, harnessing machine learning techniques to handle the growing variety of gate-to-qubit combinations is a promising approach. In this work, we introduce a comprehensive reinforcement learning environment for quantum circuit synthesis, where circuits are constructed utilizing gates from the the Clifford+T gate set to prepare specific target states. Our experiments focus on exploring the relationship between the depth of synthesized quantum circuits and the circuit depths used for target initialization, as well as qubit count. We organize the environment configurations into multiple evaluation levels and include a range of well-known quantum states for benchmarking purposes. We also lay baselines for evaluating the environment using Proximal Policy Optimization. By applying the trained agents to benchmark tests, we demonstrated their ability to reliably design minimal quantum circuits for a selection of 2-qubit Bell states. △ Less

Submitted 13 January, 2024; originally announced January 2024.

Quantum Denoising Diffusion Models

Authors: Michael Kölle, Gerhard Stenzel, Jonas Stein, Sebastian Zielinski, Björn Ommer, Claudia Linnhoff-Popien

Abstract: In recent years, machine learning models like DALL-E, Craiyon, and Stable Diffusion have gained significant attention for their ability to generate high-resolution images from concise descriptions. Concurrently, quantum computing is showing promising advances, especially with quantum machine learning which capitalizes on quantum mechanics to meet the increasing computational requirements of tradit… ▽ More In recent years, machine learning models like DALL-E, Craiyon, and Stable Diffusion have gained significant attention for their ability to generate high-resolution images from concise descriptions. Concurrently, quantum computing is showing promising advances, especially with quantum machine learning which capitalizes on quantum mechanics to meet the increasing computational requirements of traditional machine learning algorithms. This paper explores the integration of quantum machine learning and variational quantum circuits to augment the efficacy of diffusion-based image generation models. Specifically, we address two challenges of classical diffusion models: their low sampling speed and the extensive parameter requirements. We introduce two quantum diffusion models and benchmark their capabilities against their classical counterparts using MNIST digits, Fashion MNIST, and CIFAR-10. Our models surpass the classical models with similar parameter counts in terms of performance metrics FID, SSIM, and PSNR. Moreover, we introduce a consistency model unitary single sampling architecture that combines the diffusion procedure into a single step, enabling a fast one-step image generation. △ Less

Submitted 13 January, 2024; originally announced January 2024.

Quantum Advantage Actor-Critic for Reinforcement Learning

Authors: Michael Kölle, Mohamad Hgog, Fabian Ritz, Philipp Altmann, Maximilian Zorn, Jonas Stein, Claudia Linnhoff-Popien

Abstract: Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empir… ▽ More Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks. △ Less

Submitted 13 January, 2024; originally announced January 2024.

Comments: Accepted at ICAART 24

ClusterComm: Discrete Communication in Decentralized MARL using Internal Representation Clustering

Authors: Robert Müller, Hasan Turalic, Thomy Phan, Michael Kölle, Jonas Nüßlein, Claudia Linnhoff-Popien

Abstract: In the realm of Multi-Agent Reinforcement Learning (MARL), prevailing approaches exhibit shortcomings in aligning with human learning, robustness, and scalability. Addressing this, we introduce ClusterComm, a fully decentralized MARL framework where agents communicate discretely without a central control unit. ClusterComm utilizes Mini-Batch-K-Means clustering on the last hidden layer's activation… ▽ More In the realm of Multi-Agent Reinforcement Learning (MARL), prevailing approaches exhibit shortcomings in aligning with human learning, robustness, and scalability. Addressing this, we introduce ClusterComm, a fully decentralized MARL framework where agents communicate discretely without a central control unit. ClusterComm utilizes Mini-Batch-K-Means clustering on the last hidden layer's activations of an agent's policy network, translating them into discrete messages. This approach outperforms no communication and competes favorably with unbounded, continuous communication and hence poses a simple yet effective strategy for enhancing collaborative task-solving in MARL. △ Less

Submitted 7 January, 2024; originally announced January 2024.

Comments: Accepted at ICAART 2024

Permutation-invariant quantum circuits

Authors: Maximilian Balthasar Mansky, Santiago Londoño Castillo, Victor Ramos Puigvert, Claudia Linnhoff-Popien

Abstract: The implementation of physical symmetries into problem descriptions allows for the reduction of parameters and computational complexity. We show the integration of the permutation symmetry as the most restrictive discrete symmetry into quantum circuits. The permutation symmetry is the supergroup of all other discrete groups. We identify the permutation with a $\operatorname{SWAP}$ operation on the… ▽ More The implementation of physical symmetries into problem descriptions allows for the reduction of parameters and computational complexity. We show the integration of the permutation symmetry as the most restrictive discrete symmetry into quantum circuits. The permutation symmetry is the supergroup of all other discrete groups. We identify the permutation with a $\operatorname{SWAP}$ operation on the qubits. Based on the extension of the symmetry into the corresponding Lie algebra, quantum circuit element construction is shown via exponentiation. This allows for ready integration of the permutation group symmetry into quantum circuit ansatzes. The scaling of the number of parameters is found to be $\mathcal{O}(n^3)$, significantly lower than the general case and an indication that symmetry restricts the applicability of quantum computing. We also show how to adapt existing circuits to be invariant under a permutation symmetry by modification. △ Less

Submitted 22 December, 2023; originally announced December 2023.

Challenges for Reinforcement Learning in Quantum Circuit Design

Authors: Philipp Altmann, Jonas Stein, Michael Kölle, Adelina Bärligea, Thomas Gabor, Thomy Phan, Sebastian Feld, Claudia Linnhoff-Popien

Abstract: Quantum computing (QC) in the current NISQ era is still limited in size and precision. Hybrid applications mitigating those shortcomings are prevalent to gain early insight and advantages. Hybrid quantum machine learning (QML) comprises both the application of QC to improve machine learning (ML) and ML to improve QC architectures. This work considers the latter, leveraging reinforcement learning (… ▽ More Quantum computing (QC) in the current NISQ era is still limited in size and precision. Hybrid applications mitigating those shortcomings are prevalent to gain early insight and advantages. Hybrid quantum machine learning (QML) comprises both the application of QC to improve machine learning (ML) and ML to improve QC architectures. This work considers the latter, leveraging reinforcement learning (RL) to improve the search for viable quantum architectures, which we formalize by a set of generic challenges. Furthermore, we propose a concrete framework, formalized as a Markov decision process, to enable learning policies capable of controlling a universal set of continuously parameterized quantum gates. Finally, we provide benchmark comparisons to assess the shortcomings and strengths of current state-of-the-art RL algorithms. △ Less

Submitted 4 April, 2024; v1 submitted 18 December, 2023; originally announced December 2023.

Comments: 10 pages, 3 figures

Towards Efficient Quantum Anomaly Detection: One-Class SVMs using Variable Subsampling and Randomized Measurements

Authors: Michael Kölle, Afrae Ahouzi, Pascal Debus, Robert Müller, Danielle Schuman, Claudia Linnhoff-Popien

Abstract: Quantum computing, with its potential to enhance various machine learning tasks, allows significant advancements in kernel calculation and model precision. Utilizing the one-class Support Vector Machine alongside a quantum kernel, known for its classically challenging representational capacity, notable improvements in average precision compared to classical counterparts were observed in previous s… ▽ More Quantum computing, with its potential to enhance various machine learning tasks, allows significant advancements in kernel calculation and model precision. Utilizing the one-class Support Vector Machine alongside a quantum kernel, known for its classically challenging representational capacity, notable improvements in average precision compared to classical counterparts were observed in previous studies. Conventional calculations of these kernels, however, present a quadratic time complexity concerning data size, posing challenges in practical applications. To mitigate this, we explore two distinct approaches: utilizing randomized measurements to evaluate the quantum kernel and implementing the variable subsampling ensemble method, both targeting linear time complexity. Experimental results demonstrate a substantial reduction in training and inference times by up to 95\% and 25\% respectively, employing these methods. Although unstable, the average precision of randomized measurements discernibly surpasses that of the classical Radial Basis Function kernel, suggesting a promising direction for further research in scalable, efficient quantum computing applications in machine learning. △ Less

Submitted 14 December, 2023; originally announced December 2023.

Comments: Accepted at ICAART 2024

Improving Parameter Training for VQEs by Sequential Hamiltonian Assembly

Authors: Jonas Stein, Navid Roshani, Maximilian Zorn, Philipp Altmann, Michael Kölle, Claudia Linnhoff-Popien

Abstract: A central challenge in quantum machine learning is the design and training of parameterized quantum circuits (PQCs). Similar to deep learning, vanishing gradients pose immense problems in the trainability of PQCs, which have been shown to arise from a multitude of sources. One such cause are non-local loss functions, that demand the measurement of a large subset of involved qubits. To facilitate t… ▽ More A central challenge in quantum machine learning is the design and training of parameterized quantum circuits (PQCs). Similar to deep learning, vanishing gradients pose immense problems in the trainability of PQCs, which have been shown to arise from a multitude of sources. One such cause are non-local loss functions, that demand the measurement of a large subset of involved qubits. To facilitate the parameter training for quantum applications using global loss functions, we propose a Sequential Hamiltonian Assembly, which iteratively approximates the loss function using local components. Aiming for a prove of principle, we evaluate our approach using Graph Coloring problem with a Varational Quantum Eigensolver (VQE). Simulation results show, that our approach outperforms conventional parameter training by 29.99% and the empirical state of the art, Layerwise Learning, by 5.12% in the mean accuracy. This paves the way towards locality-aware learning techniques, allowing to evade vanishing gradients for a large class of practically relevant problems. △ Less

Submitted 9 December, 2023; originally announced December 2023.

Journal ref: Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2. ICAART 2024

Towards Transfer Learning for Large-Scale Image Classification Using Annealing-based Quantum Boltzmann Machines

Authors: Daniëlle Schuman, Leo Sünkel, Philipp Altmann, Jonas Stein, Christoph Roch, Thomas Gabor, Claudia Linnhoff-Popien

Abstract: Quantum Transfer Learning (QTL) recently gained popularity as a hybrid quantum-classical approach for image classification tasks by efficiently combining the feature extraction capabilities of large Convolutional Neural Networks with the potential benefits of Quantum Machine Learning (QML). Existing approaches, however, only utilize gate-based Variational Quantum Circuits for the quantum part of t… ▽ More Quantum Transfer Learning (QTL) recently gained popularity as a hybrid quantum-classical approach for image classification tasks by efficiently combining the feature extraction capabilities of large Convolutional Neural Networks with the potential benefits of Quantum Machine Learning (QML). Existing approaches, however, only utilize gate-based Variational Quantum Circuits for the quantum part of these procedures. In this work we present an approach to employ Quantum Annealing (QA) in QTL-based image classification. Specifically, we propose using annealing-based Quantum Boltzmann Machines as part of a hybrid quantum-classical pipeline to learn the classification of real-world, large-scale data such as medical images through supervised training. We demonstrate our approach by applying it to the three-class COVID-CT-MD dataset, a collection of lung Computed Tomography (CT) scan slices. Using Simulated Annealing as a stand-in for actual QA, we compare our method to classical transfer learning, using a neural network of the same order of magnitude, to display its improved classification performance. We find that our approach consistently outperforms its classical baseline in terms of test accuracy and AUC-ROC-Score and needs less training epochs to do this. △ Less

Submitted 27 November, 2023; originally announced November 2023.

Comments: 7 pages, 3 figures (5 if counting subfigures), 1 table. To be published in the proceedings of the 2023 IEEE International Conference on Quantum Computing and Engineering (QCE)

Disentangling Quantum and Classical Contributions in Hybrid Quantum Machine Learning Architectures

Authors: Michael Kölle, Jonas Maurer, Philipp Altmann, Leo Sünkel, Jonas Stein, Claudia Linnhoff-Popien

Abstract: Quantum computing offers the potential for superior computational capabilities, particularly for data-intensive tasks. However, the current state of quantum hardware puts heavy restrictions on input size. To address this, hybrid transfer learning solutions have been developed, merging pre-trained classical models, capable of handling extensive inputs, with variational quantum circuits. Yet, it rem… ▽ More Quantum computing offers the potential for superior computational capabilities, particularly for data-intensive tasks. However, the current state of quantum hardware puts heavy restrictions on input size. To address this, hybrid transfer learning solutions have been developed, merging pre-trained classical models, capable of handling extensive inputs, with variational quantum circuits. Yet, it remains unclear how much each component -- classical and quantum -- contributes to the model's results. We propose a novel hybrid architecture: instead of utilizing a pre-trained network for compression, we employ an autoencoder to derive a compressed version of the input data. This compressed data is then channeled through the encoder part of the autoencoder to the quantum component. We assess our model's classification capabilities against two state-of-the-art hybrid transfer learning architectures, two purely classical architectures and one quantum architecture. Their accuracy is compared across four datasets: Banknote Authentication, Breast Cancer Wisconsin, MNIST digits, and AudioMNIST. Our research suggests that classical components significantly influence classification in hybrid transfer learning, a contribution often mistakenly ascribed to the quantum element. The performance of our model aligns with that of a variational quantum circuit using amplitude embedding, positioning it as a feasible alternative. △ Less

Submitted 13 January, 2024; v1 submitted 9 November, 2023; originally announced November 2023.

Multi-Agent Quantum Reinforcement Learning using Evolutionary Optimization

Authors: Michael Kölle, Felix Topp, Thomy Phan, Philipp Altmann, Jonas Nüßlein, Claudia Linnhoff-Popien

Abstract: Multi-Agent Reinforcement Learning is becoming increasingly more important in times of autonomous driving and other smart industrial applications. Simultaneously a promising new approach to Reinforcement Learning arises using the inherent properties of quantum mechanics, reducing the trainable parameters of a model significantly. However, gradient-based Multi-Agent Quantum Reinforcement Learning m… ▽ More Multi-Agent Reinforcement Learning is becoming increasingly more important in times of autonomous driving and other smart industrial applications. Simultaneously a promising new approach to Reinforcement Learning arises using the inherent properties of quantum mechanics, reducing the trainable parameters of a model significantly. However, gradient-based Multi-Agent Quantum Reinforcement Learning methods often have to struggle with barren plateaus, holding them back from matching the performance of classical approaches. We build upon an existing approach for gradient free Quantum Reinforcement Learning and propose three genetic variations with Variational Quantum Circuits for Multi-Agent Reinforcement Learning using evolutionary optimization. We evaluate our genetic variations in the Coin Game environment and also compare them to classical approaches. We showed that our Variational Quantum Circuit approaches perform significantly better compared to a neural network with a similar amount of trainable parameters. Compared to the larger neural network, our approaches archive similar results using $97.88\%$ less parameters. △ Less

Submitted 13 January, 2024; v1 submitted 9 November, 2023; originally announced November 2023.

Incentivising Demand Side Response through Discount Scheduling using Hybrid Quantum Optimization

Authors: David Bucher, Jonas Nüßlein, Corey O'Meara, Ivan Angelov, Benedikt Wimmer, Kumar Ghosh, Giorgio Cortiana, Claudia Linnhoff-Popien

Abstract: Demand Side Response (DSR) is a strategy that enables consumers to actively participate in managing electricity demand. It aims to alleviate strain on the grid during high demand and promote a more balanced and efficient use of (renewable) electricity resources. We implement DSR through discount scheduling, which involves offering discrete price incentives to consumers to adjust their electricity… ▽ More Demand Side Response (DSR) is a strategy that enables consumers to actively participate in managing electricity demand. It aims to alleviate strain on the grid during high demand and promote a more balanced and efficient use of (renewable) electricity resources. We implement DSR through discount scheduling, which involves offering discrete price incentives to consumers to adjust their electricity consumption patterns to times when their local energy mix consists of more renewable energy. Since we tailor the discounts to individual customers' consumption, the Discount Scheduling Problem (DSP) becomes a large combinatorial optimization task. Consequently, we adopt a hybrid quantum computing approach, using D-Wave's Leap Hybrid Cloud. We benchmark Leap against Gurobi, a classical Mixed Integer optimizer in terms of solution quality at fixed runtime and fairness in terms of discount allocation. Furthermore, we propose a large-scale decomposition algorithm/heuristic for the DSP, applied with either quantum or classical computers running the subroutines, which significantly reduces the problem size while maintaining solution quality. Using synthetic data generated from real-world data, we observe that the classical decomposition method obtains the best overall \newp{solution quality for problem sizes up to 3200 consumers, however, the hybrid quantum approach provides more evenly distributed discounts across consumers. △ Less

Submitted 29 May, 2024; v1 submitted 11 September, 2023; originally announced September 2023.

Applying QNLP to sentiment analysis in finance

Authors: Jonas Stein, Ivo Christ, Nicolas Kraus, Maximilian Balthasar Mansky, Robert Müller, Claudia Linnhoff-Popien

Abstract: As an application domain where the slightest qualitative improvements can yield immense value, finance is a promising candidate for early quantum advantage. Focusing on the rapidly advancing field of Quantum Natural Language Processing (QNLP), we explore the practical applicability of the two central approaches DisCoCat and Quantum-Enhanced Long Short-Term Memory (QLSTM) to the problem of sentimen… ▽ More As an application domain where the slightest qualitative improvements can yield immense value, finance is a promising candidate for early quantum advantage. Focusing on the rapidly advancing field of Quantum Natural Language Processing (QNLP), we explore the practical applicability of the two central approaches DisCoCat and Quantum-Enhanced Long Short-Term Memory (QLSTM) to the problem of sentiment analysis in finance. Utilizing a novel ChatGPT-based data generation approach, we conduct a case study with more than 1000 realistic sentences and find that QLSTMs can be trained substantially faster than DisCoCat while also achieving close to classical results for their available software implementations. △ Less

Submitted 11 September, 2023; v1 submitted 20 July, 2023; originally announced July 2023.

Journal ref: QCE'23 Companion: Proceedings of the Companion IEEE International Conference on Quantum Computing and Engineering, 2023, 20-25

Improving Primate Sounds Classification using Binary Presorting for Deep Learning

Authors: Michael Kölle, Steffen Illium, Maximilian Zorn, Jonas Nüßlein, Patrick Suchostawski, Claudia Linnhoff-Popien

Abstract: In the field of wildlife observation and conservation, approaches involving machine learning on audio recordings are becoming increasingly popular. Unfortunately, available datasets from this field of research are often not optimal learning material; Samples can be weakly labeled, of different lengths or come with a poor signal-to-noise ratio. In this work, we introduce a generalized approach that… ▽ More In the field of wildlife observation and conservation, approaches involving machine learning on audio recordings are becoming increasingly popular. Unfortunately, available datasets from this field of research are often not optimal learning material; Samples can be weakly labeled, of different lengths or come with a poor signal-to-noise ratio. In this work, we introduce a generalized approach that first relabels subsegments of MEL spectrogram representations, to achieve higher performances on the actual multi-class classification tasks. For both the binary pre-sorting and the classification, we make use of convolutional neural networks (CNN) and various data-augmentation techniques. We showcase the results of this approach on the challenging \textit{ComparE 2021} dataset, with the task of classifying between different primate species sounds, and report significantly higher Accuracy and UAR scores in contrast to comparatively equipped model baselines. △ Less

Submitted 28 June, 2023; originally announced June 2023.

Comments: DeLTA

Weight Re-Map** for Variational Quantum Algorithms

Authors: Michael Kölle, Alessandro Giovagnoli, Jonas Stein, Maximilian Balthasar Mansky, Julian Hager, Tobias Rohe, Robert Müller, Claudia Linnhoff-Popien

Abstract: Inspired by the remarkable success of artificial neural networks across a broad spectrum of AI tasks, variational quantum circuits (VQCs) have recently seen an upsurge in quantum machine learning applications. The promising outcomes shown by VQCs, such as improved generalization and reduced parameter training requirements, are attributed to the robust algorithmic capabilities of quantum computing.… ▽ More Inspired by the remarkable success of artificial neural networks across a broad spectrum of AI tasks, variational quantum circuits (VQCs) have recently seen an upsurge in quantum machine learning applications. The promising outcomes shown by VQCs, such as improved generalization and reduced parameter training requirements, are attributed to the robust algorithmic capabilities of quantum computing. However, the current gradient-based training approaches for VQCs do not adequately accommodate the fact that trainable parameters (or weights) are typically used as angles in rotational gates. To address this, we extend the concept of weight re-map** for VQCs, as introduced by Kölle et al. (2023). This approach unambiguously maps the weights to an interval of length $2π$, mirroring data rescaling techniques in conventional machine learning that have proven to be highly beneficial in numerous scenarios. In our study, we employ seven distinct weight re-map** functions to assess their impact on eight classification datasets, using variational classifiers as a representative example. Our results indicate that weight re-map** can enhance the convergence speed of the VQC. We assess the efficacy of various re-map** functions across all datasets and measure their influence on the VQC's average performance. Our findings indicate that weight re-map** not only consistently accelerates the convergence of VQCs, regardless of the specific re-map** function employed, but also significantly increases accuracy in certain cases. △ Less

Submitted 9 June, 2023; originally announced June 2023.

Introducing Reduced-Width QNNs, an AI-inspired Ansatz Design Pattern

Authors: Jonas Stein, Tobias Rohe, Francesco Nappi, Julian Hager, David Bucher, Maximilian Zorn, Michael Kölle, Claudia Linnhoff-Popien

Abstract: Variational Quantum Algorithms are one of the most promising candidates to yield the first industrially relevant quantum advantage. Being capable of arbitrary function approximation, they are often referred to as Quantum Neural Networks (QNNs) when being used in analog settings as classical Artificial Neural Networks (ANNs). Similar to the early stages of classical machine learning, known schemes… ▽ More Variational Quantum Algorithms are one of the most promising candidates to yield the first industrially relevant quantum advantage. Being capable of arbitrary function approximation, they are often referred to as Quantum Neural Networks (QNNs) when being used in analog settings as classical Artificial Neural Networks (ANNs). Similar to the early stages of classical machine learning, known schemes for efficient architectures of these networks are scarce. Exploring beyond existing design patterns, we propose a reduced-width circuit ansatz design, which is motivated by recent results gained in the analysis of dropout regularization in QNNs. More precisely, this exploits the insight, that the gates of overparameterized QNNs can be pruned substantially until their expressibility decreases. The results of our case study show, that the proposed design pattern can significantly reduce training time while maintaining the same result quality as the standard "full-width" design in the presence of noise. △ Less

Submitted 8 January, 2024; v1 submitted 8 June, 2023; originally announced June 2023.

Journal ref: Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3. ICAART 2024

Benchmarking Quantum Surrogate Models on Scarce and Noisy Data

Authors: Jonas Stein, Michael Poppel, Philip Adamczyk, Ramona Fabry, Zixin Wu, Michael Kölle, Jonas Nüßlein, Daniëlle Schuman, Philipp Altmann, Thomas Ehmer, Vijay Narasimhan, Claudia Linnhoff-Popien

Abstract: Surrogate models are ubiquitously used in industry and academia to efficiently approximate given black box functions. As state-of-the-art methods from classical machine learning frequently struggle to solve this problem accurately for the often scarce and noisy data sets in practical applications, investigating novel approaches is of great interest. Motivated by recent theoretical results indicati… ▽ More Surrogate models are ubiquitously used in industry and academia to efficiently approximate given black box functions. As state-of-the-art methods from classical machine learning frequently struggle to solve this problem accurately for the often scarce and noisy data sets in practical applications, investigating novel approaches is of great interest. Motivated by recent theoretical results indicating that quantum neural networks (QNNs) have the potential to outperform their classical analogs in the presence of scarce and noisy data, we benchmark their qualitative performance for this scenario empirically. Our contribution displays the first application-centered approach of using QNNs as surrogate models on higher dimensional, real world data. When compared to a classical artificial neural network with a similar number of parameters, our QNN demonstrates significantly better results for noisy and scarce data, and thus motivates future work to explore this potential quantum advantage in surrogate modelling. Finally, we demonstrate the performance of current NISQ hardware experimentally and estimate the gate fidelities necessary to replicate our simulation results. △ Less

Submitted 9 December, 2023; v1 submitted 8 June, 2023; originally announced June 2023.

Journal ref: Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3. ICAART 2024

Approximative lookup-tables and arbitrary function rotations for facilitating NISQ-implementations of the HHL and beyond

Authors: Petros Stougiannidis, Jonas Stein, David Bucher, Sebastian Zielinski, Claudia Linnhoff-Popien, Sebastian Feld

Abstract: Many promising applications of quantum computing with a provable speedup center around the HHL algorithm. Due to restrictions on the hardware and its significant demand on qubits and gates in known implementations, its execution is prohibitive on near-term quantum computers. Aiming to facilitate such NISQ-implementations, we propose a novel circuit approximation technique that enhances the arithme… ▽ More Many promising applications of quantum computing with a provable speedup center around the HHL algorithm. Due to restrictions on the hardware and its significant demand on qubits and gates in known implementations, its execution is prohibitive on near-term quantum computers. Aiming to facilitate such NISQ-implementations, we propose a novel circuit approximation technique that enhances the arithmetic subroutines in the HHL, which resemble a particularly resource-demanding component in small-scale settings. For this, we provide a description of the algorithmic implementation of space-efficient rotations of polynomial functions that do not demand explicit arithmetic calculations inside the quantum circuit. We show how these types of circuits can be reduced in depth by providing a simple and powerful approximation technique. Moreover, we provide an algorithm that converts lookup-tables for arbitrary function rotations into a structure that allows an application of the approximation technique. This allows implementing approximate rotation circuits for many polynomial and non-polynomial functions. Experimental results obtained for realistic early-application dimensions show significant improvements compared to the state-of-the-art, yielding small circuits while achieving good approximations. △ Less

Submitted 8 June, 2023; originally announced June 2023.

Journal ref: QCE'23: Proceedings of the IEEE International Conference on Quantum Computing and Engineering, 2023, 151-160

Exploring Unsupervised Anomaly Detection with Quantum Boltzmann Machines in Fraud Detection

Authors: Jonas Stein, Daniëlle Schuman, Magdalena Benkard, Thomas Holger, Wanja Sajko, Michael Kölle, Jonas Nüßlein, Leo Sünkel, Olivier Salomon, Claudia Linnhoff-Popien

Abstract: Anomaly detection in Endpoint Detection and Response (EDR) is a critical task in cybersecurity programs of large companies. With rapidly growing amounts of data and the omnipresence of zero-day attacks, manual and rule-based detection techniques are no longer eligible in practice. While classical machine learning approaches to this problem exist, they frequently show unsatisfactory performance in… ▽ More Anomaly detection in Endpoint Detection and Response (EDR) is a critical task in cybersecurity programs of large companies. With rapidly growing amounts of data and the omnipresence of zero-day attacks, manual and rule-based detection techniques are no longer eligible in practice. While classical machine learning approaches to this problem exist, they frequently show unsatisfactory performance in differentiating malicious from benign anomalies. A promising approach to attain superior generalization than currently employed machine learning techniques are quantum generative models. Allowing for the largest representation of data on available quantum hardware, we investigate Quantum Annealing based Quantum Boltzmann Machines (QBMs) for the given problem. We contribute the first fully unsupervised approach for the problem of anomaly detection using QBMs and evaluate its performance on an EDR inspired synthetic dataset. Our results indicate that QBMs can outperform their classical analog (i.e., Restricted Boltzmann Machines) in terms of result quality and training steps in special cases. When employing Quantum Annealers from D-Wave Systems, we conclude that either more accurate classical simulators or substantially more QPU time is needed to conduct the necessary hyperparameter optimization allowing to replicate our simulation results on quantum hardware. △ Less

Submitted 17 January, 2024; v1 submitted 8 June, 2023; originally announced June 2023.

Journal ref: Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2. ICAART 2024

Combining the QAOA and HHL Algorithm to achieve a Substantial Quantum Speedup for the Unit Commitment Problem

Authors: Jonas Stein, Jezer Jojo, Afrah Farea, David Bucher, Philipp Altmann, M. Serdar Çelebi, Claudia Linnhoff-Popien

Abstract: In this paper, we propose a quantum algorithm to solve the unit commitment (UC) problem at least cubically faster than existing classical approaches. This is accomplished by calculating the energy transmission costs using the HHL algorithm inside a QAOA routine. We verify our findings experimentally using quantum circuit simulators in a small case study. Further, we postulate the applicability of… ▽ More In this paper, we propose a quantum algorithm to solve the unit commitment (UC) problem at least cubically faster than existing classical approaches. This is accomplished by calculating the energy transmission costs using the HHL algorithm inside a QAOA routine. We verify our findings experimentally using quantum circuit simulators in a small case study. Further, we postulate the applicability of the concepts developed for this algorithm to be used for a large class of optimization problems that demand solving a linear system of equations in order to calculate the cost function for a given solution. △ Less

Submitted 15 June, 2023; v1 submitted 15 May, 2023; originally announced May 2023.

Decomposition Algorithm of an Arbitrary Pauli Exponential through a Quantum Circuit

Authors: Maximilian Balthasar Mansky, Victor Ramos Puigvert, Santiago Londoño Castillo, Claudia Linnhoff-Popien

Abstract: We review the staircase algorithm to decompose the exponential of a generalized Pauli matrix and we propose two alternative recursive methods which offer more efficient quantum circuits. The first algorithm we propose, defined as the inverted staircase algorithm, is more efficient in comparison to the standard staircase algorithm in the number of one-qubit gates, giving a polynomial improvement of… ▽ More We review the staircase algorithm to decompose the exponential of a generalized Pauli matrix and we propose two alternative recursive methods which offer more efficient quantum circuits. The first algorithm we propose, defined as the inverted staircase algorithm, is more efficient in comparison to the standard staircase algorithm in the number of one-qubit gates, giving a polynomial improvement of n/2. For our second algorithm, we introduce fermionic SWAP quantum gates and a systematic way of generalizing these. Such fermionic gates offer a simplification of the number of quantum gates, in particular of CNOT gates, in most quantum circuits. Regarding the staircase algorithm, fermionic quantum gates reduce the number of CNOT gates in roughly n/2 for a large number of qubits. In the end, we discuss the difference between the probability outcomes of fermionic and non-fermionic gates and show that, in general, due to interference, one cannot substitute fermionic gates through non-fermionic gates without altering the outcome of the circuit. △ Less

Submitted 8 May, 2023; originally announced May 2023.

Comments: 9 pages, 20 figures. submitted to IEEE QCE 2023

Evidence that PUBO outperforms QUBO when solving continuous optimization problems with the QAOA

Authors: Jonas Stein, Farbod Chamanian, Maximilian Zorn, Jonas Nüßlein, Sebastian Zielinski, Michael Kölle, Claudia Linnhoff-Popien

Abstract: Quantum computing provides powerful algorithmic tools that have been shown to outperform established classical solvers in specific optimization tasks. A core step in solving optimization problems with known quantum algorithms such as the Quantum Approximate Optimization Algorithm (QAOA) is the problem formulation. While quantum optimization has historically centered around Quadratic Unconstrained… ▽ More Quantum computing provides powerful algorithmic tools that have been shown to outperform established classical solvers in specific optimization tasks. A core step in solving optimization problems with known quantum algorithms such as the Quantum Approximate Optimization Algorithm (QAOA) is the problem formulation. While quantum optimization has historically centered around Quadratic Unconstrained Optimization (QUBO) problems, recent studies show, that many combinatorial problems such as the TSP can be solved more efficiently in their native Polynomial Unconstrained Optimization (PUBO) forms. As many optimization problems in practice also contain continuous variables, our contribution investigates the performance of the QAOA in solving continuous optimization problems when using PUBO and QUBO formulations. Our extensive evaluation on suitable benchmark functions, shows that PUBO formulations generally yield better results, while requiring less qubits. As the multi-qubit interactions needed for the PUBO variant have to be decomposed using the hardware gates available, i.e., currently single- and two-qubit gates, the circuit depth of the PUBO approach outscales its QUBO alternative roughly linearly in the order of the objective function. However, incorporating the planned addition of native multi-qubit gates such as the global Molmer-Sorenson gate, our experiments indicate that PUBO outperforms QUBO for higher order continuous optimization problems in general. △ Less

Submitted 5 May, 2023; originally announced May 2023.

Journal ref: GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, 2023, 2254-2262

Pattern QUBOs: Algorithmic construction of 3SAT-to-QUBO transformations

Authors: Sebastian Zielinski, Jonas Nüßlein, Jonas Stein, Thomas Gabor, Claudia Linnhoff-Popien, Sebastian Feld

Abstract: 3SAT instances need to be transformed into instances of Quadratic Unconstrained Binary Optimization (QUBO) to be solved on a quantum annealer. Although it has been shown that the choice of the 3SAT-to-QUBO transformation can impact the solution quality of quantum annealing significantly, currently only a few 3SAT-to-QUBO transformations are known. Additionally, all of the known 3SAT-to-QUBO transf… ▽ More 3SAT instances need to be transformed into instances of Quadratic Unconstrained Binary Optimization (QUBO) to be solved on a quantum annealer. Although it has been shown that the choice of the 3SAT-to-QUBO transformation can impact the solution quality of quantum annealing significantly, currently only a few 3SAT-to-QUBO transformations are known. Additionally, all of the known 3SAT-to-QUBO transformations were created manually (and not procedurally) by an expert using reasoning, which is a rather slow and limiting process. In this paper, we will introduce the name Pattern QUBO for a concept that has been used implicitly in the construction of 3SAT-to-QUBO transformations before. We will provide an in-depth explanation for the idea behind Pattern QUBOs and show its importance by proposing an algorithmic method that uses Pattern QUBOs to create new 3SAT-to-QUBO transformations automatically. As an additional application of Pattern QUBOs and our proposed algorithmic method, we introduce approximate 3SAT-to-QUBO transformations. These transformations sacrifice optimality but use significantly fewer variables (and thus physical qubits on quantum hardware) than non-approximate 3SAT-to-QUBO transformations. We will show that approximate 3SAT-to-QUBO transformations can nevertheless be very effective in some cases. △ Less

Submitted 4 May, 2023; originally announced May 2023.

Influence of Different 3SAT-to-QUBO Transformations on the Solution Quality of Quantum Annealing: A Benchmark Study

Authors: Sebastian Zielinski, Jonas Nüßlein, Jonas Stein, Thomas Gabor, Claudia Linnhoff-Popien, Sebastian Feld

Abstract: To solve 3SAT instances on quantum annealers they need to be transformed to an instance of Quadratic Unconstrained Binary Optimization (QUBO). When there are multiple transformations available, the question arises whether different transformations lead to differences in the obtained solution quality. Thus, in this paper we conduct an empirical benchmark study, in which we compare four structurally… ▽ More To solve 3SAT instances on quantum annealers they need to be transformed to an instance of Quadratic Unconstrained Binary Optimization (QUBO). When there are multiple transformations available, the question arises whether different transformations lead to differences in the obtained solution quality. Thus, in this paper we conduct an empirical benchmark study, in which we compare four structurally different QUBO transformations for the 3SAT problem with regards to the solution quality on D-Wave's Advantage_system4.1. We show that the choice of QUBO transformation can significantly impact the number of correct solutions the quantum annealer returns. Furthermore, we show that the size of a QUBO instance (i.e., the dimension of the QUBO matrix) is not a sufficient predictor for solution quality, as larger QUBO instances may produce better results than smaller QUBO instances for the same problem. We also empirically show that the number of different quadratic values of a QUBO instance, combined with their range, can significantly impact the solution quality. △ Less

Submitted 1 May, 2023; originally announced May 2023.

CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing

Authors: Philipp Altmann, Fabian Ritz, Leonard Feuchtinger, Jonas Nüßlein, Claudia Linnhoff-Popien, Thomy Phan

Abstract: The safe application of reinforcement learning (RL) requires generalization from limited training data to unseen scenarios. Yet, fulfilling tasks under changing circumstances is a key challenge in RL. Current state-of-the-art approaches for generalization apply data augmentation techniques to increase the diversity of training data. Even though this prevents overfitting to the training environment… ▽ More The safe application of reinforcement learning (RL) requires generalization from limited training data to unseen scenarios. Yet, fulfilling tasks under changing circumstances is a key challenge in RL. Current state-of-the-art approaches for generalization apply data augmentation techniques to increase the diversity of training data. Even though this prevents overfitting to the training environment(s), it hinders policy optimization. Crafting a suitable observation, only containing crucial information, has been shown to be a challenging task itself. To improve data efficiency and generalization capabilities, we propose Compact Reshaped Observation Processing (CROP) to reduce the state information used for policy optimization. By providing only relevant information, overfitting to a specific training layout is precluded and generalization to unseen environments is improved. We formulate three CROPs that can be applied to fully observable observation- and action-spaces and provide methodical foundation. We empirically show the improvements of CROP in a distributionally shifted safety gridworld. We furthermore provide benchmark comparisons to full observability and data-augmentation in two different-sized procedurally generated mazes. △ Less

Submitted 5 December, 2023; v1 submitted 26 April, 2023; originally announced April 2023.

Comments: 9 pages, 5 figures, published at IJCAI 2023

Solving (Max) 3-SAT via Quadratic Unconstrained Binary Optimization

Authors: Jonas Nüßlein, Sebastian Zielinski, Thomas Gabor, Claudia Linnhoff-Popien, Sebastian Feld

Abstract: We introduce a novel approach to translate arbitrary 3-SAT instances to Quadratic Unconstrained Binary Optimization (QUBO) as they are used by quantum annealing (QA) or the quantum approximate optimization algorithm (QAOA). Our approach requires fewer couplings and fewer physical qubits than the current state-of-the-art, which results in higher solution quality. We verified the practical applicabi… ▽ More We introduce a novel approach to translate arbitrary 3-SAT instances to Quadratic Unconstrained Binary Optimization (QUBO) as they are used by quantum annealing (QA) or the quantum approximate optimization algorithm (QAOA). Our approach requires fewer couplings and fewer physical qubits than the current state-of-the-art, which results in higher solution quality. We verified the practical applicability of the approach by testing it on a D-Wave quantum annealer. △ Less

Submitted 7 February, 2023; originally announced February 2023.

DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training

Authors: Philipp Altmann, Thomy Phan, Fabian Ritz, Thomas Gabor, Claudia Linnhoff-Popien

Abstract: We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated b… ▽ More We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator's verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a surrogate, steering policy optimization towards more valuable regions of the reward landscape thus learning an optimal policy. Our results show that DIRECT outperforms state-of-the-art algorithms in sparse- and shifting-reward environments being able to provide a surrogate reward to the policy and direct the optimization towards valuable areas. △ Less

Submitted 18 January, 2023; originally announced January 2023.

Comments: 9 pages, 10 figures, under review

ACM Class: I.2.6

Compression of GPS Trajectories using Autoencoders

Authors: Michael Kölle, Steffen Illium, Carsten Hahn, Lorenz Schauer, Johannes Hutter, Claudia Linnhoff-Popien

Abstract: The ubiquitous availability of mobile devices capable of location tracking led to a significant rise in the collection of GPS data. Several compression methods have been developed in order to reduce the amount of storage needed while kee** the important information. In this paper, we present an lstm-autoencoder based approach in order to compress and reconstruct GPS trajectories, which is evalua… ▽ More The ubiquitous availability of mobile devices capable of location tracking led to a significant rise in the collection of GPS data. Several compression methods have been developed in order to reduce the amount of storage needed while kee** the important information. In this paper, we present an lstm-autoencoder based approach in order to compress and reconstruct GPS trajectories, which is evaluated on both a gaming and real-world dataset. We consider various compression ratios and trajectory lengths. The performance is compared to other trajectory compression algorithms, i.e., Douglas-Peucker. Overall, the results indicate that our approach outperforms Douglas-Peucker significantly in terms of the discrete Fréchet distance and dynamic time war**. Furthermore, by reconstructing every point lossy, the proposed methodology offers multiple advantages over traditional methods. △ Less

Submitted 18 January, 2023; originally announced January 2023.

Comments: Accepted at ICAART 2023

SEQUENT: Towards Traceable Quantum Machine Learning using Sequential Quantum Enhanced Training

Authors: Philipp Altmann, Leo Sünkel, Jonas Stein, Tobias Müller, Christoph Roch, Claudia Linnhoff-Popien

Abstract: Applying new computing paradigms like quantum computing to the field of machine learning has recently gained attention. However, as high-dimensional real-world applications are not yet feasible to be solved using purely quantum hardware, hybrid methods using both classical and quantum machine learning paradigms have been proposed. For instance, transfer learning methods have been shown to be succe… ▽ More Applying new computing paradigms like quantum computing to the field of machine learning has recently gained attention. However, as high-dimensional real-world applications are not yet feasible to be solved using purely quantum hardware, hybrid methods using both classical and quantum machine learning paradigms have been proposed. For instance, transfer learning methods have been shown to be successfully applicable to hybrid image classification tasks. Nevertheless, beneficial circuit architectures still need to be explored. Therefore, tracing the impact of the chosen circuit architecture and parameterization is crucial for the development of beneficially applicable hybrid methods. However, current methods include processes where both parts are trained concurrently, therefore not allowing for a strict separability of classical and quantum impact. Thus, those architectures might produce models that yield a superior prediction accuracy whilst employing the least possible quantum impact. To tackle this issue, we propose Sequential Quantum Enhanced Training (SEQUENT) an improved architecture and training process for the traceable application of quantum computing methods to hybrid machine learning. Furthermore, we provide formal evidence for the disadvantage of current methods and preliminary experimental results as a proof-of-concept for the applicability of SEQUENT. △ Less

Submitted 26 April, 2023; v1 submitted 6 January, 2023; originally announced January 2023.

Comments: Published at ICAART 2023

Journal ref: Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, 2023

Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability

Authors: Thomy Phan, Fabian Ritz, Philipp Altmann, Maximilian Zorn, Jonas Nüßlein, Michael Kölle, Thomas Gabor, Claudia Linnhoff-Popien

Abstract: Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attent… ▽ More Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attention-based Embeddings of Recurrence In multi-Agent Learning (AERIAL) to approximate value functions under stochastic partial observability. AERIAL replaces the true state with a learned representation of multi-agent recurrence, considering more accurate information about decentralized agent decisions than state-based CTDE. We then introduce MessySMAC, a modified version of SMAC with stochastic observations and higher variance in initial states, to provide a more general and configurable benchmark regarding stochastic partial observability. We evaluate AERIAL in Dec-Tiger as well as in a variety of SMAC and MessySMAC maps, and compare the results with state-based CTDE. Furthermore, we evaluate the robustness of AERIAL and state-based CTDE against various stochasticity configurations in MessySMAC. △ Less

Submitted 27 December, 2023; v1 submitted 4 January, 2023; originally announced January 2023.

Comments: Accepted to ICML 2023

Improving Convergence for Quantum Variational Classifiers using Weight Re-Map**

Authors: Michael Kölle, Alessandro Giovagnoli, Jonas Stein, Maximilian Balthasar Mansky, Julian Hager, Claudia Linnhoff-Popien

Abstract: In recent years, quantum machine learning has seen a substantial increase in the use of variational quantum circuits (VQCs). VQCs are inspired by artificial neural networks, which achieve extraordinary performance in a wide range of AI tasks as massively parameterized function approximators. VQCs have already demonstrated promising results, for example, in generalization and the requirement for fe… ▽ More In recent years, quantum machine learning has seen a substantial increase in the use of variational quantum circuits (VQCs). VQCs are inspired by artificial neural networks, which achieve extraordinary performance in a wide range of AI tasks as massively parameterized function approximators. VQCs have already demonstrated promising results, for example, in generalization and the requirement for fewer parameters to train, by utilizing the more robust algorithmic toolbox available in quantum computing. A VQCs' trainable parameters or weights are usually used as angles in rotational gates and current gradient-based training methods do not account for that. We introduce weight re-map** for VQCs, to unambiguously map the weights to an interval of length $2π$, drawing inspiration from traditional ML, where data rescaling, or normalization techniques have demonstrated tremendous benefits in many circumstances. We employ a set of five functions and evaluate them on the Iris and Wine datasets using variational classifiers as an example. Our experiments show that weight re-map** can improve convergence in all tested settings. Additionally, we were able to demonstrate that weight re-map** increased test accuracy for the Wine dataset by $10\%$ over using unmodified weights. △ Less

Submitted 15 February, 2023; v1 submitted 22 December, 2022; originally announced December 2022.

Comments: Accepted at ICAART 2023

Near-optimal quantum circuit construction via Cartan decomposition

Authors: Maximilian Balthasar Mansky, Santiago Londoño Castillo, Victor Ramos Puigvert, Claudia Linnhoff-Popien

Abstract: We show the applicability of the Cartan decomposition of Lie algebras to quantum circuits. This approach can be used to synthesize circuits that can efficiently implement any desired unitary operation. Our method finds explicit quantum circuit representations of the algebraic generators of the relevant Lie algebras allowing the direct implementation of a Cartan decomposition on a quantum computer.… ▽ More We show the applicability of the Cartan decomposition of Lie algebras to quantum circuits. This approach can be used to synthesize circuits that can efficiently implement any desired unitary operation. Our method finds explicit quantum circuit representations of the algebraic generators of the relevant Lie algebras allowing the direct implementation of a Cartan decomposition on a quantum computer. The construction is recursive and allows us to expand any circuit down to generators and rotation matrices on individual qubits, where through our recursive algorithm we find that the generators themselves can be expressed with controlled-not (CNOT) and SWAP gates explicitly. Our approach is independent of the standard CNOT implementation and can be easily adapted to other cross-qubit circuit elements. In addition to its versatility, we also achieve near-optimal counts when working with CNOT gates, achieving an asymptotic cnot cost of $\frac{21}{16}4^n$ for $n$ qubits. △ Less

Submitted 15 November, 2023; v1 submitted 25 December, 2022; originally announced December 2022.

Journal ref: Physical Review A 108, 052607 (2023)

Empirical Analysis of Limits for Memory Distance in Recurrent Neural Networks

Authors: Steffen Illium, Thore Schillman, Robert Müller, Thomas Gabor, Claudia Linnhoff-Popien

Abstract: Common to all different kinds of recurrent neural networks (RNNs) is the intention to model relations between data points through time. When there is no immediate relationship between subsequent data points (like when the data points are generated at random, e.g.), we show that RNNs are still able to remember a few data points back into the sequence by memorizing them by heart using standard backp… ▽ More Common to all different kinds of recurrent neural networks (RNNs) is the intention to model relations between data points through time. When there is no immediate relationship between subsequent data points (like when the data points are generated at random, e.g.), we show that RNNs are still able to remember a few data points back into the sequence by memorizing them by heart using standard backpropagation. However, we also show that for classical RNNs, LSTM and GRU networks the distance of data points between recurrent calls that can be reproduced this way is highly limited (compared to even a loose connection between data points) and subject to various constraints imposed by the type and size of the RNN in question. This implies the existence of a hard limit (way below the information-theoretic one) for the distance between related data points within which RNNs are still able to recognize said relation. △ Less

Submitted 20 December, 2022; originally announced December 2022.

Constructing Organism Networks from Collaborative Self-Replicators

Authors: Steffen Illium, Maximilian Zorn, Cristian Lenta, Michael Kölle, Claudia Linnhoff-Popien, Thomas Gabor

Abstract: We introduce organism networks, which function like a single neural network but are composed of several neural particle networks; while each particle network fulfils the role of a single weight application within the organism network, it is also trained to self-replicate its own weights. As organism networks feature vastly more parameters than simpler architectures, we perform our initial experime… ▽ More We introduce organism networks, which function like a single neural network but are composed of several neural particle networks; while each particle network fulfils the role of a single weight application within the organism network, it is also trained to self-replicate its own weights. As organism networks feature vastly more parameters than simpler architectures, we perform our initial experiments on an arithmetic task as well as on simplified MNIST-dataset classification as a collective. We observe that individual particle networks tend to specialise in either of the tasks and that the ones fully specialised in the secondary task may be dropped from the network without hindering the computational accuracy of the primary task. This leads to the discovery of a novel pruning-strategy for sparse neural networks △ Less

Submitted 27 February, 2023; v1 submitted 20 December, 2022; originally announced December 2022.

Comments: 2023-02-27 fixed one typo in NN formula

VoronoiPatches: Evaluating A New Data Augmentation Method

Authors: Steffen Illium, Gretchen Griffin, Michael Kölle, Maximilian Zorn, Jonas Nüßlein, Claudia Linnhoff-Popien

Abstract: Overfitting is a problem in Convolutional Neural Networks (CNN) that causes poor generalization of models on unseen data. To remediate this problem, many new and diverse data augmentation methods (DA) have been proposed to supplement or generate more training data, and thereby increase its quality. In this work, we propose a new data augmentation algorithm: VoronoiPatches (VP). We primarily utiliz… ▽ More Overfitting is a problem in Convolutional Neural Networks (CNN) that causes poor generalization of models on unseen data. To remediate this problem, many new and diverse data augmentation methods (DA) have been proposed to supplement or generate more training data, and thereby increase its quality. In this work, we propose a new data augmentation algorithm: VoronoiPatches (VP). We primarily utilize non-linear recombination of information within an image, fragmenting and occluding small information patches. Unlike other DA methods, VP uses small convex polygon-shaped patches in a random layout to transport information around within an image. Sudden transitions created between patches and the original image can, optionally, be smoothed. In our experiments, VP outperformed current DA methods regarding model variance and overfitting tendencies. We demonstrate data augmentation utilizing non-linear re-combination of information within images, and non-orthogonal shapes and structures improves CNN model robustness on unseen data. △ Less

Submitted 23 December, 2022; v1 submitted 20 December, 2022; originally announced December 2022.

Capturing Dependencies within Machine Learning via a Formal Process Model

Authors: Fabian Ritz, Thomy Phan, Andreas Sedlmeier, Philipp Altmann, Jan Wieghardt, Reiner Schmid, Horst Sauer, Cornel Klein, Claudia Linnhoff-Popien, Thomas Gabor

Abstract: The development of Machine Learning (ML) models is more than just a special case of software development (SD): ML models acquire properties and fulfill requirements even without direct human interaction in a seemingly uncontrollable manner. Nonetheless, the underlying processes can be described in a formal way. We define a comprehensive SD process model for ML that encompasses most tasks and artif… ▽ More The development of Machine Learning (ML) models is more than just a special case of software development (SD): ML models acquire properties and fulfill requirements even without direct human interaction in a seemingly uncontrollable manner. Nonetheless, the underlying processes can be described in a formal way. We define a comprehensive SD process model for ML that encompasses most tasks and artifacts described in the literature in a consistent way. In addition to the production of the necessary artifacts, we also focus on generating and validating fitting descriptions in the form of specifications. We stress the importance of further evolving the ML model throughout its life-cycle even after initial training and testing. Thus, we provide various interaction points with standard SD processes in which ML often is an encapsulated task. Further, our SD process model allows to formulate ML as a (meta-) optimization problem. If automated rigorously, it can be used to realize self-adaptive autonomous systems. Finally, our SD process model features a description of time that allows to reason about the progress within ML development processes. This might lead to further applications of formal methods within the field of ML. △ Less

Submitted 10 August, 2022; originally announced August 2022.

Comments: 10 pages, 5 figures, draft; the final version will appear in the proceedings of the International Symposium on Leveraging Applications of Formal Methods (ISoLA) 2022

Journal ref: ISoLA 2022: Leveraging Applications of Formal Methods, Verification and Validation. Adaptation and Learning. pp 249-265

Stochastic Market Games

Authors: Kyrill Schmid, Lenz Belzner, Robert Müller, Johannes Tochtermann, Claudia Linnhoff-Popien

Abstract: Some of the most relevant future applications of multi-agent systems like autonomous driving or factories as a service display mixed-motive scenarios, where agents might have conflicting goals. In these settings agents are likely to learn undesirable outcomes in terms of cooperation under independent learning, such as overly greedy behavior. Motivated from real world societies, in this work we pro… ▽ More Some of the most relevant future applications of multi-agent systems like autonomous driving or factories as a service display mixed-motive scenarios, where agents might have conflicting goals. In these settings agents are likely to learn undesirable outcomes in terms of cooperation under independent learning, such as overly greedy behavior. Motivated from real world societies, in this work we propose to utilize market forces to provide incentives for agents to become cooperative. As demonstrated in an iterated version of the Prisoner's Dilemma, the proposed market formulation can change the dynamics of the game to consistently learn cooperative policies. Further we evaluate our approach in spatially and temporally extended settings for varying numbers of agents. We empirically find that the presence of markets can improve both the overall result and agent individual returns via their trading activities. △ Less

Submitted 19 July, 2022; v1 submitted 15 July, 2022; originally announced July 2022.

Comments: IJCAI-21

Towards Turing-Complete Quantum Computing Coming From Classical Assembler

Authors: Thomas Gabor, Marian Lingsch Rosenfeld, Claudia Linnhoff-Popien

Abstract: Instead of producing quantum languages that are fit for current quantum computers, we build a language from standard classical assembler and augment it with quantum capabilities so that quantum algorithms become a subset of it. This paves the way for the development of hybrid algorithms directly from classical software, which is not feasible on today's hardware but might inspire future quantum pro… ▽ More Instead of producing quantum languages that are fit for current quantum computers, we build a language from standard classical assembler and augment it with quantum capabilities so that quantum algorithms become a subset of it. This paves the way for the development of hybrid algorithms directly from classical software, which is not feasible on today's hardware but might inspire future quantum programmers. △ Less

Submitted 28 June, 2022; originally announced June 2022.

Black Box Optimization Using QUBO and the Cross Entropy Method

Authors: Jonas Nüßlein, Christoph Roch, Thomas Gabor, Jonas Stein, Claudia Linnhoff-Popien, Sebastian Feld

Abstract: Black-box optimization (BBO) can be used to optimize functions whose analytic form is unknown. A common approach to realising BBO is to learn a surrogate model which approximates the target black-box function which can then be solved via white-box optimization methods. In this paper, we present our approach BOX-QUBO, where the surrogate model is a QUBO matrix. However, unlike in previous state-of-… ▽ More Black-box optimization (BBO) can be used to optimize functions whose analytic form is unknown. A common approach to realising BBO is to learn a surrogate model which approximates the target black-box function which can then be solved via white-box optimization methods. In this paper, we present our approach BOX-QUBO, where the surrogate model is a QUBO matrix. However, unlike in previous state-of-the-art approaches, this matrix is not trained entirely by regression, but mostly by classification between 'good' and 'bad' solutions. This better accounts for the low capacity of the QUBO matrix, resulting in significantly better solutions overall. We tested our approach against the state-of-the-art on four domains and in all of them BOX-QUBO showed better results. A second contribution of this paper is the idea to also solve white-box problems, i.e. problems which could be directly formulated as QUBO, by means of black-box optimization in order to reduce the size of the QUBOs to the information-theoretic minimum. Experiments show that this significantly improves the results for MAX-k-SAT. △ Less

Submitted 9 February, 2023; v1 submitted 24 June, 2022; originally announced June 2022.

Case-Based Inverse Reinforcement Learning Using Temporal Coherence

Authors: Jonas Nüßlein, Steffen Illium, Robert Müller, Thomas Gabor, Claudia Linnhoff-Popien

Abstract: Providing expert trajectories in the context of Imitation Learning is often expensive and time-consuming. The goal must therefore be to create algorithms which require as little expert data as possible. In this paper we present an algorithm that imitates the higher-level strategy of the expert rather than just imitating the expert on action level, which we hypothesize requires less expert data and… ▽ More Providing expert trajectories in the context of Imitation Learning is often expensive and time-consuming. The goal must therefore be to create algorithms which require as little expert data as possible. In this paper we present an algorithm that imitates the higher-level strategy of the expert rather than just imitating the expert on action level, which we hypothesize requires less expert data and makes training more stable. As a prior, we assume that the higher-level strategy is to reach an unknown target state area, which we hypothesize is a valid prior for many domains in Reinforcement Learning. The target state area is unknown, but since the expert has demonstrated how to reach it, the agent tries to reach states similar to the expert. Building on the idea of Temporal Coherence, our algorithm trains a neural network to predict whether two states are similar, in the sense that they may occur close in time. During inference, the agent compares its current state with expert states from a Case Base for similarity. The results show that our approach can still learn a near-optimal policy in settings with very little expert data, where algorithms that try to imitate the expert at the action level can no longer do so. △ Less

Submitted 12 June, 2022; originally announced June 2022.

Comments: accepted at ICCBR