Showing 1–39 of 39 results for author: Kochenderfer, M

Hierarchical Framework for Optimizing Wildfire Surveillance and Suppression using Human-Autonomous Teaming

Authors: Mahdi Al-Husseini, Kyle Wray, Mykel Kochenderfer

Abstract: The integration of manned and unmanned aircraft can help improve wildfire response. Wildfire containment failures occur when resources available to first responders, who execute the initial stages of wildfire management referred to as the initial attack, are ineffective or insufficient. Initial attack surveillance and suppression models have linked action spaces and objectives, making their optimi… ▽ More The integration of manned and unmanned aircraft can help improve wildfire response. Wildfire containment failures occur when resources available to first responders, who execute the initial stages of wildfire management referred to as the initial attack, are ineffective or insufficient. Initial attack surveillance and suppression models have linked action spaces and objectives, making their optimization computationally challenging. The initial attack may be formulated as a multi-agent partially observable Markov decision process (MPOMDP). We divide the initial attack MPOMDP into surveillance and suppression processes with their respective planners operating on different, but constant, time scales. A hierarchical framework iterates between surveillance and suppression planners while also providing collision avoidance. This framework is exemplified by a set of multi-rotor unmanned aircraft surveying an initial attack fire while a manned helicopter suppresses the fire with a water bucket. Wildfire-specific solver extensions are formulated to reduce the otherwise vast action spaces. The hierarchical framework outperforms firefighting techniques and a myopic baseline by up to 242% for moderate wildfires and 60% for rapid wildfires when simulated in abstracted and actual case studies. We also validate the early dispatching of additional suppression assets using regression models to ensure wildfire containment to thresholds established by wildfire agencies. △ Less

Submitted 24 June, 2024; originally announced June 2024.

Diffusion-Based Failure Sampling for Cyber-Physical Systems

Authors: Harrison Delecki, Marc R. Schlichting, Mansur Arief, Anthony Corso, Marcell Vazquez-Chanlatte, Mykel J. Kochenderfer

Abstract: Validating safety-critical autonomous systems in high-dimensional domains such as robotics presents a significant challenge. Existing black-box approaches based on Markov chain Monte Carlo may require an enormous number of samples, while methods based on importance sampling often rely on simple parametric families that may struggle to represent the distribution over failures. We propose to sample… ▽ More Validating safety-critical autonomous systems in high-dimensional domains such as robotics presents a significant challenge. Existing black-box approaches based on Markov chain Monte Carlo may require an enormous number of samples, while methods based on importance sampling often rely on simple parametric families that may struggle to represent the distribution over failures. We propose to sample the distribution over failures using a conditional denoising diffusion model, which has shown success in complex high-dimensional problems such as robotic task planning. We iteratively train a diffusion model to produce state trajectories closer to failure. We demonstrate the effectiveness of our approach on high-dimensional robotic validation tasks, improving sample efficiency and mode coverage compared to existing black-box techniques. △ Less

Submitted 20 June, 2024; originally announced June 2024.

Comments: Under review at RA-L

The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning

Authors: Ali Baheri, Mykel J. Kochenderfer

Abstract: This paper explores the integration of optimal transport (OT) theory with multi-agent reinforcement learning (MARL). This integration uses OT to handle distributions and transportation problems to enhance the efficiency, coordination, and adaptability of MARL. There are five key areas where OT can impact MARL: (1) policy alignment, where OT's Wasserstein metric is used to align divergent agent str… ▽ More This paper explores the integration of optimal transport (OT) theory with multi-agent reinforcement learning (MARL). This integration uses OT to handle distributions and transportation problems to enhance the efficiency, coordination, and adaptability of MARL. There are five key areas where OT can impact MARL: (1) policy alignment, where OT's Wasserstein metric is used to align divergent agent strategies towards unified goals; (2) distributed resource management, employing OT to optimize resource allocation among agents; (3) addressing non-stationarity, using OT to adapt to dynamic environmental shifts; (4) scalable multi-agent learning, harnessing OT for decomposing large-scale learning objectives into manageable tasks; and (5) enhancing energy efficiency, applying OT principles to develop sustainable MARL systems. This paper articulates how the synergy between OT and MARL can address scalability issues, optimize resource distribution, align agent policies in cooperative environments, and ensure adaptability in dynamically changing conditions. △ Less

Submitted 24 January, 2024; v1 submitted 18 January, 2024; originally announced January 2024.

SAVME: Efficient Safety Validation for Autonomous Systems Using Meta-Learning

Authors: Marc R. Schlichting, Nina V. Boord, Anthony L. Corso, Mykel J. Kochenderfer

Abstract: Discovering potential failures of an autonomous system is important prior to deployment. Falsification-based methods are often used to assess the safety of such systems, but the cost of running many accurate simulation can be high. The validation can be accelerated by identifying critical failure scenarios for the system under test and by reducing the simulation runtime. We propose a Bayesian appr… ▽ More Discovering potential failures of an autonomous system is important prior to deployment. Falsification-based methods are often used to assess the safety of such systems, but the cost of running many accurate simulation can be high. The validation can be accelerated by identifying critical failure scenarios for the system under test and by reducing the simulation runtime. We propose a Bayesian approach that integrates meta-learning strategies with a multi-armed bandit framework. Our method involves learning distributions over scenario parameters that are prone to triggering failures in the system under test, as well as a distribution over fidelity settings that enable fast and accurate simulations. In the spirit of meta-learning, we also assess whether the learned fidelity settings distribution facilitates faster learning of the scenario parameter distributions for new scenarios. We showcase our methodology using a cutting-edge 3D driving simulator, incorporating 16 fidelity settings for an autonomous vehicle stack that includes camera and lidar sensors. We evaluate various scenarios based on an autonomous vehicle pre-crash typology. As a result, our approach achieves a significant speedup, up to 18 times faster compared to traditional methods that solely rely on a high-fidelity simulator. △ Less

Submitted 30 September, 2023; v1 submitted 21 September, 2023; originally announced September 2023.

Comments: Accepted for ITSC 2023

Joint Falsification and Fidelity Settings Optimization for Validation of Safety-Critical Systems: A Theoretical Analysis

Authors: Ali Baheri, Mykel J. Kochenderfer

Abstract: Safety validation is a crucial component in the development and deployment of autonomous systems, such as self-driving vehicles and robotic systems. Ensuring safe operation necessitates extensive testing and verification of control policies, typically conducted in simulation environments. High-fidelity simulators accurately model real-world dynamics but entail high computational costs, limiting th… ▽ More Safety validation is a crucial component in the development and deployment of autonomous systems, such as self-driving vehicles and robotic systems. Ensuring safe operation necessitates extensive testing and verification of control policies, typically conducted in simulation environments. High-fidelity simulators accurately model real-world dynamics but entail high computational costs, limiting their scalability for exhaustive testing. Conversely, low-fidelity simulators offer efficiency but may not capture the intricacies of high-fidelity simulators, potentially yielding false conclusions. We propose a joint falsification and fidelity optimization framework for safety validation of autonomous systems. Our mathematical formulation combines counterexample searches with simulator fidelity improvement, facilitating more efficient exploration of the critical environmental configurations challenging the control system. Our contributions encompass a set of theorems addressing counterexample sensitivity analysis, sample complexity, convergence, the interplay between the outer and inner optimization loops, and regret bound analysis. The proposed joint optimization approach enables a more targeted and efficient testing process, optimizes the use of available computational resources, and enhances confidence in autonomous system safety validation. △ Less

Submitted 10 May, 2023; originally announced May 2023.

Comments: Submitted to the 20th International Conference on Quantitative Evaluation of Systems (QEST 2023)

Optimizing Carbon Storage Operations for Long-Term Safety

Authors: Yizheng Wang, Markus Zechner, Gege Wen, Anthony Louis Corso, John Michael Mern, Mykel J. Kochenderfer, Jef Karel Caers

Abstract: To combat global warming and mitigate the risks associated with climate change, carbon capture and storage (CCS) has emerged as a crucial technology. However, safely sequestering CO2 in geological formations for long-term storage presents several challenges. In this study, we address these issues by modeling the decision-making process for carbon storage operations as a partially observable Markov… ▽ More To combat global warming and mitigate the risks associated with climate change, carbon capture and storage (CCS) has emerged as a crucial technology. However, safely sequestering CO2 in geological formations for long-term storage presents several challenges. In this study, we address these issues by modeling the decision-making process for carbon storage operations as a partially observable Markov decision process (POMDP). We solve the POMDP using belief state planning to optimize injector and monitoring well locations, with the goal of maximizing stored CO2 while maintaining safety. Empirical results in simulation demonstrate that our approach is effective in ensuring safe long-term carbon storage operations. We showcase the flexibility of our approach by introducing three different monitoring strategies and examining their impact on decision quality. Additionally, we introduce a neural network surrogate model for the POMDP decision-making process to handle the complex dynamics of the multi-phase flow. We also investigate the effects of different fidelity levels of the surrogate model on decision qualities. △ Less

Submitted 18 April, 2023; originally announced April 2023.

Falsification of Learning-Based Controllers through Multi-Fidelity Bayesian Optimization

Authors: Zahra Shahrooei, Mykel J. Kochenderfer, Ali Baheri

Abstract: Simulation-based falsification is a practical testing method to increase confidence that the system will meet safety requirements. Because full-fidelity simulations can be computationally demanding, we investigate the use of simulators with different levels of fidelity. As a first step, we express the overall safety specification in terms of environmental parameters and structure this safety speci… ▽ More Simulation-based falsification is a practical testing method to increase confidence that the system will meet safety requirements. Because full-fidelity simulations can be computationally demanding, we investigate the use of simulators with different levels of fidelity. As a first step, we express the overall safety specification in terms of environmental parameters and structure this safety specification as an optimization problem. We propose a multi-fidelity falsification framework using Bayesian optimization, which is able to determine at which level of fidelity we should conduct a safety evaluation in addition to finding possible instances from the environment that cause the system to fail. This method allows us to automatically switch between inexpensive, inaccurate information from a low-fidelity simulator and expensive, accurate information from a high-fidelity simulator in a cost-effective way. Our experiments on various environments in simulation demonstrate that multi-fidelity Bayesian optimization has falsification performance comparable to single-fidelity Bayesian optimization but with much lower cost. △ Less

Submitted 28 April, 2023; v1 submitted 28 December, 2022; originally announced December 2022.

Comments: 7 pages, 8 figures, Accepted for the 2023 European Control Conference (ECC)

Optimality Guarantees for Particle Belief Approximation of POMDPs

Authors: Michael H. Lim, Tyler J. Becker, Mykel J. Kochenderfer, Claire J. Tomlin, Zachary N. Sunberg

Abstract: Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood w… ▽ More Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of $\mathcal{O}(C)$, where $C$ is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers. △ Less

Submitted 19 October, 2023; v1 submitted 10 October, 2022; originally announced October 2022.

Journal ref: Journal of Artificial Intelligence Research, 77, 1591-1636 (2023)

Backward Reachability Analysis of Neural Feedback Loops: Techniques for Linear and Nonlinear Systems

Authors: Nicholas Rober, Sydney M. Katz, Chelsea Sidrane, Esen Yel, Michael Everett, Mykel J. Kochenderfer, Jonathan P. How

Abstract: As neural networks (NNs) become more prevalent in safety-critical applications such as control of vehicles, there is a growing need to certify that systems with NN components are safe. This paper presents a set of backward reachability approaches for safety certification of neural feedback loops (NFLs), i.e., closed-loop systems with NN control policies. While backward reachability strategies have… ▽ More As neural networks (NNs) become more prevalent in safety-critical applications such as control of vehicles, there is a growing need to certify that systems with NN components are safe. This paper presents a set of backward reachability approaches for safety certification of neural feedback loops (NFLs), i.e., closed-loop systems with NN control policies. While backward reachability strategies have been developed for systems without NN components, the nonlinearities in NN activation functions and general noninvertibility of NN weight matrices make backward reachability for NFLs a challenging problem. To avoid the difficulties associated with propagating sets backward through NNs, we introduce a framework that leverages standard forward NN analysis tools to efficiently find over-approximations to backprojection (BP) sets, i.e., sets of states for which an NN policy will lead a system to a given target set. We present frameworks for calculating BP over approximations for both linear and nonlinear systems with control policies represented by feedforward NNs and propose computationally efficient strategies. We use numerical results from a variety of models to showcase the proposed algorithms, including a demonstration of safety certification for a 6D system. △ Less

Submitted 21 November, 2022; v1 submitted 28 September, 2022; originally announced September 2022.

Comments: 17 pages, 15 figures. Journal extension of arXiv:2204.08319

Strategic Asset Allocation with Illiquid Alternatives

Authors: Eric Luxenberg, Stephen Boyd, Mykel Kochenderfer, Misha van Beek, Wen Cao, Steven Diamond, Alex Ulitsky, Kunal Menda, Vidy Vairavamurthy

Abstract: We address the problem of strategic asset allocation (SAA) with portfolios that include illiquid alternative asset classes. The main challenge in portfolio construction with illiquid asset classes is that we do not have direct control over our positions, as we do in liquid asset classes. Instead we can only make commitments; the position builds up over time as capital calls come in, and reduces ov… ▽ More We address the problem of strategic asset allocation (SAA) with portfolios that include illiquid alternative asset classes. The main challenge in portfolio construction with illiquid asset classes is that we do not have direct control over our positions, as we do in liquid asset classes. Instead we can only make commitments; the position builds up over time as capital calls come in, and reduces over time as distributions occur, neither of which the investor has direct control over. The effect on positions of our commitments is subject to a delay, typically of a few years, and is also unknown or stochastic. A further challenge is the requirement that we can meet the capital calls, with very high probability, with our liquid assets. We formulate the illiquid dynamics as a random linear system, and propose a convex optimization based model predictive control (MPC) policy for allocating liquid assets and making new illiquid commitments in each period. Despite the challenges of time delay and uncertainty, we show that this policy attains performance surprisingly close to a fictional setting where we pretend the illiquid asset classes are completely liquid, and we can arbitrarily and immediately adjust our positions. In this paper we focus on the growth problem, with no external liabilities or income, but the method is readily extended to handle this case. △ Less

Submitted 15 July, 2022; originally announced July 2022.

Collision Risk and Operational Impact of Speed Change Advisories as Aircraft Collision Avoidance Maneuvers

Authors: Sydney M. Katz, Luis E. Alvarez, Michael Owen, Samuel Wu, Marc Brittain, Anshuman Das, Mykel J. Kochenderfer

Abstract: Aircraft collision avoidance systems have long been a key factor in kee** our airspace safe. Over the past decade, the FAA has supported the development of a new family of collision avoidance systems called the Airborne Collision Avoidance System X (ACAS X), which model the collision avoidance problem as a Markov decision process (MDP). Variants of ACAS X have been created for both manned (ACAS… ▽ More Aircraft collision avoidance systems have long been a key factor in kee** our airspace safe. Over the past decade, the FAA has supported the development of a new family of collision avoidance systems called the Airborne Collision Avoidance System X (ACAS X), which model the collision avoidance problem as a Markov decision process (MDP). Variants of ACAS X have been created for both manned (ACAS Xa) and unmanned aircraft (ACAS Xu and ACAS sXu). The variants primarily differ in the types of collision avoidance maneuvers they issue. For example, ACAS Xa issues vertical collision avoidance advisories, while ACAS Xu and ACAS sXu allow for horizontal advisories due to reduced aircraft performance capabilities. Currently, a new variant of ACAS X, called ACAS Xr, is being developed to provide collision avoidance capability to rotorcraft and Advanced Air Mobility (AAM) vehicles. Due to the desire to minimize deviation from the prescribed flight path of these aircraft, speed adjustments have been proposed as a potential collision avoidance maneuver for aircraft using ACAS Xr. In this work, we investigate the effect of speed change advisories on the safety and operational efficiency of collision avoidance systems. We develop an MDP-based collision avoidance logic that issues speed advisories and compare its performance to that of horizontal and vertical logics through Monte Carlo simulation on existing airspace encounter models. Our results show that while speed advisories are able to reduce collision risk, they are neither as safe nor as efficient as their horizontal and vertical counterparts. △ Less

Submitted 29 April, 2022; originally announced April 2022.

Comments: 10 pages, 6 figures, presented at the 2022 AIAA Aviation Forum

Model Predictive Optimized Path Integral Strategies

Authors: Dylan M. Asmar, Ransalu Senanayake, Shawn Manuel, Mykel J. Kochenderfer

Abstract: We generalize the derivation of model predictive path integral control (MPPI) to allow for a single joint distribution across controls in the control sequence. This reformation allows for the implementation of adaptive importance sampling (AIS) algorithms into the original importance sampling step while still maintaining the benefits of MPPI such as working with arbitrary system dynamics and cost… ▽ More We generalize the derivation of model predictive path integral control (MPPI) to allow for a single joint distribution across controls in the control sequence. This reformation allows for the implementation of adaptive importance sampling (AIS) algorithms into the original importance sampling step while still maintaining the benefits of MPPI such as working with arbitrary system dynamics and cost functions. The benefit of optimizing the proposal distribution by integrating AIS at each control step is demonstrated in simulated environments including controlling multiple cars around a track. The new algorithm is more sample efficient than MPPI, achieving better performance with fewer samples. This performance disparity grows as the dimension of the action space increases. Results from simulations suggest the new algorithm can be used as an anytime algorithm, increasing the value of control at each iteration versus relying on a large set of samples. △ Less

Submitted 1 March, 2023; v1 submitted 30 March, 2022; originally announced March 2022.

Comments: Repository: https://github.com/sisl/MPOPIS. Accepted to ICRA 2023

ACM Class: I.2.8; I.2.9

Dyadic Sex Composition and Task Classification Using fNIRS Hyperscanning Data

Authors: Liam A. Kruse, Allan L. Reiss, Mykel J. Kochenderfer, Stephanie Balters

Abstract: Hyperscanning with functional near-infrared spectroscopy (fNIRS) is an emerging neuroimaging application that measures the nuanced neural signatures underlying social interactions. Researchers have assessed the effect of sex and task type (e.g., cooperation versus competition) on inter-brain coherence during human-to-human interactions. However, no work has yet used deep learning-based approaches… ▽ More Hyperscanning with functional near-infrared spectroscopy (fNIRS) is an emerging neuroimaging application that measures the nuanced neural signatures underlying social interactions. Researchers have assessed the effect of sex and task type (e.g., cooperation versus competition) on inter-brain coherence during human-to-human interactions. However, no work has yet used deep learning-based approaches to extract insights into sex and task-based differences in an fNIRS hyperscanning context. This work proposes a convolutional neural network-based approach to dyadic sex composition and task classification for an extensive hyperscanning dataset with $N = 222$ participants. Inter-brain signal similarity computed using dynamic time war** is used as the input data. The proposed approach achieves a maximum classification accuracy of greater than $80$ percent, thereby providing a new avenue for exploring and understanding complex brain behavior. △ Less

Submitted 6 December, 2021; originally announced December 2021.

Comments: 20th IEEE International Conference on Machine Learning and Applications

OVERT: An Algorithm for Safety Verification of Neural Network Control Policies for Nonlinear Systems

Authors: Chelsea Sidrane, Amir Maleki, Ahmed Irfan, Mykel J. Kochenderfer

Abstract: Deep learning methods can be used to produce control policies, but certifying their safety is challenging. The resulting networks are nonlinear and often very large. In response to this challenge, we present OVERT: a sound algorithm for safety verification of nonlinear discrete-time closed loop dynamical systems with neural network control policies. The novelty of OVERT lies in combining ideas fro… ▽ More Deep learning methods can be used to produce control policies, but certifying their safety is challenging. The resulting networks are nonlinear and often very large. In response to this challenge, we present OVERT: a sound algorithm for safety verification of nonlinear discrete-time closed loop dynamical systems with neural network control policies. The novelty of OVERT lies in combining ideas from the classical formal methods literature with ideas from the newer neural network verification literature. The central concept of OVERT is to abstract nonlinear functions with a set of optimally tight piecewise linear bounds. Such piecewise linear bounds are designed for seamless integration into ReLU neural network verification tools. OVERT can be used to prove bounded-time safety properties by either computing reachable sets or solving feasibility queries directly. We demonstrate various examples of safety verification for several classical benchmark examples. OVERT compares favorably to existing methods both in computation time and in tightness of the reachable set. △ Less

Submitted 2 August, 2021; originally announced August 2021.

Comments: 44 pages, under review

MSC Class: 68Q60 (Primary) 68T07; 37N35 (Secondary) ACM Class: I.2.6; I.2.8; D.2.4

Journal ref: Journal of Machine Learning Research 23 (2022) 1-45

Runtime Safety Assurance Using Reinforcement Learning

Authors: Christopher Lazarus, James G. Lopez, Mykel J. Kochenderfer

Abstract: The airworthiness and safety of a non-pedigreed autopilot must be verified, but the cost to formally do so can be prohibitive. We can bypass formal verification of non-pedigreed components by incorporating Runtime Safety Assurance (RTSA) as mechanism to ensure safety. RTSA consists of a meta-controller that observes the inputs and outputs of a non-pedigreed component and verifies formally specifie… ▽ More The airworthiness and safety of a non-pedigreed autopilot must be verified, but the cost to formally do so can be prohibitive. We can bypass formal verification of non-pedigreed components by incorporating Runtime Safety Assurance (RTSA) as mechanism to ensure safety. RTSA consists of a meta-controller that observes the inputs and outputs of a non-pedigreed component and verifies formally specified behavior as the system operates. When the system is triggered, a verified recovery controller is deployed. Recovery controllers are designed to be safe but very likely disruptive to the operational objective of the system, and thus RTSA systems must balance safety and efficiency. The objective of this paper is to design a meta-controller capable of identifying unsafe situations with high accuracy. High dimensional and non-linear dynamics in which modern controllers are deployed along with the black-box nature of the nominal controllers make this a difficult problem. Current approaches rely heavily on domain expertise and human engineering. We frame the design of RTSA with the Markov decision process (MDP) framework and use reinforcement learning (RL) to solve it. Our learned meta-controller consistently exhibits superior performance in our experiments compared to our baseline, human engineered approach. △ Less

Submitted 20 October, 2020; originally announced October 2020.

Journal ref: 2020 IEEE/AIAA 39th Digital Avionics Systems Conference (DASC)

A Maximum Independent Set Method for Scheduling Earth Observing Satellite Constellations

Authors: Duncan Eddy, Mykel J. Kochenderfer

Abstract: Operating Earth observing satellites requires efficient planning methods that coordinate activities of multiple spacecraft. The satellite task planning problem entails selecting actions that best satisfy mission objectives for autonomous execution. Task scheduling is often performed by human operators assisted by heuristic or rule-based planning tools. This approach does not efficiently scale to m… ▽ More Operating Earth observing satellites requires efficient planning methods that coordinate activities of multiple spacecraft. The satellite task planning problem entails selecting actions that best satisfy mission objectives for autonomous execution. Task scheduling is often performed by human operators assisted by heuristic or rule-based planning tools. This approach does not efficiently scale to multiple assets as heuristics frequently fail to properly coordinate actions of multiple vehicles over long horizons. Additionally, the problem becomes more difficult to solve for large constellations as the complexity of the problem scales exponentially in the number of requested observations and linearly in the number of spacecraft. It is expected that new commercial optical and radar imaging constellations will require automated planning methods to meet stated responsiveness and throughput objectives. This paper introduces a new approach for solving the satellite scheduling problem by generating an infeasibility-based graph representation of the problem and finding a maximal independent set of vertices for the graph. The approach is tested on a scenarios of up to 10,000 requested imaging locations for the Skysat constellation of optical satellites as well as simulated constellations of up to 24 satellites. Performance is compared with contemporary graph-traversal and mixed-integer linear programming approaches. Empirical results demonstrate improvements in both the solution time along with the number of scheduled collections beyond baseline methods. For large problems, the maximum independent set approach is able find a feasible schedule with 8% more collections in 75% less time. △ Less

Submitted 15 August, 2020; originally announced August 2020.

Scalable Identification of Partially Observed Systems with Certainty-Equivalent EM

Authors: Kunal Menda, Jean de Becdelièvre, Jayesh K. Gupta, Ilan Kroo, Mykel J. Kochenderfer, Zachary Manchester

Abstract: System identification is a key step for model-based control, estimator design, and output prediction. This work considers the offline identification of partially observed nonlinear systems. We empirically show that the certainty-equivalent approximation to expectation-maximization can be a reliable and scalable approach for high-dimensional deterministic systems, which are common in robotics. We f… ▽ More System identification is a key step for model-based control, estimator design, and output prediction. This work considers the offline identification of partially observed nonlinear systems. We empirically show that the certainty-equivalent approximation to expectation-maximization can be a reliable and scalable approach for high-dimensional deterministic systems, which are common in robotics. We formulate certainty-equivalent expectation-maximization as block coordinate-ascent, and provide an efficient implementation. The algorithm is tested on a simulated system of coupled Lorenz attractors, demonstrating its ability to identify high-dimensional systems that can be intractable for particle-based approaches. Our approach is also used to identify the dynamics of an aerobatic helicopter. By augmenting the state with unobserved fluid states, a model is learned that predicts the acceleration of the helicopter better than state-of-the-art approaches. The codebase for this work is available at https://github.com/sisl/CEEM. △ Less

Submitted 20 June, 2020; originally announced June 2020.

Comments: First three authors contributed equally. Accepted at ICML 2020. Website: https://sites.google.com/stanford.edu/ceem/

A Taxonomy and Review of Algorithms for Modeling and Predicting Human Driver Behavior

Authors: Kyle Brown, Katherine Driggs-Campbell, Mykel J. Kochenderfer

Abstract: We present a review and taxonomy of 200 models from the literature on driver behavior modeling. We begin by introducing a mathematical framework for describing the dynamics of interactive multi-agent traffic. Based on the partially observable stochastic game, this framework provides a basis for discussing different driver modeling techniques. Our taxonomy is constructed around the core modeling ta… ▽ More We present a review and taxonomy of 200 models from the literature on driver behavior modeling. We begin by introducing a mathematical framework for describing the dynamics of interactive multi-agent traffic. Based on the partially observable stochastic game, this framework provides a basis for discussing different driver modeling techniques. Our taxonomy is constructed around the core modeling tasks of state estimation, intention estimation, trait estimation, and motion prediction, and also discusses the auxiliary tasks of risk estimation, anomaly detection, behavior imitation and microscopic traffic simulation. Existing driver models are categorized based on the specific tasks they address and key attributes of their approach. △ Less

Submitted 28 November, 2020; v1 submitted 15 June, 2020; originally announced June 2020.

A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems

Authors: Anthony Corso, Robert J. Moss, Mark Koren, Ritchie Lee, Mykel J. Kochenderfer

Abstract: Autonomous cyber-physical systems (CPS) can improve safety and efficiency for safety-critical applications, but require rigorous testing before deployment. The complexity of these systems often precludes the use of formal verification and real-world testing can be too dangerous during development. Therefore, simulation-based techniques have been developed that treat the system under test as a blac… ▽ More Autonomous cyber-physical systems (CPS) can improve safety and efficiency for safety-critical applications, but require rigorous testing before deployment. The complexity of these systems often precludes the use of formal verification and real-world testing can be too dangerous during development. Therefore, simulation-based techniques have been developed that treat the system under test as a black box operating in a simulated environment. Safety validation tasks include finding disturbances in the environment that cause the system to fail (falsification), finding the most-likely failure, and estimating the probability that the system fails. Motivated by the prevalence of safety-critical artificial intelligence, this work provides a survey of state-of-the-art safety validation techniques for CPS with a focus on applied algorithms and their modifications for the safety validation problem. We present and discuss algorithms in the domains of optimization, path planning, reinforcement learning, and importance sampling. Problem decomposition techniques are presented to help scale algorithms to large state spaces, which are common for CPS. A brief overview of safety-critical applications is given, including autonomous vehicles and aircraft collision avoidance systems. Finally, we present a survey of existing academic and commercially available safety validation tools. △ Less

Submitted 14 October, 2021; v1 submitted 6 May, 2020; originally announced May 2020.

Journal ref: Journal of Artificial Intelligence Research, vol. 72, p. 377-428, 2021

Structured Mechanical Models for Robot Learning and Control

Authors: Jayesh K. Gupta, Kunal Menda, Zachary Manchester, Mykel J. Kochenderfer

Abstract: Model-based methods are the dominant paradigm for controlling robotic systems, though their efficacy depends heavily on the accuracy of the model used. Deep neural networks have been used to learn models of robot dynamics from data, but they suffer from data-inefficiency and the difficulty to incorporate prior knowledge. We introduce Structured Mechanical Models, a flexible model class for mechani… ▽ More Model-based methods are the dominant paradigm for controlling robotic systems, though their efficacy depends heavily on the accuracy of the model used. Deep neural networks have been used to learn models of robot dynamics from data, but they suffer from data-inefficiency and the difficulty to incorporate prior knowledge. We introduce Structured Mechanical Models, a flexible model class for mechanical systems that are data-efficient, easily amenable to prior knowledge, and easily usable with model-based control techniques. The goal of this work is to demonstrate the benefits of using Structured Mechanical Models in lieu of black-box neural networks when modeling robot dynamics. We demonstrate that they generalize better from limited data and yield more reliable model-based controllers on a variety of simulated robotic domains. △ Less

Submitted 21 April, 2020; originally announced April 2020.

Comments: First two authors contributed equally. Accepted at L4DC2020. Source code and videos at https://sites.google.com/stanford.edu/smm/

Scalable Autonomous Vehicle Safety Validation through Dynamic Programming and Scene Decomposition

Authors: Anthony Corso, Ritchie Lee, Mykel J. Kochenderfer

Abstract: An open question in autonomous driving is how best to use simulation to validate the safety of autonomous vehicles. Existing techniques rely on simulated rollouts, which can be inefficient for finding rare failure events, while other techniques are designed to only discover a single failure. In this work, we present a new safety validation approach that attempts to estimate the distribution over f… ▽ More An open question in autonomous driving is how best to use simulation to validate the safety of autonomous vehicles. Existing techniques rely on simulated rollouts, which can be inefficient for finding rare failure events, while other techniques are designed to only discover a single failure. In this work, we present a new safety validation approach that attempts to estimate the distribution over failures of an autonomous policy using approximate dynamic programming. Knowledge of this distribution allows for the efficient discovery of many failure examples. To address the problem of scalability, we decompose complex driving scenarios into subproblems consisting of only the ego vehicle and one other vehicle. These subproblems can be solved with approximate dynamic programming and their solutions are recombined to approximate the solution to the full scenario. We apply our approach to a simple two-vehicle scenario to demonstrate the technique as well as a more complex five-vehicle scenario to demonstrate scalability. In both experiments, we observed an increase in the number of failures discovered compared to baseline approaches. △ Less

Submitted 26 June, 2020; v1 submitted 14 April, 2020; originally announced April 2020.

The Adaptive Stress Testing Formulation

Authors: Mark Koren, Anthony Corso, Mykel J. Kochenderfer

Abstract: Validation is a key challenge in the search for safe autonomy. Simulations are often either too simple to provide robust validation, or too complex to tractably compute. Therefore, approximate validation methods are needed to tractably find failures without unsafe simplifications. This paper presents the theory behind one such black-box approach: adaptive stress testing (AST). We also provide thre… ▽ More Validation is a key challenge in the search for safe autonomy. Simulations are often either too simple to provide robust validation, or too complex to tractably compute. Therefore, approximate validation methods are needed to tractably find failures without unsafe simplifications. This paper presents the theory behind one such black-box approach: adaptive stress testing (AST). We also provide three examples of validation problems formulated to work with AST. △ Less

Submitted 8 April, 2020; originally announced April 2020.

Comments: Presented at the Workshop on Robust Autonomy at RSS 2019

Adaptive Stress Testing without Domain Heuristics using Go-Explore

Authors: Mark Koren, Mykel J. Kochenderfer

Abstract: Recently, reinforcement learning (RL) has been used as a tool for finding failures in autonomous systems. During execution, the RL agents often rely on some domain-specific heuristic reward to guide them towards finding failures, but constructing such a heuristic may be difficult or infeasible. Without a heuristic, the agent may only receive rewards at the time of failure, or even rewards that gui… ▽ More Recently, reinforcement learning (RL) has been used as a tool for finding failures in autonomous systems. During execution, the RL agents often rely on some domain-specific heuristic reward to guide them towards finding failures, but constructing such a heuristic may be difficult or infeasible. Without a heuristic, the agent may only receive rewards at the time of failure, or even rewards that guide it away from failures. For example, some approaches give rewards for taking more-likely actions, because we want to find more-likely failures. However, the agent may then learn to only take likely actions, and may not be able to find a failure at all. Consequently, the problem becomes a hard-exploration problem, where rewards do not aid exploration. A new algorithm, go-explore (GE), has recently set new records on benchmarks from the hard-exploration field. We apply GE to adaptive stress testing (AST), one example of an RL-based falsification approach that provides a way to search for the most-likely failure scenario. We simulate a scenario where an autonomous vehicle drives while a pedestrian is crossing the road. We demonstrate that GE is able to find failures without domain-specific heuristics, such as the distance between the car and the pedestrian, on scenarios that other RL techniques are unable to solve. Furthermore, inspired by the robustification phase of GE, we demonstrate that the backwards algorithm (BA) improves the failures found by other RL techniques. △ Less

Submitted 18 June, 2020; v1 submitted 8 April, 2020; originally announced April 2020.

Comments: Accepted to ITSC 2020

Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing

Authors: Kyle D. Julian, Ritchie Lee, Mykel J. Kochenderfer

Abstract: Neural networks have become state-of-the-art for computer vision problems because of their ability to efficiently model complex functions from large amounts of data. While neural networks can be shown to perform well empirically for a variety of tasks, their performance is difficult to guarantee. Neural network verification tools have been developed that can certify robustness with respect to a gi… ▽ More Neural networks have become state-of-the-art for computer vision problems because of their ability to efficiently model complex functions from large amounts of data. While neural networks can be shown to perform well empirically for a variety of tasks, their performance is difficult to guarantee. Neural network verification tools have been developed that can certify robustness with respect to a given input image; however, for neural network systems used in closed-loop controllers, robustness with respect to individual images does not address multi-step properties of the neural network controller and its environment. Furthermore, neural network systems interacting in the physical world and using natural images are operating in a black-box environment, making formal verification intractable. This work combines the adaptive stress testing (AST) framework with neural network verification tools to search for the most likely sequence of image disturbances that cause the neural network controlled system to reach a failure. An autonomous aircraft taxi application is presented, and results show that the AST method finds failures with more likely image disturbances than baseline methods. Further analysis of AST results revealed an explainable cause of the failure, giving insight into the problematic scenarios that should be addressed. △ Less

Submitted 4 March, 2020; originally announced March 2020.

Comments: 7 pages, 6 figures

Optimizing Collision Avoidance in Dense Airspace using Deep Reinforcement Learning

Authors: Sheng Li, Maxim Egorov, Mykel Kochenderfer

Abstract: New methodologies will be needed to ensure the airspace remains safe and efficient as traffic densities rise to accommodate new unmanned operations. This paper explores how unmanned free-flight traffic may operate in dense airspace. We develop and analyze autonomous collision avoidance systems for aircraft operating in dense airspace where traditional collision avoidance systems fail. We propose a… ▽ More New methodologies will be needed to ensure the airspace remains safe and efficient as traffic densities rise to accommodate new unmanned operations. This paper explores how unmanned free-flight traffic may operate in dense airspace. We develop and analyze autonomous collision avoidance systems for aircraft operating in dense airspace where traditional collision avoidance systems fail. We propose a metric for quantifying the decision burden on a collision avoidance system as well as a metric for measuring the impact of the collision avoidance system on airspace. We use deep reinforcement learning to compute corrections for an existing collision avoidance approach to account for dense airspace. The results show that a corrected collision avoidance system can operate more efficiently than traditional methods in dense airspace while maintaining high levels of safety. △ Less

Submitted 20 December, 2019; originally announced December 2019.

Comments: Thirteenth USA/Europe Air Traffic Management Research and Development Seminar

Guaranteeing Safety for Neural Network-Based Aircraft Collision Avoidance Systems

Authors: Kyle D. Julian, Mykel J. Kochenderfer

Abstract: The decision logic for the ACAS X family of aircraft collision avoidance systems is represented as a large numeric table. Due to storage constraints of certified avionics hardware, neural networks have been suggested as a way to significantly compress the data while still preserving performance in terms of safety. However, neural networks are complex continuous functions with outputs that are diff… ▽ More The decision logic for the ACAS X family of aircraft collision avoidance systems is represented as a large numeric table. Due to storage constraints of certified avionics hardware, neural networks have been suggested as a way to significantly compress the data while still preserving performance in terms of safety. However, neural networks are complex continuous functions with outputs that are difficult to predict. Because simulations evaluate only a finite number of encounters, simulations are not sufficient to guarantee that the neural network will perform correctly in all possible situations. We propose a method to provide safety guarantees when using a neural network collision avoidance system. The neural network outputs are bounded using neural network verification tools like Reluplex and Reluval, and a reachability method determines all possible ways aircraft encounters will resolve using neural network advisories and assuming bounded aircraft dynamics. Experiments with systems inspired by ACAS X show that neural networks giving either horizontal or vertical maneuvers can be proven safe. We explore how relaxing the bounds on aircraft dynamics can lead to potentially unsafe encounters and demonstrate how neural network controllers can be modified to guarantee safety through online costs or lowering alerting cost. The reachability method is flexible and can incorporate uncertainties such as pilot delay and sensor error. These results suggest a method for certifying neural network collision avoidance systems for use in real aircraft. △ Less

Submitted 5 May, 2020; v1 submitted 15 December, 2019; originally announced December 2019.

Comments: 10 pages, 11 figures, presented at the 2019 AIAA Digital Avionics Systems Conference (DASC)

Journal ref: IEEE/AIAA 38th Digital Avionics Systems Conference (DASC). 2019

Markov Decision Processes For Multi-Objective Satellite Task Planning

Authors: Duncan Eddy, Mykel Kochenderfer

Abstract: This paper presents a semi-Markov decision process (SMDP) formulation of the satellite task scheduling problem. This formulation can consider multiple operational objectives simultaneously and plan transitions between distinct functional modes. We consider the problem of scheduling image collections, ground contacts, sun-pointed periods for battery recharging, and data recorder management for an a… ▽ More This paper presents a semi-Markov decision process (SMDP) formulation of the satellite task scheduling problem. This formulation can consider multiple operational objectives simultaneously and plan transitions between distinct functional modes. We consider the problem of scheduling image collections, ground contacts, sun-pointed periods for battery recharging, and data recorder management for an agile, resource-constrained Earth-observing spacecraft. By considering multiple mission objectives simultaneously, the algorithm is able to find optimized task schedule that satisfies all operational constraints in a single planning step, thus reducing the operational complexity and number of steps involved in mission planning. We present two solution approaches based on forward search and Monte Carlo Tree search. We baseline against rule-based, graph search, and mixed-integer-linear programming approaches. The SMDP formulation is evaluated in both single-objective and multi-objective scenarios. The SMDP solution is found to perform comparably with the baseline methods at greatly increased speed in single-objective scenarios and greater schedule reward in multi-objective. △ Less

Submitted 18 October, 2019; originally announced October 2019.

Comments: 11 pages, 10 figures, Submitted to IEEE Aerospace Conference 2020

Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation

Authors: Anthony Corso, Peter Du, Katherine Driggs-Campbell, Mykel J. Kochenderfer

Abstract: Determining possible failure scenarios is a critical step in the evaluation of autonomous vehicle systems. Real-world vehicle testing is commonly employed for autonomous vehicle validation, but the costs and time requirements are high. Consequently, simulation-driven methods such as Adaptive Stress Testing (AST) have been proposed to aid in validation. AST formulates the problem of finding the mos… ▽ More Determining possible failure scenarios is a critical step in the evaluation of autonomous vehicle systems. Real-world vehicle testing is commonly employed for autonomous vehicle validation, but the costs and time requirements are high. Consequently, simulation-driven methods such as Adaptive Stress Testing (AST) have been proposed to aid in validation. AST formulates the problem of finding the most likely failure scenarios as a Markov decision process, which can be solved using reinforcement learning. In practice, AST tends to find scenarios where failure is unavoidable and tends to repeatedly discover the same types of failures of a system. This work addresses these issues by encoding domain relevant information into the search procedure. With this modification, the AST method discovers a larger and more expressive subset of the failure space when compared to the original AST formulation. We show that our approach is able to identify useful failure scenarios of an autonomous vehicle policy. △ Less

Submitted 6 August, 2019; v1 submitted 2 August, 2019; originally announced August 2019.

Comments: Appears in IEEE ITSC 2019

Efficient Autonomy Validation in Simulation with Adaptive Stress Testing

Authors: Mark Koren, Mykel Kochenderfer

Abstract: During the development of autonomous systems such as driverless cars, it is important to characterize the scenarios that are most likely to result in failure. Adaptive Stress Testing (AST) provides a way to search for the most-likely failure scenario as a Markov decision process (MDP). Our previous work used a deep reinforcement learning (DRL) solver to identify likely failure scenarios. However,… ▽ More During the development of autonomous systems such as driverless cars, it is important to characterize the scenarios that are most likely to result in failure. Adaptive Stress Testing (AST) provides a way to search for the most-likely failure scenario as a Markov decision process (MDP). Our previous work used a deep reinforcement learning (DRL) solver to identify likely failure scenarios. However, the solver's use of a feed-forward neural network with a discretized space of possible initial conditions poses two major problems. First, the system is not treated as a black box, in that it requires analyzing the internal state of the system, which leads to considerable implementation complexities. Second, in order to simulate realistic settings, a new instance of the solver needs to be run for each initial condition. Running a new solver for each initial condition not only significantly increases the computational complexity, but also disregards the underlying relationship between similar initial conditions. We provide a solution to both problems by employing a recurrent neural network that takes a set of initial conditions from a continuous space as input. This approach enables robust and efficient detection of failures because the solution generalizes across the entire space of initial conditions. By simulating an instance where an autonomous car drives while a pedestrian is crossing a road, we demonstrate the solver is now capable of finding solutions for problems that would have previously been intractable. △ Less

Submitted 15 July, 2019; originally announced July 2019.

Comments: Submitted to IEEE ITSC 2019

Satellite Image Tasking Under Orbit Prediction Uncertainty

Authors: Duncan Eddy, Mykel Kochenderfer

Abstract: Small satellites have proven to be viable Earth observation platforms. These satellites operate in regimes of increased trajectory uncertainty where traditional planning approaches can lead to sub-optimal task plans, limiting science return. Previous formulations of the space mission planning problem decouple trajectory prediction and planning, which leads to task plans that are less robust to unc… ▽ More Small satellites have proven to be viable Earth observation platforms. These satellites operate in regimes of increased trajectory uncertainty where traditional planning approaches can lead to sub-optimal task plans, limiting science return. Previous formulations of the space mission planning problem decouple trajectory prediction and planning, which leads to task plans that are less robust to uncertainty. We present a Markov decision process formulation of the problem that accounts for uncertainties by incorporating a distribution of possible collection windows characterized through Monte Carlo simulation. An approximate solution technique yields tasking schedules with rewards comparable to the conventional methods while simultaneously reducing the variations caused by uncertainties and improving runtime. △ Less

Submitted 3 May, 2019; originally announced May 2019.

Comments: 7 pages, 2 figures, 3 algorithms, 5 tables. Submitted to IJCAI 2019

Rethinking System Health Management

Authors: Edward Balaban, Stephen B. Johnson, Mykel J. Kochenderfer

Abstract: Health management of complex dynamic systems has traditionally evolved separately from automated control, planning, and scheduling (generally referred to in the paper as decision making). A goal of Integrated System Health Management has been to enable coordination between system health management and decision making, although successful practical implementations have remained limited. This paper… ▽ More Health management of complex dynamic systems has traditionally evolved separately from automated control, planning, and scheduling (generally referred to in the paper as decision making). A goal of Integrated System Health Management has been to enable coordination between system health management and decision making, although successful practical implementations have remained limited. This paper proposes that, rather than being treated as connected, yet distinct entities, system health management and decision making should be unified in their formulations. Enabled by advances in modeling and computing, we argue that the unified approach will increase a system's operational effectiveness and may also lead to a lower overall system complexity. We overview the prevalent system health management methodology and illustrate its limitations through numerical examples. We then describe the proposed unification approach and show how it accommodates the typical system health management concepts. △ Less

Submitted 10 March, 2019; originally announced March 2019.

Comments: Published in the proceedings of the 2018 AAAI Fall Symposium on Integrating Planning, Diagnosis, and Causal Reasoning

Verifying Aircraft Collision Avoidance Neural Networks Through Linear Approximations of Safe Regions

Authors: Kyle D. Julian, Shivam Sharma, Jean-Baptiste Jeannin, Mykel J. Kochenderfer

Abstract: The next generation of aircraft collision avoidance systems frame the problem as a Markov decision process and use dynamic programming to optimize the alerting logic. The resulting system uses a large lookup table to determine advisories given to pilots, but these tables can grow very large. To enable the system to operate on limited hardware, prior work investigated compressing the table using a… ▽ More The next generation of aircraft collision avoidance systems frame the problem as a Markov decision process and use dynamic programming to optimize the alerting logic. The resulting system uses a large lookup table to determine advisories given to pilots, but these tables can grow very large. To enable the system to operate on limited hardware, prior work investigated compressing the table using a deep neural network. However, ensuring that the neural network reliably issues safe advisories is important for certification. This work defines linearized regions where each advisory can be safely provided, allowing Reluplex, a neural network verification tool, to check if unsafe advisories are ever issued. A notional collision avoidance policy is generated and used to train a neural network representation. The neural networks are checked for unsafe advisories, resulting in the discovery of thousands of unsafe counterexamples. △ Less

Submitted 2 March, 2019; originally announced March 2019.

A Reachability Method for Verifying Dynamical Systems with Deep Neural Network Controllers

Authors: Kyle D. Julian, Mykel J. Kochenderfer

Abstract: Deep neural networks can be trained to be efficient and effective controllers for dynamical systems; however, the mechanics of deep neural networks are complex and difficult to guarantee. This work presents a general approach for providing guarantees for deep neural network controllers over multiple time steps using a combination of reachability methods and open source neural network verification… ▽ More Deep neural networks can be trained to be efficient and effective controllers for dynamical systems; however, the mechanics of deep neural networks are complex and difficult to guarantee. This work presents a general approach for providing guarantees for deep neural network controllers over multiple time steps using a combination of reachability methods and open source neural network verification tools. By bounding the system dynamics and neural network outputs, the set of reachable states can be over-approximated to provide a guarantee that the system will never reach states outside the set. The method is demonstrated on the mountain car problem as well as an aircraft collision avoidance problem. Results show that this approach can provide neural network guarantees given a bounded dynamic model. △ Less

Submitted 3 June, 2019; v1 submitted 1 March, 2019; originally announced March 2019.

A General Framework for Structured Learning of Mechanical Systems

Authors: Jayesh K. Gupta, Kunal Menda, Zachary Manchester, Mykel J. Kochenderfer

Abstract: Learning accurate dynamics models is necessary for optimal, compliant control of robotic systems. Current approaches to white-box modeling using analytic parameterizations, or black-box modeling using neural networks, can suffer from high bias or high variance. We address the need for a flexible, gray-box model of mechanical systems that can seamlessly incorporate prior knowledge where it is avail… ▽ More Learning accurate dynamics models is necessary for optimal, compliant control of robotic systems. Current approaches to white-box modeling using analytic parameterizations, or black-box modeling using neural networks, can suffer from high bias or high variance. We address the need for a flexible, gray-box model of mechanical systems that can seamlessly incorporate prior knowledge where it is available, and train expressive function approximators where it is not. We propose to parameterize a mechanical system using neural networks to model its Lagrangian and the generalized forces that act on it. We test our method on a simulated, actuated double pendulum. We show that our method outperforms a naive, black-box model in terms of data-efficiency, as well as performance in model-based reinforcement learning. We also conduct a systematic study of our method's ability to incorporate available prior knowledge about the system to improve data efficiency. △ Less

Submitted 1 March, 2019; v1 submitted 22 February, 2019; originally announced February 2019.

Comments: 10 pages, 7 figures. First two authors contributed equally. Submitted to IROS/RA-L. Code at https://github.com/sisl/mechamodlearn/

Using Neural Networks to Generate Information Maps for Mobile Sensors

Authors: Louis Dressel, Mykel J. Kochenderfer

Abstract: Target localization is a critical task for mobile sensors and has many applications. However, generating informative trajectories for these sensors is a challenging research problem. A common method uses information maps that estimate the value of taking measurements from any point in the sensor state space. These information maps are used to generate trajectories; for example, a trajectory might… ▽ More Target localization is a critical task for mobile sensors and has many applications. However, generating informative trajectories for these sensors is a challenging research problem. A common method uses information maps that estimate the value of taking measurements from any point in the sensor state space. These information maps are used to generate trajectories; for example, a trajectory might be designed so its distribution of measurements matches the distribution of the information map. Regardless of the trajectory generation method, generating information maps as new observations are made is critical. However, it can be challenging to compute these maps in real-time. We propose using convolutional neural networks to generate information maps from a target estimate and sensor model in real-time. Simulations show that maps are accurately rendered while offering orders of magnitude reduction in computation time. △ Less

Submitted 26 September, 2018; originally announced September 2018.

Comments: Accepted to the 2018 IEEE Conference on Decision and Control (CDC)

On the Optimality of Ergodic Trajectories for Information Gathering Tasks

Authors: Louis Dressel, Mykel J. Kochenderfer

Abstract: Recently, ergodic control has been suggested as a means to guide mobile sensors for information gathering tasks. In ergodic control, a mobile sensor follows a trajectory that is ergodic with respect to some information density distribution. A trajectory is ergodic if time spent in a state space region is proportional to the information density of the region. Although ergodic control has shown prom… ▽ More Recently, ergodic control has been suggested as a means to guide mobile sensors for information gathering tasks. In ergodic control, a mobile sensor follows a trajectory that is ergodic with respect to some information density distribution. A trajectory is ergodic if time spent in a state space region is proportional to the information density of the region. Although ergodic control has shown promising experimental results, there is little understanding of why it works or when it is optimal. In this paper, we study a problem class under which optimal information gathering trajectories are ergodic. This class relies on a submodularity assumption for repeated measurements from the same state. It is assumed that information available in a region decays linearly with time spent there. This assumption informs selection of the horizon used in ergodic trajectory generation. We support our claims with a set of experiments that demonstrate the link between ergodicity, optimal information gathering, and submodularity. △ Less

Submitted 20 August, 2018; originally announced August 2018.

Comments: Presented at 2018 American Control Conference (ACC)

Estimation and Control Using Sampling-Based Bayesian Reinforcement Learning

Authors: Patrick Slade, Zachary N. Sunberg, Mykel J. Kochenderfer

Abstract: Real-world autonomous systems operate under uncertainty about both their pose and dynamics. Autonomous control systems must simultaneously perform estimation and control tasks to maintain robustness to changing dynamics or modeling errors. However, information gathering actions often conflict with optimal actions for reaching control objectives, requiring a trade-off between exploration and exploi… ▽ More Real-world autonomous systems operate under uncertainty about both their pose and dynamics. Autonomous control systems must simultaneously perform estimation and control tasks to maintain robustness to changing dynamics or modeling errors. However, information gathering actions often conflict with optimal actions for reaching control objectives, requiring a trade-off between exploration and exploitation. The specific problem setting considered here is for discrete-time nonlinear systems, with process noise, input-constraints, and parameter uncertainty. This article frames this problem as a Bayes-adaptive Markov decision process and solves it online using Monte Carlo tree search with an unscented Kalman filter to account for process noise and parameter uncertainty. This method is compared with certainty equivalent model predictive control and a tree search method that approximates the QMDP solution, providing insight into when information gathering is useful. Discrete time simulations characterize performance over a range of process noise and bounds on unknown parameters. An offline optimization method is used to select the Monte Carlo tree search parameters without hand-tuning. In lieu of recursive feasibility guarantees, a probabilistic bounding heuristic is offered that increases the probability of kee** the state within a desired region. △ Less

Submitted 31 July, 2018; originally announced August 2018.

Comments: 10 pages, 6 figures. arXiv admin note: text overlap with arXiv:1707.09055

Online algorithms for POMDPs with continuous state, action, and observation spaces

Authors: Zachary Sunberg, Mykel Kochenderfer

Abstract: Online solvers for partially observable Markov decision processes have been applied to problems with large discrete state spaces, but continuous state, action, and observation spaces remain a challenge. This paper begins by investigating double progressive widening (DPW) as a solution to this challenge. However, we prove that this modification alone is not sufficient because the belief representat… ▽ More Online solvers for partially observable Markov decision processes have been applied to problems with large discrete state spaces, but continuous state, action, and observation spaces remain a challenge. This paper begins by investigating double progressive widening (DPW) as a solution to this challenge. However, we prove that this modification alone is not sufficient because the belief representations in the search tree collapse to a single particle causing the algorithm to converge to a policy that is suboptimal regardless of the computation time. This paper proposes and evaluates two new algorithms, POMCPOW and PFT-DPW, that overcome this deficiency by using weighted particle filtering. Simulation results show that these modifications allow the algorithms to be successful where previous approaches fail. △ Less

Submitted 5 September, 2018; v1 submitted 18 September, 2017; originally announced September 2017.

Comments: Added Multilane section

Journal ref: Short version published in 2018 proceedings of the International Conference on Automated Planning and Scheduling (ICAPS)

Simultaneous active parameter estimation and control using sampling-based Bayesian reinforcement learning

Authors: Patrick Slade, Preston Culbertson, Zachary Sunberg, Mykel Kochenderfer

Abstract: Robots performing manipulation tasks must operate under uncertainty about both their pose and the dynamics of the system. In order to remain robust to modeling error and shifts in payload dynamics, agents must simultaneously perform estimation and control tasks. However, the optimal estimation actions are often not the optimal actions for accomplishing the control tasks, and thus agents trade betw… ▽ More Robots performing manipulation tasks must operate under uncertainty about both their pose and the dynamics of the system. In order to remain robust to modeling error and shifts in payload dynamics, agents must simultaneously perform estimation and control tasks. However, the optimal estimation actions are often not the optimal actions for accomplishing the control tasks, and thus agents trade between exploration and exploitation. This work frames the problem as a Bayes-adaptive Markov decision process and solves it online using Monte Carlo tree search and an extended Kalman filter to handle Gaussian process noise and parameter uncertainty in a continuous space. MCTS selects control actions to reduce model uncertainty and reach the goal state nearly optimally. Certainty equivalent model predictive control is used as a benchmark to compare performance in simulations with varying process noise and parameter uncertainty. △ Less

Submitted 27 July, 2017; originally announced July 2017.