-
An Efficient Off-Policy Reinforcement Learning Algorithm for the Continuous-Time LQR Problem
Authors:
Victor G. Lopez,
Matthias A. Müller
Abstract:
In this paper, an off-policy reinforcement learning algorithm is designed to solve the continuous-time LQR problem using only input-state data measured from the system. Different from other algorithms in the literature, we propose the use of a specific persistently exciting input as the exploration signal during the data collection step. We then show that, using this persistently excited data, the…
▽ More
In this paper, an off-policy reinforcement learning algorithm is designed to solve the continuous-time LQR problem using only input-state data measured from the system. Different from other algorithms in the literature, we propose the use of a specific persistently exciting input as the exploration signal during the data collection step. We then show that, using this persistently excited data, the solution of the matrix equation in our algorithm is guaranteed to exist and to be unique at every iteration. Convergence of the algorithm to the optimal control input is also proven. Moreover, we formulate the policy evaluation step as the solution of a Sylvester-transpose equation, which increases the efficiency of its solution. Finally, a method to determine a stabilizing policy to initialize the algorithm using only measured data is proposed.
△ Less
Submitted 31 March, 2023;
originally announced March 2023.
-
Informed Circular Fields for Global Reactive Obstacle Avoidance of Robotic Manipulators
Authors:
Marvin Becker,
Philipp Caspers,
Tom Hattendorf,
Torsten Lilge,
Sami Haddadin,
Matthias A. Müller
Abstract:
In this paper a global reactive motion planning framework for robotic manipulators in complex dynamic environments is presented. In particular, the circular field predictions (CFP) planner from Becker et al. (2021) is extended to ensure obstacle avoidance of the whole structure of a robotic manipulator. Towards this end, a motion planning framework is developed that leverages global information ab…
▽ More
In this paper a global reactive motion planning framework for robotic manipulators in complex dynamic environments is presented. In particular, the circular field predictions (CFP) planner from Becker et al. (2021) is extended to ensure obstacle avoidance of the whole structure of a robotic manipulator. Towards this end, a motion planning framework is developed that leverages global information about promising avoidance directions from arbitrary configuration space motion planners, resulting in improved global trajectories while reactively avoiding dynamic obstacles and decreasing the required computational power. The resulting motion planning framework is tested in multiple simulations with complex and dynamic obstacles and demonstrates great potential compared to existing motion planning approaches.
△ Less
Submitted 4 August, 2023; v1 submitted 12 December, 2022;
originally announced December 2022.
-
Motion Planning using Reactive Circular Fields: A 2D Analysis of Collision Avoidance and Goal Convergence
Authors:
Marvin Becker,
Johannes Köhler,
Sami Haddadin,
Matthias A. Müller
Abstract:
Recently, many reactive trajectory planning approaches were suggested in the literature because of their inherent immediate adaption in the ever more demanding cluttered and unpredictable environments of robotic systems. However, typically those approaches are only locally reactive without considering global path planning and no guarantees for simultaneous collision avoidance and goal convergence…
▽ More
Recently, many reactive trajectory planning approaches were suggested in the literature because of their inherent immediate adaption in the ever more demanding cluttered and unpredictable environments of robotic systems. However, typically those approaches are only locally reactive without considering global path planning and no guarantees for simultaneous collision avoidance and goal convergence can be given. In this paper, we study a recently developed circular field (CF)-based motion planner that combines local reactive control with global trajectory generation by adapting an artificial magnetic field such that multiple trajectories around obstacles can be evaluated. In particular, we provide a mathematically rigorous analysis of this planner in a planar environment to ensure safe motion of the controlled robot. Contrary to existing results, the derived collision avoidance analysis covers the entire CF motion planning algorithm including attractive forces for goal convergence and is not limited to a specific choice of the rotation field, i.e., our guarantees are not limited to a specific potentially suboptimal trajectory. Our Lyapunov-type collision avoidance analysis is based on the definition of an (equivalent) two-dimensional auxiliary system, which enables us to provide tight, if and only if conditions for the case of a collision with point obstacles. Furthermore, we show how this analysis naturally extends to multiple obstacles and we specify sufficient conditions for goal convergence. Finally, we provide a challenging simulation scenario with multiple non-convex point cloud obstacles and demonstrate collision avoidance and goal convergence.
△ Less
Submitted 3 November, 2023; v1 submitted 28 October, 2022;
originally announced October 2022.
