Contextualized Hybrid Ensemble Q-learning:
Learning Fast with Control Priors
Abstract
Combining Reinforcement Learning (RL) with a prior controller can yield the best out of two worlds: RL can solve complex nonlinear problems, while the control prior ensures safer exploration and speeds up training. Prior work largely blends both components with a fixed weight, neglecting that the RL agent’s performance varies with the training progress and across regions in the state space. Therefore, we advocate for an adaptive strategy that dynamically adjusts the weighting based on the RL agent’s current capabilities. We propose a new adaptive hybrid RL algorithm, Contextualized Hybrid Ensemble Q-learning (CHEQ). CHEQ combines three key ingredients: (i) a time-invariant formulation of the adaptive hybrid RL problem treating the adaptive weight as a context variable, (ii) a weight adaption mechanism based on the parametric uncertainty of a critic ensemble, and (iii) ensemble-based acceleration for data-efficient RL. Evaluating CHEQ on a car racing task reveals substantially stronger data efficiency, exploration safety, and transferability to unknown scenarios than state-of-the-art adaptive hybrid RL methods.
1 Introduction
Deep reinforcement learning (RL) methods have shown great success in challenging control problems such as gameplay (Mnih et al., 2015; Silver et al., 2018a; OpenAI et al., 2019) and robotic manipulation (Gupta et al., 2021; Büchler et al., 2022). Despite the great potential of RL methods, their data inefficiency, unstructured exploration behavior, and inability to generalize to unknown scenarios represent a significant hurdle to their application to real-world problems.
A prime reason for limited real-world applications is the task-agnostic architecture of state-of-the-art RL approaches (Schulman et al., 2017; Haarnoja et al., 2018) that does not incorporate prior knowledge on how to solve the task at hand. In contrast, control theory provides a rich set of methods for deriving near-optimal controllers in many applications. This motivates the drive for hybrid RL methods (Silver et al., 2018b; Johannink et al., 2019) that blend control priors with deep RL policies. Hybrid algorithms thus combine the prior controller’s generalization capabilities and informed behavior with the power of deep RL for solving general nonlinear problems.
Notwithstanding the conceptual benefit of hybrid RL formulations, how to systematically combine the control prior with the RL agent largely remains an open problem. The majority of prior work (Silver et al., 2018b; Johannink et al., 2019; Schoettler et al., 2020; Ceola et al., 2024) proposes a fixed weighting between the control prior and the RL agent. A fixed blending, however, disregards the fact that the capability of the RL agent depends on training time and state. In general, as more data is observed, the RL agent improves its behavior, ultimately outperforming the control prior in large portions of the domain. The core idea of our approach is to adapt the weighting between RL agent and control prior based on the agent’s confidence. As the RL agent improves over time, this induces a time-variant weighting mechanism. This time dependency leads to structural problems of prior formulations in uncertainty-adapted hybrid RL (Cheng et al., 2019; Rana et al., 2023).
We provide a unified view on hybrid RL that allows us to classify prior work within a general framework. Analyzing this framework highlights the necessity for a novel adaptive hybrid RL formulation with descriptive, time-invariant dynamics. We define the contextualized hybrid Markov decision process (MDP), introducing the adaptive weight as a context variable. Building upon this formulation, we propose the Contextualized Hybrid Ensemble Q-learning (CHEQ) algorithm that systematically adapts the weighting between RL agent and control prior based on an uncertainty estimate of a critic ensemble. CHEQ combines the contextualized hybrid RL formulation with uncertainty-based weight adaption and existing ensemble-based acceleration techniques for data-efficient RL.
We evaluate our algorithm on a racing task (Schier et al., 2023), which requires operating a car close to its stability limits in order to achieve maximum return. We find that compared to prior work in adaptive hybrid RL, the CHEQ algorithm shows (i) reduced failures during training, (ii) increased sample efficiency, and (iii) improved transfer behavior on unseen race tracks.
In summary, our main contributions are:
-
•
A unified framework that allows us to classify existing approaches and reveal key limitations.
-
•
A hybrid MDP formulation, introducing the adaptive weight as a context variable and thus addressing structural problems of prior work in hybrid RL with adaptive weighting.
-
•
A novel hybrid RL algorithm, CHEQ, that systematically adapts the weighting between RL agent and control prior based on Q-ensemble disagreement.
2 Related Work
This section discusses relevant prior work combining RL and a control prior. We distinguish two types; hybrid RL with fixed and adaptive weighting between the RL agent and controller.
Hybrid Reinforcement Learning with Fixed Weighting. Two concurrent works (Silver et al., 2018b; Johannink et al., 2019) first combined RL and a control prior and introduced the term residual RL. In residual RL, the control prior is assumed to be fixed, and the RL agent learns a residual on top of this. In this work, we use the general term hybrid RL to include approaches that adapt the controller’s weight. Silver et al. (2018a) and Johannink et al. (2019) show advantages of hybrid RL, such as sample efficiency, improved sim-to-real transfer, and robustness towards uncertainties. Hybrid RL with fixed weights has then successfully been applied to real robot insertion tasks (Schoettler et al., 2020), peg insertion under uncertainty (Ranjbar et al., 2021), driving (Kerbel et al., 2022) and to learn a residual RL policy on top of a pre-trained RL agent (Ceola et al., 2024). A fixed mixing, however, does not allow one to consider the improving capabilities of the RL agent.
Hybrid Reinforcement Learning with Adaptive Weighting. Daoudi et al. (2023) assume a given controller confidence function, employing a controller in instances of high confidence and an RL agent in other scenarios. Our work focuses on the RL agent’s confidence and proposes to estimate the confidence based on a critic ensemble. Similar to our approach, Hoel et al. (2020a; b) train an ensemble of bootstrapped Q-networks for a driving task with discrete actions. They evaluate the uncertainty as the coefficient of variation of Q-estimates and resort to safe fallback actions in case of high uncertainty. However, they do not combine controller and RL agent but switch between both. In this work, we investigate a seamless blending approach for continuous control. Rana et al. (2020b) estimate the policy uncertainty using Monte-Carlo dropout and based on this uncertainty either sample from a residual policy or the controller alone. Rana et al. (2020a) directly fuse a prior control distribution with an RL policy in a multiplicative fashion and anneal the influence of the control prior over training time. Rana et al. (2023) use a policy ensemble to estimate how certain the RL agent is in the current action. The combined action is then computed as the Bayesian posterior of control prior and policy distribution. Cheng et al. (2019) use the TD-error as an uncertainty estimate and combine controller and RL agent based on this. Both Rana et al. (2023) and Cheng et al. (2019) base their adaption mechanism on a form of policy uncertainty. Both approaches train based on the combined action, which becomes brittle when facing large distributional shifts. We further discuss this limitation in Section 5.
3 Background
The following introduces the key components and the general concept of hybrid RL.
Reinforcement Learning. RL is a method for solving sequential decision problems based on the interaction between an agent and an environment (Sutton & Barto, 2018). The environment is modeled as a discounted Markov decision process defined by the tuple , with state space , action space , and start state distribution . The commonly unknown transition function describes transitions between states and actions . During transitions, rewards are emitted according to a reward function . The objective of the RL agent is to learn a policy that maximizes the expected cumulative sum of rewards discounted by . This results in the RL objective
(1) |
The discounted sum of rewards is referred to as return and is accumulated along trajectories under the policy and the environment MDP . State value functions condition expected return on a particular state while, action value or Q-functions condition expected return on specific state action pairs .
Control Prior. The prior policy represents prior knowledge for solving the RL objective (1), while typically not providing the optimal policy over the whole domain . This work focuses on control priors based on classic control theory. These can be derived with limited effort in many applications and often provide a good baseline for interaction with . We assume a control prior that is time-invariant and without an internal state.
Hybrid Reinforcement Learning. Hybrid RL combines the control prior and the RL agent by blending their actions via some mixing function depending on a weight .
4 A Unified View on Hybrid Reinforcement Learning
Next, we develop a unified view on hybrid RL that allows classifying prior methods (cf. Section 2). In the standard RL setup depicted in Figure 1(a), the RL agent interacts with the time-invariant MDP with dynamics that represents the controlled system. Hybrid RL (see Figure 1(b)) incorporates a control prior which requires reformulating the standard framework. Here, and apply a combined action to . The mixing function generates a combined action by blending the individual actions based on a weighting vector provided by a weight adaption function . Within this generalized framework, prior work in hybrid RL can be categorized based on the choice of and .
4.1 Mixing Function
We consider mixing functions based on a weighted sum with a weighting vector
(2) |
This formulation allows to distinguish a residual and a regularized setting.
In the residual setting, is typically constant while can be variable (Silver et al., 2018b; Johannink et al., 2019; Schoettler et al., 2020). Thus, the RL agent interacts with the closed control loop between and and learns a residual action on top of . Consequently, modulates the RL agent’s impact on the closed loop dynamics. As the control prior is not scaled down, it might interpret the RL agent as a disturbance and counteract it (Ranjbar et al., 2021), which can limit the overall performance of residual formulations.
In the regularized setting, both weights are adaptable such that (Cheng et al., 2019; Rana et al., 2023). This results in a mixing function of the form
(3) |
with . In the limit , the control prior interacts with without the interference of the RL agent, while the regularized setting reduces to the standard RL problem for . Thus, indicates not only the impact of but also whether the RL agent interacts with the open loop dynamics of or the closed loop dynamics as in the residual setting. Consequently, the control prior can be interpreted as a regularization of the RL agent. Our proposed algorithm operates in the regularized setting, allowing the agent to take over complete control when .
4.2 Weight Adaption Function
Hybrid RL approaches can further be classified, based on the choice of the weight adaption function modulating the weighting vector of the mixing function .
A large body of work, which we refer to as fixed-weight hybrid RL (Silver et al., 2018b; Johannink et al., 2019; Schoettler et al., 2020; Ranjbar et al., 2021; Ceola et al., 2024) chooses fixed throughout training. Neglecting the time- and state-dependent capabilities of the RL agent.
Approaches that adapt , which we refer to as adaptive hybrid RL methods, rely on different mechanisms. Scheduling approaches (Rana et al., 2020a) change the weight explicitly with time, i.e. , typically increasing the weight of the RL agent as training progresses. Domain-based approaches (Kulkarni et al., 2022; Daoudi et al., 2023) adapt the weights based on the point of operation within the domain , i.e. . Uncertainty-based approaches (Cheng et al., 2019; Rana et al., 2023) adapt the weight based on the confidence of the RL agent, indicated by an uncertainty estimate , giving more weight to the RL agent when it has high confidence. Thus, they aim to leverage the benefits of the RL agent whenever possible, while resorting to a safe controller in situations where the RL agent has not seen enough data. The time dependency of the uncertainty estimate, however, increases the complexity of the hybrid RL setting, requiring a reformulation of the learning problem. In Section 5, we discuss the shortcomings of prior formulations and propose our own.
![Refer to caption](x1.png)
![Refer to caption](x2.png)
![Refer to caption](x3.png)
5 Contextualized Hybrid Reinforcement Learning
In Section 5.1, we propose a novel contextualized formulation of the adaptive hybrid RL problem and illustrate its benefits over prior approaches in Section 5.2. Based on that framework, we propose the Contextualized Hybrid Ensemble Q-learning (CHEQ) algorithm in Section 5.3.
5.1 General Concept of Contextualized Hybrid Reinforcement Learning
Based on the unified view provided in Section 4, we propose a general formulation for the hybrid RL problem we call contextualized hybrid RL.
The environment in the hybrid setting not only consists of the controlled system but also comprises the control prior, the mixing function, and the weight adaption function. We consider both the control prior and the mixing function to be time-invariant. In contrast, the weight adaption function can have time-varying behavior, i.e. , as discussed in Section 4.2. This leads to time-varying dynamics of the hybrid environment, violating the assumption of time-invariance in the MDP formulation (Bellemare et al., 2023). Instead, we exclude from the definition of the hybrid environment and introduce the adaptive weight vector as a context variable to the agent and the environment (see Figure 1(c)). We model the hybrid environment introducing the contextualized hybrid MDP with the set of weighting vectors and the contextualized dynamics function .
The MDP formulation induces the contextualized hybrid RL objective
(4) |
which enforces to learn a policy that maximizes expected return in . Introducing as a context variable further yields the contextualized hybrid value functions and . Thus, we can optimize (4) using standard RL methods by additionally conditioning on . The general mechanism of contextualized hybrid RL is illustrated in Algorithm 1.
Prior work takes different approaches to formulating the hybrid learning problem. Approaches with a time-invariant weight adaption function, such as fixed-weight hybrid methods, include in the definition of the environment MDP with dynamics . This formulation is directly applicable to the standard RL objective (1), however, does not generalize to time-varying adaption mechanisms such as uncertainty-adapted methods. Approaches with a time-varying adaption mechanism (Cheng et al., 2019; Rana et al., 2023) typically formulate the hybrid RL problem concerning the controlled system MDP and the combined action with dynamics . This likewise yields a formulation that is directly applicable to (1), however, this can lead to problems as the agent is unaware of the downstream mixing process. Furthermore, this introduces a distributional shift between trained policy and data-collecting behavior policy. The distributional shift can lead to training instability and divergence (Kumar et al., 2020; Fujimoto et al., 2018).
5.2 Illustrative Example
We exemplify the strength of the contextualized hybrid RL formulation based on introduced in Section 5.1 by comparing it to prior approaches on the cart pole system depicted in Figure 2(a). The goal is to balance the pole upright while kee** the cart close to its initial position. The system is controlled via continuous forces on the cart. We choose to apply a constant force to the left, which destabilizes formulations unconscious of the mixing process. We investigate a time-invariant fixed weight setting as well as a time-varying schedule-based weight adaption setting to highlight the capability of the respective formulations to deal with both scenarios. This simple example illustrates that the contextualized hybrid MDP formulation can deal with destabilizing control priors and time-varying weight adaption functions while prior formulations fail.
![Refer to caption](x4.png)
![Refer to caption](x5.png)
![Refer to caption](x6.png)
Fixed Weighting. First, we consider a residual setting with fixed weights . Figure 2(b) depicts the performance of RL agents trained under , , and with respective dynamics , , and . While agents trained under and learn to stabilize the cart pole, the hybrid formulation concerning fails. When formulating the RL problem concerning the RL agent observes the combined action in its data and therefore learns the combined action in its policy. This, however, neglects the fact that the policy action is mixed with the controller action before being applied to . Assuming the cart pole is not moving, and the pole is upright, an agent trained under provides the optimal combined action, namely applying no force, while pushes the pole to the left. This results in pointing to the left, causing the pole to fall while giving the agent no mechanism to observe and counteract this phenomenon. Instead, formulating the hybrid RL problem concerning allows the agent to observe the mixing mechanism and compensate for the destabilizing control prior.
Adaptive Weighting. Second, we consider an adaptive hybrid RL problem with time-varying . We choose a schedule-based approach with a regularizing mixing function (3) and linearly increasing over time. Figure 2(c) shows the performance of agents trained under formulations based on , , and . While agents trained under and fail, the formulation based on succeeds. In the beginning, when the RL agent is given only low weight, the formulation under suffers from a high distributional shift between the action of the behavior policy and the action of the target policy . The large distributional shift causes the agent to diverge (Kumar et al., 2020; Fujimoto et al., 2018). Although the distributional shift decreases with increasing weight lambda, the agent does not manage to recover. The formulation under fails due to the missing information about the time-variant behavior of the mixing process. The proposed contextualized hybrid RL formulation solves these issues by formulating the task concerning and introducing the context variable .
5.3 Contextualized Hybrid Ensemble Q-learning (CHEQ)
Based on the contextualized hybrid RL formulation introduced in Section 5.1, we propose the Contextualized Hybrid Ensemble Q-learning (CHEQ) algorithm. At the heart of CHEQ is a critic ensemble that (i) provides an uncertainty estimate enabling an uncertainty-adapted hybrid RL mechanism, and (ii) allows to incorporate ensemble-based acceleration techniques for data-efficient RL.
We base CHEQ on the Soft Actor-Critic (SAC) (Haarnoja et al., 2018) algorithm and a regularizing mixing function (3) with . The weight adaption mechanism relies on a critic ensemble comprising of contextualized Q-functions with parameters , and corresponding target Q-functions with parameters , . We update the critics with the mechanism of Randomized Ensemble Double Q-learning (REDQ) (Chen et al., 2021) and enforce sufficient independence between Q-estimates using Bernoulli masking of the training data (Osband et al., 2016; Lee et al., 2021; Mai et al., 2022). Model ensembles estimate parametric uncertainty, referred to as epistemic uncertainty, from disagreement between individual models within the ensemble. If different critics disagree about the outcome of taking action in while weighting with , this indicates a weak understanding of the task in the particular area of . Thus, the control prior should be prioritized over the RL agent in such situations. Therefore, epistemic uncertainty represents a suitable quantity for adapting the weighting of control prior and RL agent. We define epistemic uncertainty as the standard deviation of critic predictions
(5) |
with . We aim to give low weight to the RL agent in areas of high uncertainty and vice versa. Thus, the weight adaption function maps the critics epistemic uncertainty to the weighting factor via the piece-wise linear function
(6) |
Besides providing an uncertainty estimate of the RL agent, the critic ensemble used in CHEQ has proven effective in mitigating overestimation bias (Thrun & Schwartz, 1993) in Q-learning-based approaches (Lan et al., 2021; Wang et al., 2021; Chen et al., 2021). The Update-To-Data (UTD) ratio describes the number of gradient steps per environment interaction. Due to the reduction of the overestimation bias, the critic ensemble allows for increasing the UTD ratio while maintaining stable learning. This substantially improves the data efficiency of value-based actor-critic methods (Chen et al., 2021). A detailed pseudocode algorithm of CHEQ is provided in Algorithm 2 of Appendix A.
6 Experiments
We evaluate CHEQ on a racing task and compare it to standard RL, fixed-weighting hybrid RL, and state-of-the-art adaptive hybrid RL. In our experiments, CHEQ yields substantial improvements in (i) data efficiency compared to other hybrid methods, as well as (ii) exploration safety, and (iii) zero-shot transferability to unknown scenarios as compared to all competitor approaches.
6.1 Experimental Setup
We base our evaluation on a car racing setting adapted from (Schier et al., 2023). Achieving high returns requires advanced trajectory planning and control while operating the vehicle close to stability limits, including tire slip. The control prior is a trajectory-following task along the center line of the track using a Stanley controller (Thrun et al., 2006) for lateral and a proportional controller for longitudinal control. Further details are provided in Appendix B.
We compare CHEQ against the standard RL approaches SAC (Haarnoja et al., 2018) and REDQ (Chen et al., 2021), fixed-weighting hybrid RL based on SAC, and the state-of-the-art adaptive hybrid RL methods Controller Regularized RL (CORE) (Cheng et al., 2019) and Bayesian Controller Fusion (BCF) (Rana et al., 2023). In all experiments, we provide results for CHEQ with a high UTD ratio (CHEQ-UTD20) to demonstrate the capabilities of the approach and a low UTD ratio (CHEQ-UTD1) for a fair comparison to SAC-based methods. All implementations111The code is available at github.com/Data-Science-in-Mechanical-Engineering/cheq . are based on either the Clean RL library (Huang et al., 2022) or the original paper implementation (Rana et al., 2023). We provide a detailed description of the hyperparameter settings in Appendix A.1.
We train all our approaches on ten random seeds and one fixed race track. We report return and cumulative training failures. Runs are considered a failure when the car leaves the track. For zero-shot transfer, we evaluate ten trained models per algorithm on ten unseen racetracks. Return and failure plots show the respective mean (solid lines) and \qty95 confidence interval (shaded areas).
6.2 Evaluation on the Car Racing Environment
We compare CHEQ to standard RL, fixed-weight hybrid RL, and adaptive hybrid RL concerning learning performance (see Figure 3) and zero-shot transfer to unknown tracks (see Figure 4).
![Refer to caption](x7.png)
![Refer to caption](x8.png)
![Refer to caption](x9.png)
Comparison against Fixed-Weight Hybrid RL and Standard RL. Comparing the CHEQ algorithm based on SAC (CHEQ-UTD1) to a standalone SAC agent and the control prior in Figure 3(a) illustrates the general benefit of hybrid RL. While the control prior operates safely without failing, it shows limited performance due to the conservative driving policy. SAC shows strong asymptotic performance at the cost of frequent failures throughout training. CHEQ-UTD1 considerably outperforms SAC concerning data efficiency and exploration safety, learning faster with fewer failures while yielding comparable asymptotic performance. The comparison of CHEQ to fixed-weight hybrid RL methods further illustrates the advantage of an adaptive weighting scheme. The fixed-weight hybrid RL approaches (0.5-SAC, 0.7-SAC) combine the control prior with a SAC agent using the mixing function in (3) with and , respectively. Here, the choice of represents a trade-off between exploration safety and asymptotic performance, where a higher enables better performance while reducing safety. A fixed weight of arguably reduces failures compared to CHEQ-UTD1, however, this comes at the cost of substantially lower performance.
Update-To-Data Ratio. As discussed in Section 5.3, the critic ensemble of CHEQ allows the use of acceleration techniques originally proposed in the REDQ algorithm. Increasing the UTD ratio to notably improves data efficiency as compared to SAC, both as a standalone RL algorithm (REDQ) and as an adaptive hybrid RL algorithm (CHEQ-UTD20). The speed-up in training helps to reduce training failures as REDQ reports a drastically reduced number of failures compared to SAC. The benefit is further amplified in the adaptive hybrid formulation of CHEQ-UTD20 as indicated by its strong initial performance and the ability to reduce the mean cumulative fails to less than .
Comparison against state-of-the-art Adaptive Hybrid RL. Finally, Figure 3(c) compares CHEQ to the most relevant adaptive hybrid RL methods. CHEQ-UTD1 shows similar data efficiency and performance compared to CORE and BCF while considerably reducing accumulated fails. CHEQ-UTD20 substantially outperforms all competitor approaches in all performance metrics. A more detailed hyperparameter analysis of the prior approaches, as well as results for reformulations of CORE and BCF as contextualized hybrid RL methods are provided in in Appendix C.
![Refer to caption](x10.png)
![Refer to caption](x11.png)
![Refer to caption](x12.png)
Zero-shot Transfer. Next, we perform a zero-shot transfer of the trained agents. Returns are depicted in Figure 4(a) while Figure 4(b) shows the success rate of the respective methods. CHEQ-UTD1, CHEQ-UTD20 and the control prior achieve a success rate of \qty97, \qty95, and \qty100, respectively. The other standard and hybrid RL methods frequently fail in unseen scenarios. While the CHEQ variants fail slightly more often than the controller, they drive notably faster, i.e., they achieve higher returns. Figure 4(c) illustrates the adaption mechanism of CHEQ on one example track. In challenging and unseen curves, the agent gradually hands over to the control prior as can be seen in Figure 4(c). We find that in the few failure cases (3 out of 100 for CHEQ-UTD1 and 5 out of 100 for CHEQ-UT20), the agent correctly identifies its uncertainty, and hands over to the controller, but the controller is unable to navigate the situation safely. We provide an illustration of all transfer tracks, as well as the weight adaption of CHEQ-UTD20 on these tracks in Appendix C.2. In summary, CHEQ shows strong zero-shot transfer behavior, driving faster than the controller with only a few failure cases.
![Refer to caption](x13.png)
![Refer to caption](x14.png)
![Refer to caption](x15.png)
Summary. We summarize the trade-off between failures and asymptotic performance in Figure 5. Figure 5(a) illustrates the training results of the respective approaches while Figure 5(b) depicts the transfer results. Fixed weight hybrid RL effectively reduces failures as compared to standard SAC. This, however, comes at the cost of asymptotic performance. Our adaptive CHEQ algorithm avoids this trade-off, achieving high return with only a few failures. In zero-shot transfer, the CHEQ agent again performs best due to its ability to detect unforeseen situations reliably and then fall back to the safe control prior.
7 Conclusion
This work addresses how to systematically combine an RL agent with a control prior. We propose a novel formulation of the adaptive hybrid RL problem which introduces the adaptive weighting parameter as a context variable of the MDP, and based on this, propose the Contextualized Hybrid Ensemble Q-learning (CHEQ) algorithm. CHEQ combines a reliable critic uncertainty-based weight adaption mechanism with the data efficiency of critic ensemble methods, yielding substantially stronger results than state-of-the-art adaptive hybrid RL methods on a racing task concerning data efficiency, exploration safety, and transferability.
Acknowledgments
We thank Paul Brunzema, Johanna Menn, and David Stenger for their helpful comments. We also thank Devdutt Subhasish and Lukas Kesper for their help with the cartpole example. This work was funded in part by the German Federal Ministry of Education and Research (“Demonstrations- und Transfernetzwerk KI in der Produktion (ProKI-Netz)” initiative, grant number 02P22A010) and the German Federal Ministry for Economic Affairs and Climate Action (project EEMotion). Computations were performed with computing resources granted by RWTH Aachen University under the projects <thes1594>, <rwth1490>, and <rwth1501>.
References
- Bellemare et al. (2023) Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2023. http://www.distributional-rl.org.
- Büchler et al. (2022) Dieter Büchler, Simon Guist, Roberto Calandra, Vincent Berenz, Bernhard Schölkopf, and Jan Peters. Learning to Play Table Tennis From Scratch Using Muscular Robots. IEEE Transactions on Robotics, 2022.
- Ceola et al. (2024) Federico Ceola, Lorenzo Rosasco, and Lorenzo Natale. RESPRECT: Speeding-up Multi-fingered Gras** with Residual Reinforcement Learning, 2024. arXiv:2401.14858 [cs].
- Chen et al. (2021) Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model, 2021.
- Cheng et al. (2019) Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control Regularization for Reduced Variance Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019.
- Daoudi et al. (2023) Paul Daoudi, Bogdan Robu, Christophe Prieur, Ludovic Dos Santos, and Merwan Barlier. Enhancing Reinforcement Learning Agents with Local Guides. In International Conference on Autonomous Agents and Multiagent Systems, 2023.
- Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018.
- Gupta et al. (2021) Abhishek Gupta, Justin Yu, Tony Z. Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
- Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. PMLR, 2018.
- Hoel et al. (2020a) Carl-Johan Hoel, Tommy Tram, and Jonas Sjöberg. Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersections. In IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020a.
- Hoel et al. (2020b) Carl-Johan Hoel, Krister Wolff, and Leo Laine. Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation. In IEEE Intelligent Vehicles Symposium (IV), 2020b.
- Huang et al. (2022) Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G. M. Araújo. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal of Machine Learning Research, 2022.
- Johannink et al. (2019) Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In International Conference on Robotics and Automation (ICRA), 2019.
- Kerbel et al. (2022) Lindsey Kerbel, Beshah Ayalew, Andrej Ivanco, and Keith Loiselle. Residual Policy Learning for Powertrain Control. IFAC-PapersOnLine, 2022.
- Kulkarni et al. (2022) Padmaja Kulkarni, Jens Kober, Robert Babuška, and Cosimo Della Santina. Learning assembly tasks in a few minutes by combining impedance control and residual recurrent reinforcement learning. Advanced Intelligent Systems, 2022.
- Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
- Lan et al. (2021) Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Controlling the Estimation Bias of Q-learning, 2021.
- Lee et al. (2021) Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning. PMLR, 2021.
- Mai et al. (2022) Vincent Mai, Kaustubh Mani, and Liam Paull. Sample efficient deep reinforcement learning via uncertainty estimation. arXiv preprint arXiv:2201.01666, 2022.
- Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, and Georg Ostrovski. Human-level control through deep reinforcement learning. Nature, 518(7540), 2015.
- OpenAI et al. (2019) OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforcement Learning, 2019.
- Osband et al. (2016) Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep Exploration via Bootstrapped DQN. In Advances in Neural Information Processing Systems, volume 29, 2016.
- Rana et al. (2020a) Krishan Rana, Vibhavari Dasagi, Ben Talbot, Michael Milford, and Niko Sünderhauf. Multiplicative Controller Fusion: Leveraging Algorithmic Priors for Sample-efficient Reinforcement Learning and Safe Sim-To-Real Transfer. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020a.
- Rana et al. (2020b) Krishan Rana, Ben Talbot, Vibhavari Dasagi, Michael Milford, and Niko Sünderhauf. Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments. In 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020b.
- Rana et al. (2023) Krishan Rana, Vibhavari Dasagi, Jesse Haviland, Ben Talbot, Michael Milford, and Niko Sünderhauf. Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics. The International Journal of Robotics Research, 2023.
- Ranjbar et al. (2021) Alireza Ranjbar, Ngo Anh Vien, Hanna Ziesche, Joschka Boedecker, and Gerhard Neumann. Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
- Schier et al. (2023) Maximilian Schier, Christoph Reinders, and Bodo Rosenhahn. Learned Fourier Bases for Deep Set Feature Extractors in Automotive Reinforcement Learning. In IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), Bilbao, Spain, 2023.
- Schoettler et al. (2020) Gerrit Schoettler, Ashvin Nair, Jianlan Luo, Shikhar Bahl, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
- Schulman et al. (2017) J. Schulman, F. Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. ArXiv, 2017.
- Silver et al. (2018a) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 2018a.
- Silver et al. (2018b) Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. arXiv preprint arXiv:1812.06298, 2018b.
- Sutton & Barto (2018) Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
- Thrun & Schwartz (1993) Sebastian Thrun and Anton Schwartz. Issues in Using Function Approximation for Reinforcement Learning. In Proceedings of the Fourth Connectionist Models Summer School, 1993.
- Thrun et al. (2006) Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James Diebel, Philip Fong, John Gale, Morgan Halpenny, Gabriel Hoffmann, Kenny Lau, Celia Oakley, Mark Palatucci, Vaughan Pratt, Pascal Stang, Sven Strohband, Cedric Dupont, Lars-Erik Jendrossek, Christian Koelen, Charles Markey, Carlo Rummel, Joe van Niekerk, Eric Jensen, Philippe Alessandrini, Gary Bradski, Bob Davies, Scott Ettinger, Adrian Kaehler, Ara Nefian, and Pamela Mahoney. Stanley: The robot that won the DARPA Grand Challenge. Journal of Field Robotics, 2006.
- Wang et al. (2021) Hang Wang, Sen Lin, and Junshan Zhang. Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback. In Advances in Neural Information Processing Systems, volume 34, 2021.
Appendix A Algorithmic Details
Algorithm 2 shows the pseudocode of the Contextualized Hybrid Ensemble Q-learning algorithm.
A.1 Hyperparameters Settings
We build our SAC implementation based on CleanRL (Huang et al., 2022). All SAC-specific hyperparameters are kept consistent between all approaches and reported in Table 1.
Hyperparameter | Value |
---|---|
number of steps | |
batch size | 256 |
learning rate actor | |
learning rate critic | |
target entropy | |
replay buffer size | |
discount factor | |
gradient update start | steps |
Polyak averaging factor |
CHEQ (UTD1 and UTD20) uses an ensemble of critics. We set the upper bound of the uncertainty as and the lower bound as . Further we set and . We use a Bernoulli masking rate of and minimization targets.
BCF trains an ensemble of policy networks. We maintain the original ensemble size from (Rana et al., 2023) which uses ten policy networks. We set the standard deviation of the control prior in BCF as .
For the uncertainty estimate in CORE, we set , . Note that in the original paper, is denoted as , which we change to avoid ambiguous notation.
SAC uses a UTD ratio of . REDQ implementation uses an ensemble size of and a UTD ratio of .
For all algorithms, we include a random sampling phase for the first steps where we sample the RL action uniformly random and do not update our agent. In this setting, we keep small for the hybrid agents. For CHEQ we vary between [0.2, 0.3]. As CORE and BCF are unable to observe changes in we use a fixed which has shown to be favorable in our experiments. After the random sampling phase, agent training starts, but is kept small for another steps and afterward, adaption starts.
For performance evaluation, we conduct a greedy evaluation run every episodes. Evaluation happens in an adapted setting, together with the controller where the weight is calculated as in the training procedure.
Appendix B Environment Details
B.1 Racing Environment
We test our agent on the simulated racing task adapted from Schier et al. (2023). Figure 6 shows an example of the environment.
![Refer to caption](x16.jpg)
![Refer to caption](x17.png)
The vehicle uses a dynamic single-track model with a coupled Dugoff tire model. The throttle, brake, and steering are continuous actions. The vehicle is a front-wheel drive. The RL agent may learn to control brake balance by applying throttle and brake individually. We define the state of the RL agent as , with the ego vehicle’s velocity vector in vehicle reference frame, steering angle , and yaw rate . The observation of the track is given as a vector of Cartesian distances to the centerline of the track. The points are sampled equidistantly from the \qty60 track segment ahead.
We use the original reward formulation from Schier et al. (2023) where the RL agent receives a penalty whenever it collides with the track boundary and a penalty for leaving the track with the center of mass. The latter case also terminates the episode. The RL agent receives a positive reward for driving fast: the scalar projection of its velocity vector onto the forward track direction . The complete reward is then given by
(7) |
B.2 Control Prior
For the racing task, we design a simple path-following controller with adaptive speeds. For the lateral control, we use a Stanley Controller (Thrun et al., 2006) following the steering control law
where denotes the heading error, denotes the crosstrack error of the front axle and describes the velocity of the vehicle.
For the longitudinal control, we design two symmetric P-controllers; one for the brake and one for the throttle. First, we compute the target velocity dependent on the curve radius of the track directly in front of the vehicle as
where is the maximum desired velocity. Then, we design the throttle control as
and the brake control as
with shared gain . Following this control law, the control prior accelerates if it is going too slow and brakes if it is going too fast. It never uses the brake and the throttle at the same time.
To avoid aggressive braking behavior when the RL agent hands over to the controller in risky situations (high velocity around curves), we additionally introduce a simple clipped linear gain schedule on attenuating the braking control for higher velocities as
We tuned the controller gains and coefficients to , , , , , , and .
Appendix C Additional Results and Ablation Study
In this section, we formulate contextualized hybrid variants of CORE and BCF, C-CORE and C-BCF. Further, we present additional results on hyperparameter sensitivity and the distribution of for CHEQ-UTD1, CORE, and BCF. For a reasonable comparison, we focus mainly on CHEQ-UTD1 using a UTD ratio of .
C.1 Contextualized Hybrid Variants of Prior Work
To further substantiate our claim that the contextualized hybrid RL formulation aids the training progress, we developed contextualized variants of the CORE and BCF algorithm, which we call C-CORE and C-BCF. To use the adaptive weight as a context variable, a weighting parameter needs to be determined.
The CORE algorithm comes with a direct weight estimate(Cheng et al., 2019), which can be written as
where with the TD-error and , being tuning parameters222CORE (Cheng et al., 2019) uses the term instead of . As we use in a different context, we stick to here..
For C-BCF, we derive a pseudo-weight, as the BCF algorithm (Rana et al., 2023) does not have a straightforward weighting factor . In BCF, at timestep , the fusion of the prior and the SAC-ensemble results in a Gaussian distribution with mean
Thus, in BCF the weight has the dimension of the action space, whereas the contextualized mechanism requires a scalar weight . For C-BCF, we compute a scalar weight
The next action is computed according to Equation 3 where and .
For C-BCF and C-CORE we use the same warm-up phase as for our CHEQ agent.
C.2 Additional Results and Hyperparameter Tuning for CHEQ
![Refer to caption](x18.png)
![Refer to caption](x19.png)
![Refer to caption](x20.png)
![Refer to caption](x21.png)
Our algorithm has only two important hyperparameters, upper and lower bounds of the uncertainty and . We chose these hyperparameters by conducting one training run and investigating the uncertainty range within this run. CHEQ is generally robust against changes in these thresholds. We observe slightly lower final return and fewer fails for lower upper bounds . This is to be expected as frequent handover to the control prior results in lower velocities and thus lower return. Figure 7 shows the return and the number of fails during training for our CHEQ-UTD1 variants. For CHEQ-UTD20 we were able to use the same upper and lower uncertainty bounds as for CHEQ-UTD1.
Figure 8 shows the distribution of the weight over the training progress. We find that for CHEQ-UTD1 the agent starts with an almost uniform distribution of the weight and slowly moves towards a regime. Even in later training stages, the agent hands over to the controller from time to time.
Lastly, we investigated the transfer behavior of the CHEQ-UTD20 agent further. Figure 9 shows the ten transfer tracks. We plot over the track for one evaluated model. Here, we find that the agent frequently becomes uncertain and hands over to the controller, especially in unknown curves. In plot C we see one of the two failure cases (out of 100 runs) that we experience during transfer. We find that the agent becomes uncertain and hands over to the controller. In this specific scenario, however, the controller is not able to safely navigate the situation and leaves the track.
C.3 Additional Results and Hyperparameter Tuning for CORE and C-CORE
![Refer to caption](x22.png)
![Refer to caption](x23.png)
In Figure 10 we compare different parameters . We find stable training progress and high return for but since this uses high values, this setting results in many failures. Figure 8 illustrates the high regime of the agent. C-CORE using our contextualized hybrid framework, notably outperforms CORE in terms of asymptotic return, training stability, and the number of training failures. Using the contextualized formulation, C-CORE can use a much wider distribution (see Figure 8).
C.4 Additional Results and Hyperparameter Tuning for BCF and C-BCF
The BCF algorithm is sensitive to the parameters of the uncertainty threshold, which in this case is the variance of the control prior . Higher variances, lead to less weight on the control prior and thus high regimes. In Figure 11 we compared different parameters . We find stable training progress and high return for . However, this setting, as expected, uses a regime close to one and thus results in a high number of failures. Figure 8 illustrates this regime.
Our C-BCF variant can resolve this problem only partially. Due to its construction, the BCF algorithm has a separate weighting factor for each action of which we take the mean. In addition, our pseudo factor is only a rough estimate of the actual mixing as BCF samples from the posterior distribution. Both factors result in information loss and make the weight only a rough estimate. We find that C-BCF-2.0 and BCF-6.0 achieve similar asymptotic performance, while C-BCF-2.0 leads to fewer failures.
![Refer to caption](x24.png)
![Refer to caption](x25.png)
C.5 Return vs. Failure Comparison for all trained Models
Figure 12 shows a scatter plot of the final return and the accumulated failures during training for all hybrid algorithms discussed in this paper. This final comparison shows that if prior methods are trained with the contextualized framework and tuned well ( C-BCF-0.8, C-CORE-0.4, C-CORE-0.8), they achieve high returns while maintaining fewer failures than their non-contextualized counterparts. Our algorithm (CHEQ-UTD1, CHEQ-UTD20) achieve the highest return while maintaining the lowest number of cumulative failures.
![Refer to caption](x26.png)