The Curse of Diversity in Ensemble-Based
Exploration

Ensemble-based exploration, i.e. training a diverse ensemble of data-sharing agents, underlies many successful deep reinforcement learning (deep RL) methods (Osband et al., 2016; 2018; Liu et al., 2020; Schmitt et al., 2020; Peng et al., 2020; Hong et al., 2020; Januszewski et al., 2021). The potential benefits of a diverse ensemble are twofold. At training time, it enables concurrent exploration with multiple distinct policies without the need for additional samples. At test time, the learned policies can be aggregated into a robust ensemble policy, via aggregation methods such as majority voting (Osband et al., 2016) or averaging (Januszewski et al., 2021).

Despite the generally positive perception of ensemble-based exploration, we argue that this approach has a potentially negative aspect that has been long overlooked. As shown in Figure 1, for each member in a data-sharing ensemble, only a small proportion of its training data comes from its own interaction with the environment. The majority of its training data is generated by other members of the ensemble, whose policies might be distinct from its own policy. This type of off-policy learning has been shown to be highly challenging in previous work (Ostrovski et al., 2021). We thus hypothesize that similar learning difficulties can also occur in ensemble-based exploration.

We show that several intuitive solutions – such as a larger replay buffer or a smaller ensemble size – either fail to consistently mitigate the performance loss or undermine the advantages of ensembling. Inspired by previous work’s finding that network representations play a crucial role in related settings (Ostrovski et al., 2021; Kumar et al., 2021), we investigate whether representation learning can mitigate the curse of diversity. Specifically, we propose a novel method named Cross-Ensemble Representation Learning (CERL) in which individual ensemble members learn each other’s value function as an auxiliary task. Our results show that CERL mitigates the curse of diversity in both Atari and MuJoCo environments and outperforms the single-agent and ensemble-based baselines when combined with policy aggregation.

We summarize our contributions as follows:

1. 1.

   We expose the curse of diversity phenomenon in ensemble-based exploration: individual members in a data-sharing ensemble can vastly underperform their single-agent counterparts.

2. 2.

   We pinpoint the off-policy learning challenges posed by a diverse ensemble as the main cause of the curse of diversity and provide extensive analysis.

3. 3.

   We show the potential of representation learning to mitigate the curse of diversity with a novel method named Cross-Ensemble Representation Learning (CERL) in both discrete and continuous control domains.

We outline some important specifications that we use throughout this work.

Ensemble-based exploration strategy We focus our discussion on a simple ensemble-based exploration strategy that underlies many previous works (Osband et al., 2016; 2018; Liu et al., 2020; Schmitt et al., 2020; Peng et al., 2020; Hong et al., 2020; Januszewski et al., 2021), depicted in Figure 1. The defining characteristics of this strategy are as follows:

1. 1.

   Temporally coherent exploration: At training time, within each episode, only the policy of one ensemble member is used for selecting the actions.

2. 2.

   Relative independence between ensemble members: Each ensemble member has its own policy (may be implicit), value function, and target value function. Most importantly, the regression target in Temporal Difference (TD) updates should be computed separately for each ensemble member with their own target network.

3. 3.

   Off-policy RL algorithms with a shared replay buffer: Different ensemble members share their collected data via a central replay buffer (Lin, 1992). To allow data sharing, the underlying RL algorithm should be off-policy in nature.

Environments and algorithms We use $55$ Atari games from the Arcade Learning Environment and $4$ Gym MuJoCo tasks for our analysis. We train for $200$M frames for Atari games and $1$M steps for MuJoCo tasks. For aggregate results over $55$ Atari games, we use the interquartile mean (IQM) of human-normalized scores (HNS) as recommended by Agarwal et al. (2021). In some results where HNS is less appropriate, we exploit another score normalization scheme where Double DQN and a random agent have a normalized score of $1$ and $0$ respectively. This will be referred to as Double DQN normalized scores. More experimental details can be found in Appendix B.

For our analysis, we use Bootstrapped DQN (Osband et al., 2016) with Double DQN updates (Hasselt et al., 2015) for Atari and an ensemble version of the SAC (Haarnoja et al., 2018b) algorithm (referred to as Ensemble SAC in the following) for MuJoCo tasks. Ensemble SAC follows the same recipe as Bootstrapped DQN except with SAC as the base algorithm. We provide pseudocode for these algorithms in Appendix A. Correspondingly, for the single-agent baselines, we use Double DQN and SAC. For analysis purposes, for continuous control, we use a replay buffer of size $200$k by default (as opposed to the usual $1$M) since we find the curse of diversity is more evident with smaller replay buffers in these tasks. Also, to avoid the confounding factor of representation learning, we do not share the networks across the ensemble in Bootstrapped DQN by default. These factors will be analyzed in Section 3.3. Following Osband et al. (2016)’s setup for Atari, we do not use data bootstrap** in our analysis. Throughout this work, we use $L$ to denote the number of shared layers across the ensemble members and $N$ to denote the ensemble size. Unless otherwise specified, we use $L=0$ and $N=10$. For each ensemble algorithm X – where X can be either Bootstrapped DQN or Ensemble SAC – we consider two different evaluation methods:

1. 1.

   X (aggregated) or X (agg.): we aggregate the policies of all ensemble members during testing. For discrete action tasks, we use majority voting (Osband et al., 2016). For continuous control, we average the actions of all policies as in  Januszewski et al. (2021).

2. 2.

   X (individual) or X (indiv.): for each evaluation episode, we randomly sample one ensemble member for acting. This interaction protocol is exactly the same as the one used during training and aims to measure the performance of the individual ensemble members.

We emphasize that these two methods only differ at test-time. We only train X once and then obtain the results of X (agg.) and X (indiv.) using the above interaction protocols during evaluation. Policy aggregation is never used during training. Also, the performance of X (agg.) is often what we eventually care about. More implementation details can be found in Appendix C.

The central finding of this paper is as follows:

We name the performance degradation in individual ensemble members due to challenging off-policy learning posed by a diverse ensemble the curse of diversity. In the following, we demonstrate this phenomenon, verify its cause, and provide extensive analysis.

Our method is motivated by a simple hypothesis. We conjecture that the reason why sharing network layers mitigates the curse of diversity is twofold. First, as mentioned above, sharing layers reduces ensemble diversity, thus making the generated data “less off-policy” to the individual members. Second, the shared network simultaneously learns the value functions of multiple ensemble members, leading to improved representations. This may lead to more efficient off-policy learning by allowing the Q-value networks to better generalize to state-action pairs that have high probability under the current policy but are under-represented in the data. As we have shown, the diversity reduction aspect has the undesirable side effect of reducing the benefits of policy aggregation. We thus propose Cross-Ensemble Representation Learning (CERL), a novel method that benefits from the similar representation learning effect of network sharing without actually needing to share the networks, thus preserving the diversity of the ensemble.

Method CERL, shown in Figure 6 (right), is an auxiliary task that can be applied to most ensemble-based exploration methods that follow the recipe we outline in Section 2. For ease of exposition, we use Ensemble SAC as an example. Extension to other methods is trivial. In CERL, for each Q-value network, we conceptually split it into an encoder $f$ and a head $g$. Our goal in CERL is to force the Q-value network encoder of each ensemble member to learn the value functions of all the ensemble members, similar to what happens when using explicit network sharing. To this end, each ensemble member $i$ has $N$ Q-value heads $\{Q_{i}^{j}(s,a)=[g_{i}^{j}(f_{i}(s,a))]\}_{j=1}^{N}$. For ensemble member $i$, $Q^{i}_{i}(s,a)$ is the “main head” that defines member $i$’s value function estimation. Each ensemble member $i$ still has only one policy $\pi_{i}(a|s)$, and the main head $Q_{i}^{i}(s,a)$ is the only head used to update the policy. A head $Q_{i}^{j}(s,a)$ of ensemble member $i$ where $j\neq i$ is used to learn the value function of ensemble member $j$ as an auxiliary task. These heads are referred to as “auxiliary heads” because their sole purpose is to provide better representations for the main heads. Specifically, given a transition $(s,a,r,s^{\prime})$, we perform the following TD update for all $N\times N$ heads in parallel as follows:

where $\bar{Q}_{j}^{j}$ is the target network for $Q_{j}^{j}$ and $a_{j}^{\prime}\sim\pi_{j}(\cdot|s^{\prime})$. As usual, the update rule is implemented as a one-step stochastic gradient descent.

Besides being conceptually simple, CERL is easy to implement. In our experiments, we find it sufficient to duplicate the last linear layers of the networks as the auxiliary heads, which can be implemented by increasing the networks’ final output dimensions. For the same reason, CERL is computationally efficient. For example, for the Nature DQN network (Mnih et al., 2015) used in this work, applying CERL to Bootstrapped DQN with $N=10$ and $L=0$ increases the number of parameters by no more than $5\%$, and the increase in wall clock time is barely noticeable. We provide pseudocode for CERL with Ensemble SAC and Bootstrapped DQN in Appendix A.

Experiments We focus on two questions: (1) Can CERL mitigate the curse of diversity, i.e. the performance gap between individual ensemble members and their single-agent counterparts? (2) Do the improvements in individual ensemble members translate into a better aggregate policy? To answer these questions, we test CERL on Bootstrapped DQN in $55$ Atari games (Figure 7 (top)) and on Ensemble SAC in $4$ MuJoCo tasks (Figure 7 (bottom)). We compare with Bootstrapped DQN and Ensemble SAC without CERL as well as the single-agent Double DQN and SAC. To show the advantage of CERL over explicit network sharing, we also include Bootstrapped DQN with network sharing and report ensemble diversity as we did in Section 3.3. We use $L$ to denote the number of shared layers across the ensemble. As the curse of diversity is sensitive to replay buffer size in MuJoCo tasks, we show results with different replay buffer sizes for these tasks. Additional results, including ensemble size ablations and an alternative design of CERL, can be found in Appendix D.4.

As shown in these results, CERL consistently mitigates the curse of diversity across the tested environments. For example, applying CERL to Ensemble SAC in Humanoid with a $0.2$M-sized replay buffer reduces the performance gap between SAC and Ensemble SAC (indiv.) from roughly $3000$ to around $500$. More importantly, the improvements in individual policies do translate into improvements in aggregate policies, which enables CERL to achieve the best performance with policy aggregation in both domains. In contrast, even though Bootstrapped DQN ($L=3$) also performs better than Bootstrapped DQN ($L=0$) when comparing individual policies, it does not provide any gain over Bootstrapped DQN ($L=0$) when comparing aggregate policies, likely due to significantly lower ensemble diversity than Bootstrapped DQN ($L=0$) as shown in Figure 7 (top right).

Ensemble-based exploration The idea of training an ensemble of data-sharing agents that concurrently explore has been employed in many deep RL algorithms (Osband et al., 2016; 2018; Liu et al., 2020; Schmitt et al., 2020; Peng et al., 2020; Hong et al., 2020; Januszewski et al., 2021). Most of these works focus on algorithmic design and the discussion of the potential negative effects of the increased “off-policy-ness” compared to single-agent training has largely been missing. To the best of our knowledge, the only work that explicitly discusses the potential difficulties of learning from the off-policy data generated by other ensemble members is Schmitt et al. (2020). However, since Schmitt et al. (2020) does not maintain explicit Q-values and relies on V-trace (Espeholt et al., 2018) for off-policy correction, algorithmic changes that allow stable off-policy learning are a requirement in their work. In contrast, we show that even for Q-learning-based methods that do not require explicit off-policy correction, ensemble-based exploration can still lead to performance degradation. There also exist methods that use multiple ensemble members on a per-step basis (Chen et al., 2018; Lee et al., 2021; Ishfaq et al., 2021; Li et al., 2023) as opposed to one ensemble member for each episode. The discussion of these methods is more subtle and is left for future work. Sun et al. (2022) uses an ensemble of Q-value networks to trade off exploration and exploitation. However, their method only uses one policy so our discussion does not apply to it.

Ensemble RL methods for other purposes Ensemble methods have also been employed in RL for purposes other than exploration, for example to produce robust value estimations  (Anschel et al., 2016; Lan et al., 2020; Agarwal et al., 2020; Peer et al., 2021; Chen et al., 2021; An et al., 2021; Wu et al., 2021; Liang et al., 2022) or model predictions (Chua et al., 2018; Kurutach et al., 2018). Despite the use of ensembles, most of these methods are still single-agent in nature (i.e., there is only one policy interacting with the environment). Thus, our discussion does not apply to these methods.

Mutual distillation Mutual/collaborative learning in supervised learning (Zhang et al., 2017; Anil et al., 2018; Guo et al., 2020; Wu & Gong, 2020) aims to train a cohort of networks and share knowledge between them via mutual distillation. Similar ideas have also been employed in RL (Czarnecki et al., 2018; Xue et al., 2020; Zhao & Hospedales, 2020; Reid & Mukhopadhyay, 2021). CERL is distinct from these works in that it does not try to distill different ensemble members’ predictions (i.e., the value functions) into each other. Instead, CERL is only an auxiliary task, and different ensemble members in CERL only affect each other’s representations via auxiliary losses.

Auxiliary tasks in RL Facilitating representation learning with auxiliary tasks has been shown to be effective in RL (Jaderberg et al., 2016; Mirowski et al., 2016; Fedus et al., 2019; Kartal et al., 2019; Dabney et al., 2020; Schwarzer et al., 2020). In the context of multi-agent RL, He & Boyd-Graber (2016), Hong et al. (2017); Hernandez-Leal et al. (2019) and Hernandez et al. (2022) model the policies of external agents as an auxiliary task, and Barde et al. (2019) promotes coordination between several trainable agents by maximizing their mutual action predictability. Besides the clear differences in the problem domains and motivations, these multi-agent works only predict the actions of other agents, which typically contain less information than the $Q$-function used in CERL’s auxiliary task.

In line with recent efforts to advance the understanding of deep RL through purposeful experiments (Ostrovski et al., 2021; Schaul et al., 2022; Nikishin et al., 2022; Sokar et al., 2023), our work builds on extensive, carefully designed empirical analyses. It offers valuable insights into a previously overlooked pitfall of the well-established approach of ensemble-based exploration and presents opportunities for future work. As with most empirical works, an important avenue for future research lies in develo** a theoretical understanding of the phenomenon we reveal in this work.

A limitation of CERL is its reliance on separate networks for high ensemble diversity, which may become infeasible with very large networks. A simple improvement to CERL is thus to combine CERL with network sharing and encourage diversity with other mechanisms, such as randomized prior functions (Osband et al., 2018) and explicit diversity regularization (Peng et al., 2020).

Detailed pseudocode of the ensemble algorithms used in this work is provided in Appendix A. Experimental and implementation details are given in Appendix B and Appendix C respectively. The source code is available at the following repositories:

• •

  Atari: https://github.com/zhixuan-lin/ensemble-rl-discrete

• •

  MuJoCo: https://github.com/zhixuan-lin/ensemble-rl-continuous

This research was enabled in part by support and compute resources provided by Mila (mila.quebec), Calcul Québec (www.calculquebec.ca), and the Digital Research Alliance of Canada (alliancecan.ca). We thank Sony for their financial support of Zhixuan Lin throughout this work.