Original: W EIGHTED
Revised: W EIGHTED

Original: B ELLMAN
Revised: B ELLMAN

Original: B ACKUPS
Revised: B ACKUPS

Original: FOR
Revised: FOR

Original: I MPROVED
Revised: I MPROVED

Original: S IGNAL - TO -N OISE IN Q-U PDATES
Revised: S IGNAL - TO -N OISE IN Q-U PDATES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Off-policy deep reinforcement learning (RL) has been successful in a range of challenging domains.
Revised: Off-policy deep reinforcement learning (RL) has been successful in a range of challenging domains.

Original: However, standard off-policy RL algorithms can suffer from low signal and even instability in Q-learning because target values are derived from current Q-estimates, which are often noisy.
Revised: However, standard off-policy RL algorithms can suffer from low signal and even instability in Q-learning because target values are derived from current Q-estimates, which are often noisy.

Original: To mitigate the issue, we propose ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble.
Revised: To mitigate the issue, we propose ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble.

Original: We empirically observe that the proposed method stabilizes and improves learning on both continuous and discrete control benchmarks.
Revised: We empirically observe that the proposed method stabilizes and improves learning on both continuous and discrete control benchmarks.

Original: We also speciﬁcally investigate the signal-to-noise aspect by studying environments with noisy rewards, and ﬁnd that weighted Bellman backups signiﬁcantly outperform standard Bellman backups.
Revised: We also speciﬁcally investigate the signal-to-noise aspect by studying environments with noisy rewards, and ﬁnd that weighted Bellman backups signiﬁcantly outperform standard Bellman backups.

Original: Furthermore, since our weighted Bellman backups rely on maintaining an ensemble, we investigate how weighted Bellman backups interact with UCB Exploration.
Revised: Furthermore, since our weighted Bellman backups rely on maintaining an ensemble, we investigate how weighted Bellman backups interact with UCB Exploration.

Original: By enforcing the diversity between agents using Bootstrap, we show that these different ideas are largely orthogonal and can be fruitfully integrated, together further improving the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both lowdimensional and high-dimensional environments.
Revised: By enforcing the diversity between agents using Bootstrap, we show that these different ideas are largely orthogonal and can be fruitfully integrated, together further improving the performance of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, for both continuous and discrete control tasks on both lowdimensional and high-dimensional environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Model-free reinforcement learning (RL), with high-capacity function approximators, such as deep neural networks (DNNs), has been used to solve a variety of sequential decision-making problems, including board games (Silver et al., 2017; 2018), video games (Mnih et al., 2015; Vinyals et al., 2019), and robotic manipulation (Kalashnikov et al., 2018).
Revised: Model-free reinforcement learning (RL), with high-capacity function approximators, such as deep neural networks (DNNs), has been used to solve a variety of sequential decision-making problems, including board games (Silver et al., 2017; 2018), video games (Mnih et al., 2015; Vinyals et al., 2019), and robotic manipulation (Kalashnikov et al., 2018).

Original: It has been well established that the above successes are highly sample inefﬁcient (Kaiser et al., 2020).
Revised: It has been well established that the above successes are highly sample inefﬁcient (Kaiser et al., 2020).

Original: Recently, a lot of progress has been made in more sample-efﬁcient model-free RL algorithms through improvements in off-policy learning both in discrete and continuous domains (Fujimoto et al., 2018; Haarnoja et al., 2018;
Revised: Recently, a lot of progress has been made in more sample-efﬁcient model-free RL algorithms through improvements in off-policy learning both in discrete and continuous domains (Fujimoto et al., 2018; Haarnoja et al., 2018;

Original: Hessel et al., 2018; Amos et al., 2020).
Revised: Hessel et al., 2018; Amos et al., 2020).

Original: However, standard off-policy RL algorithms can suffer from instability in Q-learning due to error propagation in the Bellman backup, i.e., the errors induced in the target value can lead to an increase in overall error in the Q-function (Kumar et al., 2019; 2020).
Revised: However, standard off-policy RL algorithms can suffer from instability in Q-learning due to error propagation in the Bellman backup, i.e., the errors induced in the target value can lead to an increase in overall error in the Q-function (Kumar et al., 2019; 2020).

Original: One way to address the error propagation issue is to use ensemble methods, which combine multiple models of the value function (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et al., 2018).
Revised: One way to address the error propagation issue is to use ensemble methods, which combine multiple models of the value function (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et al., 2018).

Original: For discrete control tasks, double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016) addressed the value overestimation by maintaining two independent estimators of the action values and later extended to continuous control tasks in TD3 (Fujimoto et al., 2018).
Revised: For discrete control tasks, double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016) addressed the value overestimation by maintaining two independent estimators of the action values and later extended to continuous control tasks in TD3 (Fujimoto et al., 2018).

Original: While most prior work has improved the stability by taking the minimum over Q-functions, this also needlessly loses signal, and we propose an alternative way that utilizes ensembles to estimate uncertainty and provide more stable backups.
Revised: While most prior work has improved the stability by taking the minimum over Q-functions, this also needlessly loses signal, and we propose an alternative way that utilizes ensembles to estimate uncertainty and provide more stable backups.

Original: In this paper, we propose ensemble-based weighted Bellman backups that can be applied to most modern off-policy RL algorithms, such as Q-learning and actor-critic algorithms.
Revised: In this paper, we propose ensemble-based weighted Bellman backups that can be applied to most modern off-policy RL algorithms, such as Q-learning and actor-critic algorithms.

Original: Our main idea is to reweight sample transitions based on uncertainty estimates from a Q-ensemble.
Revised: Our main idea is to reweight sample transitions based on uncertainty estimates from a Q-ensemble.

Original: Because prediction errors can be characterized by uncertainty estimates from ensembles (i.e., variance of predictions) as shown in Figure 1(b), we ﬁnd that the proposed method signiﬁcantly improves the signal-to-noise in the Q-updates and stabilizes the learning process.
Revised: Because prediction errors can be characterized by uncertainty estimates from ensembles (i.e., variance of predictions) as shown in Figure 1(b), we ﬁnd that the proposed method signiﬁcantly improves the signal-to-noise in the Q-updates and stabilizes the learning process.

Original: Finally, we present a uniﬁed framework, coined
Revised: Finally, we present a uniﬁed framework, coined

Original: SUNRISE, that combines our weighted Bellman backups with an inference method that selects actions using highest upper-conﬁdence bounds (UCB) for efﬁcient exploration (Chen et al., 2017).
Revised: SUNRISE, that combines our weighted Bellman backups with an inference method that selects actions using highest upper-conﬁdence bounds (UCB) for efﬁcient exploration (Chen et al., 2017).

Original: We ﬁnd that these different ideas can be fruitfully integrated, and they are largely complementary (see Figure 1(a)).
Revised: We ﬁnd that these different ideas can be fruitfully integrated, and they are largely complementary (see Figure 1(a)).

Original: We demonstrate the effectiveness of the proposed method using Soft Actor-Critic (SAC; Haarnoja et al. 2018) for continuous control benchmarks (speciﬁcally, OpenAI Gym (Brockman et al., 2016) and DeepMind Control Suite (Tassa et al., 2018)) and Rainbow DQN (Hessel et al., 2018) for discrete control benchmarks (speciﬁcally, Atari games (Bellemare et al., 2013)).
Revised: We demonstrate the effectiveness of the proposed method using Soft Actor-Critic (SAC; Haarnoja et al. 2018) for continuous control benchmarks (speciﬁcally, OpenAI Gym (Brockman et al., 2016) and DeepMind Control Suite (Tassa et al., 2018)) and Rainbow DQN (Hessel et al., 2018) for discrete control benchmarks (speciﬁcally, Atari games (Bellemare et al., 2013)).

Original: In our experiments,
Revised: In our experiments,

Original: SUNRISE consistently improves the performance of existing off-policy RL methods.
Revised: SUNRISE consistently improves the performance of existing off-policy RL methods.

Original: Furthermore, we ﬁnd that the proposed weighted Bellman backups yield improvements in environments with noisy reward, which have a low signal-to-noise ratio.
Revised: Furthermore, we ﬁnd that the proposed weighted Bellman backups yield improvements in environments with noisy reward, which have a low signal-to-noise ratio.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Off-policy RL algorithms .
Revised: Off-policy RL algorithms .

Original: Recently, various off-policy RL algorithms have provided large gains in sample-efﬁciency by reusing past experiences (Fujimoto et al., 2018;
Revised: Recently, various off-policy RL algorithms have provided large gains in sample-efﬁciency by reusing past experiences (Fujimoto et al., 2018;

Original: Haarnoja et al., 2018;
Revised: Haarnoja et al., 2018;

Original: Hessel et al., 2018).
Revised: Hessel et al., 2018).

Original: Rainbow DQN (Hessel et al., 2018) achieved state-of-the-art performance on the Atari games (Bellemare et al., 2013) by combining several techniques, such as double Qlearning (Van Hasselt et al., 2016) and distributional DQN (Bellemare et al., 2017).
Revised: Rainbow DQN (Hessel et al., 2018) achieved state-of-the-art performance on the Atari games (Bellemare et al., 2013) by combining several techniques, such as double Qlearning (Van Hasselt et al., 2016) and distributional DQN (Bellemare et al., 2017).

Original: For continuous control tasks, SAC (Haarnoja et al., 2018) achieved state-of-the-art sample-efﬁciency results by incorporating the maximum entropy framework.
Revised: For continuous control tasks, SAC (Haarnoja et al., 2018) achieved state-of-the-art sample-efﬁciency results by incorporating the maximum entropy framework.

Original: Our ensemble method brings orthogonal beneﬁts and is complementary and compatible with these existing state-of-the-art algorithms.
Revised: Our ensemble method brings orthogonal beneﬁts and is complementary and compatible with these existing state-of-the-art algorithms.

Original: Stabilizing Q-learning .
Revised: Stabilizing Q-learning .

Original: It has been empirically observed that instability in Q-learning can be caused by applying the Bellman backup on the learned value function (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et al., 2018;
Revised: It has been empirically observed that instability in Q-learning can be caused by applying the Bellman backup on the learned value function (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et al., 2018;

Original: Song et al., 2019;
Revised: Song et al., 2019;

Original: Kim et al., 2019; Kumar et al., 2019; 2020).
Revised: Kim et al., 2019; Kumar et al., 2019; 2020).

Original: By following the principle of double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016), twin-Q trick (Fujimoto et al., 2018) was proposed to handle the overestimation of value functions for continuous control tasks.
Revised: By following the principle of double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016), twin-Q trick (Fujimoto et al., 2018) was proposed to handle the overestimation of value functions for continuous control tasks.

Original: Song et al.
Revised: Song et al.

Original: and Kim et al.
Revised: and Kim et al.

Original: (2019) proposed to replace the max operator with
Revised: (2019) proposed to replace the max operator with

Original: Softmax and Mellowmax, respectively, to reduce the overestimation error.
Revised: Softmax and Mellowmax, respectively, to reduce the overestimation error.

Original: Recently, Kumar et al.
Revised: Recently, Kumar et al.

Original: (2020) handled the error propagation issue by reweighting the Bellman backup based on cumulative
Revised: (2020) handled the error propagation issue by reweighting the Bellman backup based on cumulative

Original: Bellman errors.
Revised: Bellman errors.

Original: However, our method is different in that we propose an alternative way that also utilizes ensembles to estimate uncertainty and provide more stable, higher-signal-to-noise backups.
Revised: However, our method is different in that we propose an alternative way that also utilizes ensembles to estimate uncertainty and provide more stable, higher-signal-to-noise backups.

Original: Ensemble methods in RL .
Revised: Ensemble methods in RL .

Original: Ensemble methods have been studied for different purposes in RL (Wiering & Van Hasselt, 2008;
Revised: Ensemble methods have been studied for different purposes in RL (Wiering & Van Hasselt, 2008;

Original: Osband et al., 2016a; Anschel et al., 2017; Agarwal et al., 2020; Lan et al., 2020).
Revised: Osband et al., 2016a; Anschel et al., 2017; Agarwal et al., 2020; Lan et al., 2020).

Original: Chua et al.
Revised: Chua et al.

Original: (2018) showed that modeling errors in model-based RL can be reduced using an ensemble of dynamics models, and Kurutach et al.
Revised: (2018) showed that modeling errors in model-based RL can be reduced using an ensemble of dynamics models, and Kurutach et al.

Original: (2018) accelerated policy learning by generating imagined experiences from the ensemble of dynamics models.
Revised: (2018) accelerated policy learning by generating imagined experiences from the ensemble of dynamics models.

Original: For efﬁcient exploration, Osband et al.
Revised: For efﬁcient exploration, Osband et al.

Original: (2016a)
Revised: (2016a)

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2017) also leveraged the ensemble of Q-functions.
Revised: (2017) also leveraged the ensemble of Q-functions.

Original: However, most prior works have studied the various axes of improvements from ensemble methods in isolation, while we propose a uniﬁed framework that handles various issues in off-policy RL algorithms.
Revised: However, most prior works have studied the various axes of improvements from ensemble methods in isolation, while we propose a uniﬁed framework that handles various issues in off-policy RL algorithms.

Original: Exploration in RL .
Revised: Exploration in RL .

Original: To balance exploration and exploitation, several methods, such as the maximum entropy frameworks (Ziebart, 2010; Haarnoja et al., 2018), exploration bonus rewards (Bellemare et al., 2016; Houthooft et al., 2016; Pathak et al., 2017; Choi et al., 2019) and randomization (Osband et al., 2016a;b), have been proposed.
Revised: To balance exploration and exploitation, several methods, such as the maximum entropy frameworks (Ziebart, 2010; Haarnoja et al., 2018), exploration bonus rewards (Bellemare et al., 2016; Houthooft et al., 2016; Pathak et al., 2017; Choi et al., 2019) and randomization (Osband et al., 2016a;b), have been proposed.

Original: Despite the success of these exploration methods, a potential drawback is that agents can focus on irrelevant aspects of the environment because these methods do not depend on the rewards.
Revised: Despite the success of these exploration methods, a potential drawback is that agents can focus on irrelevant aspects of the environment because these methods do not depend on the rewards.

Original: To handle this issue, Chen et al.
Revised: To handle this issue, Chen et al.

Original: (2017) proposed an exploration strategy that considers both best estimates (i.e., mean) and uncertainty (i.e., variance) of Q-functions for discrete control tasks.
Revised: (2017) proposed an exploration strategy that considers both best estimates (i.e., mean) and uncertainty (i.e., variance) of Q-functions for discrete control tasks.

Original: We further extend this strategy to continuous control tasks and show that it can be combined with other techniques.
Revised: We further extend this strategy to continuous control tasks and show that it can be combined with other techniques.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Reinforcement learning .
Revised: Reinforcement learning .

Original: We consider a standard RL framework where an agent interacts with an environment in discrete time.
Revised: We consider a standard RL framework where an agent interacts with an environment in discrete time.

Original: Formally, at each timestep t , the agent receives a state s t from the environment and chooses an action a t based on its policy π .
Revised: Formally, at each timestep t , the agent receives a state s t from the environment and chooses an action a t based on its policy π .

Original: The environment returns a reward r t and the agent transitions to the next state s t +1 .
Revised: The environment returns a reward r t and the agent transitions to the next state s t +1 .

Original: The return R t = (cid:80) ∞ k =0 γ k r t
Revised: The return R t = (cid:80) ∞ k =0 γ k r t

Original: + k is the total accumulated rewards from timestep t with a discount factor γ ∈ [0 , 1) .
Revised: + k is the total accumulated rewards from timestep t with a discount factor γ ∈ [0 , 1) .

Original: RL then maximizes the expected return.
Revised: RL then maximizes the expected return.

Original: [Figure]
Revised: [Figure]

Original: Soft Actor-Critic .
Revised: Soft Actor-Critic .

Original: SAC (Haarnoja et al., 2018) is an off-policy actor-critic method based on the maximum entropy RL framework (Ziebart, 2010), which encourages the robustness to noise and exploration by maximizing a weighted objective of the reward and the policy entropy (see Appendix A for further details).
Revised: SAC (Haarnoja et al., 2018) is an off-policy actor-critic method based on the maximum entropy RL framework (Ziebart, 2010), which encourages the robustness to noise and exploration by maximizing a weighted objective of the reward and the policy entropy (see Appendix A for further details).

Original: To update the parameters, SAC alternates between a soft policy evaluation and a soft policy improvement.
Revised: To update the parameters, SAC alternates between a soft policy evaluation and a soft policy improvement.

Original: At the soft policy evaluation step, a soft Q-function, which is modeled as a neural network with parameters θ , is updated by minimizing the following soft Bellman residual:
Revised: At the soft policy evaluation step, a soft Q-function, which is modeled as a neural network with parameters θ , is updated by minimizing the following soft Bellman residual:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where τ t =
Revised: where τ t =

Original: ( s t , a t , r t , s t +1 ) is a transition, B is a replay buffer, ¯ θ are the delayed parameters, and α is a temperature parameter.
Revised: ( s t , a t , r t , s t +1 ) is a transition, B is a replay buffer, ¯ θ are the delayed parameters, and α is a temperature parameter.

Original: At the soft policy improvement step, the policy π with its parameter φ is updated by minimizing the following objective:
Revised: At the soft policy improvement step, the policy π with its parameter φ is updated by minimizing the following objective:

Original: L SACactor ( φ ) =
Revised: L SACactor ( φ ) =

Original: E s t ∼B (cid:2)
Revised: E s t ∼B (cid:2)

Original: L π ( s t , φ ) (cid:3) , where L π ( s t , φ ) =
Revised: L π ( s t , φ ) (cid:3) , where L π ( s t , φ ) =

Original: E a t ∼ π φ (cid:2)
Revised: E a t ∼ π φ (cid:2)

Original: α log π φ ( a t | s t ) − Q θ ( s t , a t ) (cid:3) .
Revised: α log π φ ( a t | s t ) − Q θ ( s t , a t ) (cid:3) .

Original: Here, the policy is modeled as a Gaussian with mean and covariance given by neural networks to handle continuous action spaces.
Revised: Here, the policy is modeled as a Gaussian with mean and covariance given by neural networks to handle continuous action spaces.

Original:  SUNRISE
Revised:  SUNRISE

Original: In this section, we propose the ensemble-based weighted Bellman backups, and then introduce SUNRISE: S imple UN iﬁed framework for R e I nforcement learning using en SE mbles, which combines various ensemble methods.
Revised: In this section, we propose the ensemble-based weighted Bellman backups, and then introduce SUNRISE: S imple UN iﬁed framework for R e I nforcement learning using en SE mbles, which combines various ensemble methods.

Original: In principle, our method can be used in conjunction with most modern off-policy RL algorithms, such as SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018).
Revised: In principle, our method can be used in conjunction with most modern off-policy RL algorithms, such as SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018).

Original: For the exposition, we describe only the SAC version in the main body.
Revised: For the exposition, we describe only the SAC version in the main body.

Original: The Rainbow DQN version follows the same principles and is fully described in Appendix B.
Revised: The Rainbow DQN version follows the same principles and is fully described in Appendix B.

Original: 4. W EIGHTED B ELLMAN BACKUPS TO IMPROVE SIGNAL - TO - NOISE IN Q- UPDATES
Revised: 4. W EIGHTED B ELLMAN BACKUPS TO IMPROVE SIGNAL - TO - NOISE IN Q- UPDATES

Original: Formally, we consider an ensemble of N SAC agents, i.e., { Q θ i , π φ i } Ni =1 , where θ
Revised: Formally, we consider an ensemble of N SAC agents, i.e., { Q θ i , π φ i } Ni =1 , where θ

Original: i and φ i denote the parameters of the i -th soft Q-function and policy.
Revised: i and φ i denote the parameters of the i -th soft Q-function and policy.

Original: 1 Since conventional Q-learning is based on the Bellman backup in (2), it can be affected by error propagation.
Revised: 1 Since conventional Q-learning is based on the Bellman backup in (2), it can be affected by error propagation.

Original: I.e., error in the target Q-function
Revised: I.e., error in the target Q-function

Original: Q ¯ θ ( s t +1 , a t +1 ) gets propagated into the Q-function Q θ ( s t , a t ) at the current state.
Revised: Q ¯ θ ( s t +1 , a t +1 ) gets propagated into the Q-function Q θ ( s t , a t ) at the current state.

Revised: In other words, errors in the previous Q-function induce the “noise” to the learning “signal” (i.e., true Q-value) of the current Q-function.

Original: Recently, Kumar et al.
Revised: Recently, Kumar et al.

Original: (2020) showed that this error propagation can cause inconsistency and unstable convergence.
Revised: (2020) showed that this error propagation can cause inconsistency and unstable convergence.

Original: To mitigate this issue, for each agent i , we consider a weighted Bellman backup as follows:
Revised: To mitigate this issue, for each agent i , we consider a weighted Bellman backup as follows:

Original: [Equation]
Revised: [Equation]

Original: where τ t =
Revised: where τ t =

Original: ( s t , a t , r t , s t +1 ) is a transition, a t +1 ∼
Revised: ( s t , a t , r t , s t +1 ) is a transition, a t +1 ∼

Original: π φ ( a | s t ) , and w ( s, a ) is a conﬁdence weight based on ensemble of target Q-functions:
Revised: π φ ( a | s t ) , and w ( s, a ) is a conﬁdence weight based on ensemble of target Q-functions:

Original: [Equation]
Revised: [Equation]

Original: where T > 0 is a temperature, σ is the sigmoid function, and ¯ Q std ( s, a ) is the empirical standard deviation of all target Q-functions { Q ¯ θ i } Ni =1 .
Revised: where T > 0 is a temperature, σ is the sigmoid function, and ¯ Q std ( s, a ) is the empirical standard deviation of all target Q-functions { Q ¯ θ i } Ni =1 .

Original: Note that the conﬁdence weight is bounded in [0 . 5 , 1 . 0] because standard deviation is always positive.
Revised: Note that the conﬁdence weight is bounded in [0 . 5 , 1 . 0] because standard deviation is always positive.

Original: The proposed objective L WQ down-weights the sample transitions with high variance across target Q-functions, resulting in a loss function for the
Revised: The proposed objective L WQ down-weights the sample transitions with high variance across target Q-functions, resulting in a loss function for the

Original: Q-updates that has a better signal-to-noise ratio.
Revised: Q-updates that has a better signal-to-noise ratio.

Original: 4. C OMBINATION WITH ADDITIONAL TECHNIQUES THAT LEVERAGE ENSEMBLES
Revised: 4. C OMBINATION WITH ADDITIONAL TECHNIQUES THAT LEVERAGE ENSEMBLES

Original: We integrate the proposed weighted Bellman backup with UCB exploration into a single framework by utilizing the bootstrap with random initialization.
Revised: We integrate the proposed weighted Bellman backup with UCB exploration into a single framework by utilizing the bootstrap with random initialization.

Original: Bootstrap with random initialization .
Revised: Bootstrap with random initialization .

Original: To train the ensemble of agents, we use the bootstrap with random initialization (Efron, 1982; Osband et al., 2016a), which enforces the diversity between agents through two simple ideas: First, we initialize the model parameters of all agents with random parameter values for inducing an initial diversity in the models.
Revised: To train the ensemble of agents, we use the bootstrap with random initialization (Efron, 1982; Osband et al., 2016a), which enforces the diversity between agents through two simple ideas: First, we initialize the model parameters of all agents with random parameter values for inducing an initial diversity in the models.

Original: Second, we apply different samples to train each agent.
Revised: Second, we apply different samples to train each agent.

Original: Speciﬁcally, for each SAC agent i in each timestep t , we draw the binary masks m t,i from the Bernoulli distribution with parameter β ∈ (0 , 1] , and store them in the replay buffer.
Revised: Speciﬁcally, for each SAC agent i in each timestep t , we draw the binary masks m t,i from the Bernoulli distribution with parameter β ∈ (0 , 1] , and store them in the replay buffer.

Original: Then, when updating the model parameters of agents, we multiply the bootstrap mask to each objective function, such as: m t,i L π ( s t , φ i ) and m t,i L WQ ( τ t , θ i ) in (3) and (4), respectively.
Revised: Then, when updating the model parameters of agents, we multiply the bootstrap mask to each objective function, such as: m t,i L π ( s t , φ i ) and m t,i L WQ ( τ t , θ i ) in (3) and (4), respectively.

Original: We remark that Osband et al.
Revised: We remark that Osband et al.

Original: (2016a) applied this simple technique to train an ensemble of DQN (Mnih et al., 2015) only for discrete control tasks, while we apply to SAC (Haarnoja et al., 2018) and Rainbow
Revised: (2016a) applied this simple technique to train an ensemble of DQN (Mnih et al., 2015) only for discrete control tasks, while we apply to SAC (Haarnoja et al., 2018) and Rainbow

Original: DQN (Hessel et al., 2018) for both continuous and discrete tasks with additional techniques.
Revised: DQN (Hessel et al., 2018) for both continuous and discrete tasks with additional techniques.

Original: UCB exploration .
Revised: UCB exploration .

Original: The ensemble can also be leveraged for efﬁcient exploration (Chen et al., 2017;
Revised: The ensemble can also be leveraged for efﬁcient exploration (Chen et al., 2017;

Original: Osband et al., 2016a) because it can express higher uncertainty on unseen samples.
Revised: Osband et al., 2016a) because it can express higher uncertainty on unseen samples.

Original: Motivated by this, by following the idea of Chen et al.
Revised: Motivated by this, by following the idea of Chen et al.

Original: (2017), we consider an optimism-based exploration that chooses the action that maximizes
Revised: (2017), we consider an optimism-based exploration that chooses the action that maximizes

Original: [Equation]
Revised: [Equation]

Original: where Q mean ( s, a ) and Q std ( s, a ) are the empirical mean and standard deviation of all Q-functions { Q θ i } Ni =1 , and the λ > 0 is a hyperparameter.
Revised: where Q mean ( s, a ) and Q std ( s, a ) are the empirical mean and standard deviation of all Q-functions { Q θ i } Ni =1 , and the λ > 0 is a hyperparameter.

Original: This inference method can encourage exploration by adding an exploration bonus (i.e., standard deviation Q std ) for visiting unseen state-action pairs similar to the UCB algorithm (Auer et al., 2002).
Revised: This inference method can encourage exploration by adding an exploration bonus (i.e., standard deviation Q std ) for visiting unseen state-action pairs similar to the UCB algorithm (Auer et al., 2002).

Original: We remark that this inference method was originally proposed in Chen et al.
Revised: We remark that this inference method was originally proposed in Chen et al.

Original: (2017) for efﬁcient exploration in discrete action spaces.
Revised: (2017) for efﬁcient exploration in discrete action spaces.

Original: However, in continuous action spaces, ﬁnding the action that maximizes the UCB is not straightforward.
Revised: However, in continuous action spaces, ﬁnding the action that maximizes the UCB is not straightforward.

Original: To handle this issue, we propose a simple approximation scheme, which ﬁrst generates N candidate action set from ensemble policies { π φ i } Ni =1 , and then chooses the action that maximizes the UCB (Linein Algorithm 1).
Revised: To handle this issue, we propose a simple approximation scheme, which ﬁrst generates N candidate action set from ensemble policies { π φ i } Ni =1 , and then chooses the action that maximizes the UCB (Linein Algorithm 1).

Original: For evaluation, we approximate the maximum a posterior action by averaging the mean of Gaussian distributions modeled by each ensemble policy.
Revised: For evaluation, we approximate the maximum a posterior action by averaging the mean of Gaussian distributions modeled by each ensemble policy.

Original: The full procedure of our uniﬁed framework, coined SUNRISE, is summarized in Algorithm 1.
Revised: The full procedure of our uniﬁed framework, coined SUNRISE, is summarized in Algorithm 1.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We designed our experiments to answer the following questions:
Revised: We designed our experiments to answer the following questions:

Original: • Can SUNRISE improve off-policy RL algorithms, such as SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018), for both continuous (see Table 1 and Table 2) and discrete (see Table 3) control tasks?
Revised: • Can SUNRISE improve off-policy RL algorithms, such as SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018), for both continuous (see Table 1 and Table 2) and discrete (see Table 3) control tasks?

Original: • How crucial is the proposed weighted Bellman backups in (4) for improving the signal-to-noise in Q-updates (see Figure 2)?
Revised: • How crucial is the proposed weighted Bellman backups in (4) for improving the signal-to-noise in Q-updates (see Figure 2)?

Original: • Can UCB exploration be useful for solving tasks with sparse rewards (see Figure 3(b))?
Revised: • Can UCB exploration be useful for solving tasks with sparse rewards (see Figure 3(b))?

Original: • Is SUNRISE better than a single agent with more updates and parameters (see Figure 3(c))?
Revised: • Is SUNRISE better than a single agent with more updates and parameters (see Figure 3(c))?

Original: • How does ensemble size affect the performance (see Figure 3(d))?
Revised: • How does ensemble size affect the performance (see Figure 3(d))?

Original: 1: for each iteration do 2: for each timestep t do 3: //
Revised: 1: for each iteration do 2: for each timestep t do 3: //

Original: UCB EXPLORATION 4:
Revised: UCB EXPLORATION 4:

Original: Collect N action samples: A t = { a t,i ∼ π φ i ( a | s t ) | i ∈ { 1 , . . .
Revised: Collect N action samples: A t = { a t,i ∼ π φ i ( a | s t ) | i ∈ { 1 , . . .

Original: , N }} 5:
Revised: , N }} 5:

Original: Choose the action that maximizes UCB: a t = arg max a t,i ∈A t Q mean ( s t , a t,i )+ λQ std ( s t , a t,i ) 6:
Revised: Choose the action that maximizes UCB: a t = arg max a t,i ∈A t Q mean ( s t , a t,i )+ λQ std ( s t , a t,i ) 6:

Original: Collect state s t +1 and reward r t from the environment by taking action a t 7:
Revised: Collect state s t +1 and reward r t from the environment by taking action a t 7:

Original: Sample bootstrap masks M t = { m t,i ∼ Bernoulli ( β ) — i ∈ { 1 , . . .
Revised: Sample bootstrap masks M t = { m t,i ∼ Bernoulli ( β ) — i ∈ { 1 , . . .

Original: , N }} 8:
Revised: , N }} 8:

Original: Store transitions τ t = ( s t , a t , s t +1 , r t ) and masks in replay buffer B ← B ∪ { ( τ t , M t ) } 9: end for 10: // U PDATE AGENTS
Revised: Store transitions τ t = ( s t , a t , s t +1 , r t ) and masks in replay buffer B ← B ∪ { ( τ t , M t ) } 9: end for 10: // U PDATE AGENTS

Original: VIA BOOTSTRAP AND WEIGHTED B ELLMAN BACKUP 11: for each gradient step do 12:
Revised: VIA BOOTSTRAP AND WEIGHTED B ELLMAN BACKUP 11: for each gradient step do 12:

Original: Sample random minibatch { ( τ j , M j ) } Bj =1 ∼ B 13: for each agent i do 14:
Revised: Sample random minibatch { ( τ j , M j ) } Bj =1 ∼ B 13: for each agent i do 14:

Original: Update the Q-function by minimizing 1 B (cid:80) Bj =

Revised: Update the Q-function by minimizing 1 B (cid:80) Bj =

Original: m j,i L WQ ( τ j , θ i ) in (4) 15:
Revised: m j,i L WQ ( τ j , θ i ) in (4) 15:

Original: Update the policy by minimizing 1 B (cid:80) Bj =

Revised: Update the policy by minimizing 1 B (cid:80) Bj =

Original: m j,i L π ( s j , φ i ) in (3) 16: end for 17: end for 18: end for
Revised: m j,i L π ( s j , φ i ) in (3) 16: end for 17: end for 18: end for

Original: 5. S ETUPS
Revised: 5. S ETUPS

Original: Continuous control tasks .
Revised: Continuous control tasks .

Original: We evaluate SUNRISE on several continuous control tasks using simulated robots from OpenAI Gym (Brockman et al., 2016) and DeepMind Control Suite (Tassa et al., 2018).
Revised: We evaluate SUNRISE on several continuous control tasks using simulated robots from OpenAI Gym (Brockman et al., 2016) and DeepMind Control Suite (Tassa et al., 2018).

Original: For OpenAI Gym experiments with proprioceptive inputs (e.g., positions and velocities), we compare to PETS (Chua et al., 2018), a state-of-the-art model-based RL method based on ensembles of dynamics models; POPLIN-P (Wang & Ba, 2020), a state-of-the-art model-based RL method which uses a policy network to generate actions for planning; POPLIN-A (Wang & Ba, 2020), variant of POPLIN-P which adds noise in the action space; METRPO (Kurutach et al., 2018), a hybrid
Revised: For OpenAI Gym experiments with proprioceptive inputs (e.g., positions and velocities), we compare to PETS (Chua et al., 2018), a state-of-the-art model-based RL method based on ensembles of dynamics models; POPLIN-P (Wang & Ba, 2020), a state-of-the-art model-based RL method which uses a policy network to generate actions for planning; POPLIN-A (Wang & Ba, 2020), variant of POPLIN-P which adds noise in the action space; METRPO (Kurutach et al., 2018), a hybrid

Original: RL method which augments TRPO (Schulman et al., 2015) using ensembles of dynamics models; and two state-of-the-art model-free RL methods, TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018).
Revised: RL method which augments TRPO (Schulman et al., 2015) using ensembles of dynamics models; and two state-of-the-art model-free RL methods, TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018).

Original: For our method, we consider a combination of SAC and SUNRISE, as described in
Revised: For our method, we consider a combination of SAC and SUNRISE, as described in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Following the setup in Wang & Ba (2020) and Wang et al.
Revised: Following the setup in Wang & Ba (2020) and Wang et al.

Original: (2019), we report the mean and standard deviation across ten runs after 200K timesteps on ﬁve complex environments: Cheetah,
Revised: (2019), we report the mean and standard deviation across ten runs after 200K timesteps on ﬁve complex environments: Cheetah,

Original: Walker, Hopper, Ant and SlimHumanoid with early termination (ET).
Revised: Walker, Hopper, Ant and SlimHumanoid with early termination (ET).

Original: More experimental details and learning curves with 1M timesteps are in Appendix D.
Revised: More experimental details and learning curves with 1M timesteps are in Appendix D.

Original: For DeepMind Control Suite with image inputs, we compare to PlaNet (Hafner et al., 2019), a model-based RL method which learns a latent dynamics model and uses it for planning; Dreamer
Revised: For DeepMind Control Suite with image inputs, we compare to PlaNet (Hafner et al., 2019), a model-based RL method which learns a latent dynamics model and uses it for planning; Dreamer

Original: (Hafner et al., 2020), a hybrid RL method which utilizes the latent dynamics model to generate synthetic roll-outs; SLAC (Lee et al., 2020), a hybrid RL method which combines the latent dynamics model with SAC; and three state-of-the-art model-free RL methods which apply contrastive learning (CURL; Srinivas et al. 2020) or data augmentation (RAD (Laskin et al., 2020) and DrQ (Kostrikov et al., 2020)) to SAC.
Revised: (Hafner et al., 2020), a hybrid RL method which utilizes the latent dynamics model to generate synthetic roll-outs; SLAC (Lee et al., 2020), a hybrid RL method which combines the latent dynamics model with SAC; and three state-of-the-art model-free RL methods which apply contrastive learning (CURL; Srinivas et al. 2020) or data augmentation (RAD (Laskin et al., 2020) and DrQ (Kostrikov et al., 2020)) to SAC.

Original: For our method, we consider a combination of RAD (i.e., SAC with random crop) and SUNRISE.
Revised: For our method, we consider a combination of RAD (i.e., SAC with random crop) and SUNRISE.

Original: Following the setup in RAD, we report the mean and standard deviation across ﬁve runs after 100k (i.e., low sample regime) and 500k (i.e., asymptotically optimal regime) environment steps on six environments: Finger-spin, Cartpole-swing, Reacher-easy, Cheetah-run,
Revised: Following the setup in RAD, we report the mean and standard deviation across ﬁve runs after 100k (i.e., low sample regime) and 500k (i.e., asymptotically optimal regime) environment steps on six environments: Finger-spin, Cartpole-swing, Reacher-easy, Cheetah-run,

Original: Walker-walk, and Cup-catch.
Revised: Walker-walk, and Cup-catch.

Original: More experimental details and learning curves are in Appendix F.
Revised: More experimental details and learning curves are in Appendix F.

Original: Discrete control benchmarks .
Revised: Discrete control benchmarks .

Original: For discrete control tasks, we demonstrate the effectiveness of SUNRISE on several Atari games (Bellemare et al., 2013).
Revised: For discrete control tasks, we demonstrate the effectiveness of SUNRISE on several Atari games (Bellemare et al., 2013).

Original: We compare to SimPLe (Kaiser et al., 2020), a hybrid RL method which updates the policy only using samples generated by learned dynamics model; Rainbow DQN (Hessel et al., 2018) with modiﬁed hyperparameters for sample-efﬁciency (van Hasselt et al., 2019); Random agent (Kaiser et al., 2020); two state-of-the-art model-free RL methods which apply the contrastive learning (CURL; Srinivas et al. 2020) and data augmentation (DrQ; Kostrikov et al. 2020) to Rainbow DQN; and Human performances reported in Kaiser et al.
Revised: We compare to SimPLe (Kaiser et al., 2020), a hybrid RL method which updates the policy only using samples generated by learned dynamics model; Rainbow DQN (Hessel et al., 2018) with modiﬁed hyperparameters for sample-efﬁciency (van Hasselt et al., 2019); Random agent (Kaiser et al., 2020); two state-of-the-art model-free RL methods which apply the contrastive learning (CURL; Srinivas et al. 2020) and data augmentation (DrQ; Kostrikov et al. 2020) to Rainbow DQN; and Human performances reported in Kaiser et al.

Original: (2020) and van Hasselt et al.
Revised: (2020) and van Hasselt et al.

Original: Following the setups in SimPLe, we report the mean across
Revised: Following the setups in SimPLe, we report the mean across

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For three runs after 100K interactions (i.e., 400K frames with action repeat of 4).
Revised: For three runs after 100K interactions (i.e., 400K frames with action repeat of 4).

Original: For our method, we consider a combination of sample-efﬁcient versions of Rainbow DQN and SUNRISE (see Algorithm 3 in Appendix B).
Revised: For our method, we consider a combination of sample-efﬁcient versions of Rainbow DQN and SUNRISE (see Algorithm 3 in Appendix B).

Original: More experimental details and learning curves are in Appendix G.
Revised: More experimental details and learning curves are in Appendix G.

Original: For our method, we do not alter any hyperparameters of the original RL algorithms and train ﬁve ensemble agents.
Revised: For our method, we do not alter any hyperparameters of the original RL algorithms and train ﬁve ensemble agents.

Original: There are only three additional hyperparameters β , T , and λ for bootstrap, weighted
Revised: There are only three additional hyperparameters β , T , and λ for bootstrap, weighted

Original: Bellman backup, and UCB exploration, where we provide details in Appendix D, F, and G.
Revised: Bellman backup, and UCB exploration, where we provide details in Appendix D, F, and G.

Original: 5. C OMPARATIVE EVALUATION
Revised: 5. C OMPARATIVE EVALUATION

Original: OpenAI Gym .
Revised: OpenAI Gym .

Original: Table 1 shows the average returns of evaluation roll-outs for all methods.
Revised: Table 1 shows the average returns of evaluation roll-outs for all methods.

Original: SUNRISE consistently improves the performance of SAC across all environments and outperforms the model-based RL methods, such as POPLIN-P and PETS, on all environments except Ant and
Revised: SUNRISE consistently improves the performance of SAC across all environments and outperforms the model-based RL methods, such as POPLIN-P and PETS, on all environments except Ant and

Original: SlimHumanoid-ET.
Revised: SlimHumanoid-ET.

Original: Even though we focus on performance after small samples because of the recent emphasis on making RL more sample efﬁcient, we ﬁnd that the gain from SUNRISE becomes even more signiﬁcant when training longer (see Figure 3(c) and Appendix D).
Revised: Even though we focus on performance after small samples because of the recent emphasis on making RL more sample efﬁcient, we ﬁnd that the gain from SUNRISE becomes even more signiﬁcant when training longer (see Figure 3(c) and Appendix D).

Original: We remark that SUNRISE is more compute-efﬁcient than modern model-based RL methods, such as POPLIN and PETS, because they also utilize ensembles (of dynamics models) and perform planning to select actions.
Revised: We remark that SUNRISE is more compute-efﬁcient than modern model-based RL methods, such as POPLIN and PETS, because they also utilize ensembles (of dynamics models) and perform planning to select actions.

Original: Namely, SUNRISE is simple to implement, computationally efﬁcient, and readily parallelizable.
Revised: Namely, SUNRISE is simple to implement, computationally efﬁcient, and readily parallelizable.

Original: DeepMind Control Suite .
Revised: DeepMind Control Suite .

Original: As shown in Table 2, SUNRISE also consistently improves the performance of RAD (i.e., SAC with random crop) on all environments from DeepMind Control Suite.
Revised: As shown in Table 2, SUNRISE also consistently improves the performance of RAD (i.e., SAC with random crop) on all environments from DeepMind Control Suite.

Original: This implies that the proposed method can be useful for high-dimensional and complex input observations.
Revised: This implies that the proposed method can be useful for high-dimensional and complex input observations.

Original: Moreover, our method outperforms existing pixel-based RL methods in almost all environments.
Revised: Moreover, our method outperforms existing pixel-based RL methods in almost all environments.

Original: We remark that SUNRISE can also be combined with DrQ, and expect that it can achieve better performances on Cartpole-swing and Cup-catch at 100K environment steps.
Revised: We remark that SUNRISE can also be combined with DrQ, and expect that it can achieve better performances on Cartpole-swing and Cup-catch at 100K environment steps.

Original: [Table]
Revised: [Table]

Original: A v e r age
Revised: A v e r age

Original: [Figure]
Revised: [Figure]

Original: Atari games .
Revised: Atari games .

Original: We also evaluate SUNRISE on discrete control tasks from the Atari benchmark using
Revised: We also evaluate SUNRISE on discrete control tasks from the Atari benchmark using

Original: Rainbow DQN.
Revised: Rainbow DQN.

Original: Table 3 shows that SUNRISE improves the performance of Rainbow in almost all environments
Revised: Table 3 shows that SUNRISE improves the performance of Rainbow in almost all environments

Original: [Figure]
Revised: [Figure]

Original: and outperforms the state-of-the-art CURL and SimPLe on 11 out of 26 Atari games.
Revised: and outperforms the state-of-the-art CURL and SimPLe on 11 out of 26 Atari games.

Original: Here, we remark that SUNRISE is also compatible with CURL, which could enable even better performance.
Revised: Here, we remark that SUNRISE is also compatible with CURL, which could enable even better performance.

Original: These results demonstrate that SUNRISE is a general approach
Revised: These results demonstrate that SUNRISE is a general approach

Original: [Figure]
Revised: [Figure]

Original: 5. A BLATION STUDY
Revised: 5. A BLATION STUDY

Original: Effects of weighted Bellman backups .
Revised: Effects of weighted Bellman backups .

Original: To verify the effectiveness of the proposed weighted Bellman backup (4) in improving signal-to-noise in Q-updates, we evaluate on a modiﬁed OpenAI Gym environments with noisy rewards.
Revised: To verify the effectiveness of the proposed weighted Bellman backup (4) in improving signal-to-noise in Q-updates, we evaluate on a modiﬁed OpenAI Gym environments with noisy rewards.

Original: Following Kumar et al.
Revised: Following Kumar et al.

Original: [Figure]
Revised: [Figure]

Original: we add Gaussian noise to the reward function: r (cid:48)
Revised: we add Gaussian noise to the reward function: r (cid:48)

Original: ( s, a ) =
Revised: ( s, a ) =

Original: r ( s, a ) + z, where z ∼ N (0 , 1) only during training, and report the deterministic ground-truth reward during evaluation.
Revised: r ( s, a ) + z, where z ∼ N (0 , 1) only during training, and report the deterministic ground-truth reward during evaluation.

Original: For our method, we also consider a variant of
Revised: For our method, we also consider a variant of

Original: SUNRISE, which updates Q-functions without the proposed weighted Bellman backup to isolate its effect.
Revised: SUNRISE, which updates Q-functions without the proposed weighted Bellman backup to isolate its effect.

Original: We compare to DisCor (Kumar et al., 2020), which improves SAC by reweighting the
Revised: We compare to DisCor (Kumar et al., 2020), which improves SAC by reweighting the

Original: Bellman backup based on estimated cumulative Bellman errors (see Appendix E for more details).
Revised: Bellman backup based on estimated cumulative Bellman errors (see Appendix E for more details).

Original: [Figure]
Revised: [Figure]

Original: Figure 2 shows the learning curves of all methods on OpenAI Gym with noisy rewards.
Revised: Figure 2 shows the learning curves of all methods on OpenAI Gym with noisy rewards.

Original: The proposed weighted Bellman backup signiﬁcantly improves both sample-efﬁciency and asymptotic performance of SUNRISE, and outperforms baselines such as SAC and DisCor.
Revised: The proposed weighted Bellman backup signiﬁcantly improves both sample-efﬁciency and asymptotic performance of SUNRISE, and outperforms baselines such as SAC and DisCor.

Original: One can note the performance gain due to our weighted Bellman backup becomes more signiﬁcant in complex environments, such as SlimHumanoid-ET.
Revised: One can note the performance gain due to our weighted Bellman backup becomes more signiﬁcant in complex environments, such as SlimHumanoid-ET.

Original: We remark that DisCor still suffers from error propagation issues in complex environments like SlimHumanoid-ET and Ant because there are some approximation errors in estimating cumulative Bellman errors (see Section 6.1 for more detailed discussion).
Revised: We remark that DisCor still suffers from error propagation issues in complex environments like SlimHumanoid-ET and Ant because there are some approximation errors in estimating cumulative Bellman errors (see Section 6.1 for more detailed discussion).

Original: These results imply that errors in the target Q-function can be characterized by the proposed conﬁdent weight in equation 5 effectively.
Revised: These results imply that errors in the target Q-function can be characterized by the proposed conﬁdent weight in equation 5 effectively.

Original: We also consider another variant of SUNRISE, which updates Q-functions with random weights sampled from [0 . 5 , 1 . 0] uniformly at random.
Revised: We also consider another variant of SUNRISE, which updates Q-functions with random weights sampled from [0 . 5 , 1 . 0] uniformly at random.

Original: In order to evaluate the performance of SUNRISE, we increase the noise rate by adding Gaussian noise with a large standard deviation to the reward function: r (cid:48)
Revised: In order to evaluate the performance of SUNRISE, we increase the noise rate by adding Gaussian noise with a large standard deviation to the reward function: r (cid:48)

Original: ( s, a ) =
Revised: ( s, a ) =

Original: r ( s, a ) + z, where z ∼ N (0 , 5) .
Revised: r ( s, a ) + z, where z ∼ N (0 , 5) .

Original: Figure 3(a) shows the learning curves of all methods on the SlimHumanoid-ET environment over 10 random seeds.
Revised: Figure 3(a) shows the learning curves of all methods on the SlimHumanoid-ET environment over 10 random seeds.

Original: We remark that SUNRISE with the proposed weighted Bellman backups (blue curve) outperforms all baselines including
Revised: We remark that SUNRISE with the proposed weighted Bellman backups (blue curve) outperforms all baselines including

Original: SUNRISE with random weight (red curve) and only with UCB exploration (green curve).
Revised: SUNRISE with random weight (red curve) and only with UCB exploration (green curve).

Original: This implies that the proposed weighted Bellman backups can handle the error propagation effectively even though there is a large noise in reward function.
Revised: This implies that the proposed weighted Bellman backups can handle the error propagation effectively even though there is a large noise in reward function.

Original: Effects of UCB exploration .
Revised: Effects of UCB exploration .

Original: To verify the advantage of UCB exploration in (6), we evaluate on Cartpole-swing with sparse-reward from DeepMind Control Suite.
Revised: To verify the advantage of UCB exploration in (6), we evaluate on Cartpole-swing with sparse-reward from DeepMind Control Suite.

Original: For our method, we consider a variant of SUNRISE, which selects action without UCB exploration.
Revised: For our method, we consider a variant of SUNRISE, which selects action without UCB exploration.

Original: As shown in Fig 3(b),
Revised: As shown in Fig 3(b),

Original: SUNRISE with UCB exploration (blue curve) signiﬁcantly improves the sample-efﬁciency on the environment with sparse rewards.
Revised: SUNRISE with UCB exploration (blue curve) signiﬁcantly improves the sample-efﬁciency on the environment with sparse rewards.

Original: Comparison with a single agent with more updates/parameters .
Revised: Comparison with a single agent with more updates/parameters .

Original: One concern in utilizing the ensemble method is that its gains may come from more gradient updates and parameters.
Revised: One concern in utilizing the ensemble method is that its gains may come from more gradient updates and parameters.

Original: To clarify this concern, we compare SUNRISE (5 ensembles using 2-layer MLPs with 256 hidden units each) to a single agent, which consists of 2-layer MLPs with 1024 (and 256) hidden units with 5 updates using different random minibatches.
Revised: To clarify this concern, we compare SUNRISE (5 ensembles using 2-layer MLPs with 256 hidden units each) to a single agent, which consists of 2-layer MLPs with 1024 (and 256) hidden units with 5 updates using different random minibatches.

Original: Figure 3(c) shows that the learning curves on SlimHumanoidET, where SUNRISE outperforms all baselines.
Revised: Figure 3(c) shows that the learning curves on SlimHumanoidET, where SUNRISE outperforms all baselines.

Original: This implies that the gains from SUNRISE can not be achieved by simply increasing the number of updates/parameters.
Revised: This implies that the gains from SUNRISE can not be achieved by simply increasing the number of updates/parameters.

Original: More experimental results on other environments are also available in Appendix D.
Revised: More experimental results on other environments are also available in Appendix D.

Original: Effects of ensemble size .
Revised: Effects of ensemble size .

Original: We analyze the effects of ensemble size N on the Ant environment from
Revised: We analyze the effects of ensemble size N on the Ant environment from

Original: OpenAI Gym.
Revised: OpenAI Gym.

Original: Figure 3(d) shows that the performance can be improved by increasing the ensemble size, but the improvement is saturated around N = 5 .
Revised: Figure 3(d) shows that the performance can be improved by increasing the ensemble size, but the improvement is saturated around N = 5 .

Original: Thus, we use ﬁve ensemble agents for all experiments.
Revised: Thus, we use ﬁve ensemble agents for all experiments.

Original: More experimental results on other environments are also available in Appendix D, where the overall trend is similar.
Revised: More experimental results on other environments are also available in Appendix D, where the overall trend is similar.

Original:  D ISCUSSION 6. C ONNECTION WITH D IS C OR
Revised:  D ISCUSSION 6. C ONNECTION WITH D IS C OR

Original: Kumar et al.
Revised: Kumar et al.

Original: (2020) show that naive Bellman backups can suffer from slow learning in certain environments, requiring exponentially many updates.
Revised: (2020) show that naive Bellman backups can suffer from slow learning in certain environments, requiring exponentially many updates.

Original: To handle this problem, they propose the weighted
Revised: To handle this problem, they propose the weighted

Original: Bellman backups, which make steady learning progress by inducing some optimal data distribution (see (Kumar et al., 2020) for more details).
Revised: Bellman backups, which make steady learning progress by inducing some optimal data distribution (see (Kumar et al., 2020) for more details).

Original: Speciﬁcally, in addition to a standard Q-learning, DisCor trains an error model ∆ ψ ( s, a ) , which approximates the cumulative sum of discounted Bellman errors over the past iterations of training.
Revised: Speciﬁcally, in addition to a standard Q-learning, DisCor trains an error model ∆ ψ ( s, a ) , which approximates the cumulative sum of discounted Bellman errors over the past iterations of training.

Original: Then, using the error model, DisCor reweights the Bellman backups based on a conﬁdence weight deﬁned as follows:
Revised: Then, using the error model, DisCor reweights the Bellman backups based on a conﬁdence weight deﬁned as follows:

Original: w ( s
Revised: w ( s

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , where γ is a discount factor and T is a temperature.
Revised: (cid:17) , where γ is a discount factor and T is a temperature.

Original: However, we remark that DisCor can still suffer from the error propagation issues because there is also an approximation error in estimating cumulative Bellman errors.
Revised: However, we remark that DisCor can still suffer from the error propagation issues because there is also an approximation error in estimating cumulative Bellman errors.

Original: Therefore, we consider an alternative approach to utilize the uncertainty from ensembles.
Revised: Therefore, we consider an alternative approach that utilizes the uncertainty from ensembles.

Original: Because it has been observed that the ensemble can produce well-calibrated uncertainty estimates (i.e., variance) on unseen samples (Lakshminarayanan et al., 2017), we expect that the weighted Bellman backups based on ensembles can handle error propagation more effectively.
Revised: Because it has been observed that the ensemble can produce well-calibrated uncertainty estimates (i.e., variance) on unseen samples (Lakshminarayanan et al., 2017), we expect that the weighted Bellman backups based on ensembles can handle error propagation more effectively.

Original: Indeed, in our experiments, we ﬁnd that ensemble-based weighted Bellman backups can give rise to more stable training and improve the data-efﬁciency of various off-policy RL algorithms.
Revised: Indeed, in our experiments, we ﬁnd that ensemble-based weighted Bellman backups can give rise to more stable training and improve the data-efﬁciency of various off-policy RL algorithms.

Original: 6. C OMPUTATION OVERHEAD
Revised: 6. C OMPUTATION OVERHEAD

Original: One can expect that there is an additional computation overhead by introducing ensembles.
Revised: One can expect that there is an additional computation overhead by introducing ensembles.

Original: When we have N ensemble agents, our method requires N × inferences for weighted Bellman backups and 2 N × inferences ( N for actors and N for critics).
Revised: When we have N ensemble agents, our method requires N × inferences for weighted Bellman backups and 2 N × inferences ( N for actors and N for critics).

Original: However, we remark that our method can be more computationally efﬁcient because it is parallelizable.
Revised: However, we remark that our method can be more computationally efﬁcient because it is parallelizable.

Original: Also, as shown in Figure 3(c), the gains from SUNRISE can not be achieved by simply increasing the number of updates/parameters.
Revised: Also, as shown in Figure 3(c), the gains from SUNRISE can not be achieved by simply increasing the number of updates/parameters.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present the ensemble-based weighted Bellman backups, which is compatible with various off-policy RL algorithms.
Revised: In this paper, we present the ensemble-based weighted Bellman backups, which is compatible with various off-policy RL algorithms.

Original: By re-weighting target Q-values based on uncertainty estimates, we stabilize and improve the learning process on both continuous and discrete control benchmarks.
Revised: By re-weighting target Q-values based on uncertainty estimates, we stabilize and improve the learning process on both continuous and discrete control benchmarks.

Original: Additionally, we introduce SUNRISE, a simple uniﬁed ensemble method, which integrates the proposed weighted Bellman backups with bootstrap with random initialization, and UCB exploration to handle various issues in off-policy RL algorithms.
Revised: Additionally, we introduce SUNRISE, a simple uniﬁed ensemble method, which integrates the proposed weighted Bellman backups with bootstrap with random initialization, and UCB exploration to handle various issues in off-policy RL algorithms.

Original: Our experiments show that SUNRISE consistently improves the performances of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, and outperforms state-of-the-art RL algorithms for both continuous and discrete control tasks on both low-dimensional and high-dimensional environments.
Revised: Our experiments show that SUNRISE consistently improves the performances of existing off-policy RL algorithms, such as Soft Actor-Critic and Rainbow DQN, and outperforms state-of-the-art RL algorithms for both continuous and discrete control tasks on both low-dimensional and high-dimensional environments.

Original: We hope that
Revised: We hope that

Original: SUNRISE could be useful to other relevant topics such as sim-to-real transfer (Tobin et al., 2017), imitation learning (Torabi et al., 2018), understanding the connection between on-policy and offpolicy RL (Schulman et al., 2017), ofﬂine RL (Agarwal et al., 2020), and planning (Srinivas et al., 2018; Tamar et al., 2016).
Revised: SUNRISE could be useful to other relevant topics such as sim-to-real transfer (Tobin et al., 2017), imitation learning (Torabi et al., 2018), understanding the connection between on-policy and offpolicy RL (Schulman et al., 2017), ofﬂine RL (Agarwal et al., 2020), and planning (Srinivas et al., 2018; Tamar et al., 2016).

Original: Appendix A SUNRISE: S OFT ACTOR - CRITIC
Revised: Appendix A SUNRISE: S OFT ACTOR - CRITIC

Original: Background .
Revised: Background .

Original: SAC (Haarnoja et al., 2018) is a state-of-the-art off-policy algorithm for continuous control problems.
Revised: SAC (Haarnoja et al., 2018) is a state-of-the-art off-policy algorithm for continuous control problems.

Original: SAC learns a policy, π φ ( a | s ) , and a critic, Q θ ( s, a ) , and aims to maximize a weighted objective of the reward and the policy entropy, E s t ,
Revised: SAC learns a policy, π φ ( a | s ) , and a critic, Q θ ( s, a ) , and aims to maximize a weighted objective of the reward and the policy entropy, E s t ,

Original: a t ∼ π (cid:2)(cid:80) t γ t − 1 r t + α H
Revised: a t ∼ π (cid:2)(cid:80) t γ t − 1 r t + α H

Original: ( π φ ( ·| s t ))
Revised: ( π φ ( ·| s t ))

Original: (cid:3) .
Revised: (cid:3) .

Original: To update the parameters, SAC alternates between a soft policy evaluation and a soft policy improvement.
Revised: To update the parameters, SAC alternates between a soft policy evaluation and a soft policy improvement.

Original: At the soft policy evaluation step, a soft Q-function, which is modeled as a neural network with parameters θ , is updated by minimizing the following soft Bellman residual:
Revised: At the soft policy evaluation step, a soft Q-function, which is modeled as a neural network with parameters θ , is updated by minimizing the following soft Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: where τ t =
Revised: where τ t =

Original: ( s t , a t , r t , s t +1 ) is a transition, B is a replay buffer, ¯ θ are the delayed parameters, and α is a temperature parameter.
Revised: ( s t , a t , r t , s t +1 ) is a transition, B is a replay buffer, ¯ θ are the delayed parameters, and α is a temperature parameter.

Original: At the soft policy improvement step, the policy π with its parameter φ is updated by minimizing the following objective:
Revised: At the soft policy improvement step, the policy π with its parameter φ is updated by minimizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: We remark that this corresponds to minimizing the Kullback-Leibler divergence between the policy and a Boltzmann distribution induced by the current soft Q-function.
Revised: We remark that this corresponds to minimizing the Kullback-Leibler divergence between the policy and a Boltzmann distribution induced by the current soft Q-function.

Original: SUNRISE without UCB exploration .
Revised: SUNRISE without UCB exploration .

Original: For SUNRISE without UCB exploration, we use random inference proposed in Bootstrapped DQN (Osband et al., 2016a), which randomly selects an index of policy uniformly at random and generates the action from the selected actor for the duration of that episode (see Line 3 in Algorithm 2).
Revised: For SUNRISE without UCB exploration, we use random inference proposed in Bootstrapped DQN (Osband et al., 2016a), which randomly selects an index of policy uniformly at random and generates the action from the selected actor for the duration of that episode (see Line 3 in Algorithm 2).

Original: 1: for each iteration do 2: //
Revised: 1: for each iteration do 2: //

Original: R ANDOM INFERENCE 3:
Revised: R ANDOM INFERENCE 3:

Original: Select an index of policy using (cid:98) i ∼ Uniform { 1 , · · · , N } 4: for each timestep t do 5:
Revised: Select an index of policy using (cid:98) i ∼ Uniform { 1 , · · · , N } 4: for each timestep t do 5:

Original: Get the action from selected policy: a t ∼ π φ (cid:98) i ( a | s t ) 6:
Revised: Get the action from selected policy: a t ∼ π φ (cid:98) i ( a | s t ) 6:

Original: Collect state s t +1 and reward r t from the environment by taking action a t 7:
Revised: Collect state s t +1 and reward r t from the environment by taking action a t 7:

Original: Sample bootstrap masks M t = { m t,i ∼ Bernoulli ( β ) — i ∈ { 1 , . . .
Revised: Sample bootstrap masks M t = { m t,i ∼ Bernoulli ( β ) — i ∈ { 1 , . . .

Original: , N }} 8:
Revised: , N }} 8:

Original: Store transitions τ t = ( s t , a t , s t +1 , r t ) and masks in replay buffer B ← B ∪ { ( τ t , M t ) } 9: end for 10: // U PDATE AGENTS
Revised: Store transitions τ t = ( s t , a t , s t +1 , r t ) and masks in replay buffer B ← B ∪ { ( τ t , M t ) } 9: end for 10: // U PDATE AGENTS

Original: VIA BOOTSTRAP AND WEIGHTED B ELLMAN BACKUP 11: for each gradient step do 12:
Revised: VIA BOOTSTRAP AND WEIGHTED B ELLMAN BACKUP 11: for each gradient step do 12:

Original: Sample random minibatch { ( τ j , M j ) } Bj =1 ∼ B 13: for each agent i do 14:
Revised: Sample random minibatch { ( τ j , M j ) } Bj =1 ∼ B 13: for each agent i do 14:

Original: Update the Q-function by minimizing 1 B (cid:80) Bj =

Revised: Update the Q-function by minimizing 1 B (cid:80) Bj =

Original: m j,i L WQ ( τ j , θ i ) 15:
Revised: m j,i L WQ ( τ j , θ i ) 15:

Original: Update the policy by minimizing 1 B (cid:80) Bj =

Revised: Update the policy by minimizing 1 B (cid:80) Bj =

Original: m j,i L π ( s j , φ i ) 16: end for 17: end for 18: end for
Revised: m j,i L π ( s j , φ i ) 16: end for 17: end for 18: end for

Original: B E XTENSION TO R AINBOW DQN B. P RELIMINARIES : R AINBOW DQN
Revised: B E XTENSION TO R AINBOW DQN B. P RELIMINARIES : R AINBOW DQN

Original: Background .
Revised: Background .

Original: DQN algorithm (Mnih et al., 2015) learns a Q-function, which is modeled as a neural network with parameters θ , by minimizing the following Bellman residual:
Revised: DQN algorithm (Mnih et al., 2015) learns a Q-function, which is modeled as a neural network with parameters θ , by minimizing the following Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: where τ t =
Revised: where τ t =

Original: ( s t , a t , r t , s t +1 ) is a transition, B is a replay buffer, and ¯ θ are the delayed parameters.
Revised: ( s t , a t , r t , s t +1 ) is a transition, B is a replay buffer, and ¯ θ are the delayed parameters.

Original: Even though Rainbow DQN integrates several techniques, such as double Q-learning (Van Hasselt et al., 2016) and distributional DQN (Bellemare et al., 2017), applying SUNRISE to Rainbow DQN can be described based on the standard DQN algorithm.
Revised: Even though Rainbow DQN integrates several techniques, such as double Q-learning (Van Hasselt et al., 2016) and distributional DQN (Bellemare et al., 2017), applying SUNRISE to Rainbow DQN can be described based on the standard DQN algorithm.

Original: For exposition, we refer the reader to Hessel et al.
Revised: For exposition, we refer the reader to Hessel et al.

Original: (2018) for more detailed explanations of Rainbow DQN.
Revised: (2018) for more detailed explanations of Rainbow DQN.

Original: 1: for each iteration do 2: for each timestep t do 3: //
Revised: 1: for each iteration do 2: for each timestep t do 3: //

Original: UCB EXPLORATION 4:
Revised: UCB EXPLORATION 4:

Original: Choose the action that maximizes UCB: a t = arg max a t,i ∈A Q mean ( s t , a t,i )+ λQ std ( s t , a t,i ) 5:
Revised: Choose the action that maximizes UCB: a t = arg max a t,i ∈A Q mean ( s t , a t,i )+ λQ std ( s t , a t,i ) 5:

Original: Collect state s t +1 and reward r t from the environment by taking action a t 6:
Revised: Collect state s t +1 and reward r t from the environment by taking action a t 6:

Original: Sample bootstrap masks M t = { m t,i ∼ Bernoulli ( β ) — i ∈ { 1 , . . .
Revised: Sample bootstrap masks M t = { m t,i ∼ Bernoulli ( β ) — i ∈ { 1 , . . .

Original: , N }} 7:
Revised: , N }} 7:

Original: Store transitions τ t = ( s t , a t , s t +1 , r t ) and masks in replay buffer B ← B ∪ { ( τ t , M t ) } 8: end for 9: // U
Revised: Store transitions τ t = ( s t , a t , s t +1 , r t ) and masks in replay buffer B ← B ∪ { ( τ t , M t ) } 8: end for 9: // U

Original: Q-
Revised: Q-

Original: 10: for each gradient step do 11:
Revised: 10: for each gradient step do 11:

Original: Sample random minibatch { ( τ j , M j ) } Bj =1 ∼ B 12: for each agent i do 13:
Revised: Sample random minibatch { ( τ j , M j ) } Bj =1 ∼ B 12: for each agent i do 13:

Original: Update the Q-function by minimizing 1 B (cid:80) Bj =

Revised: Update the Q-function by minimizing 1 B (cid:80) Bj =

Original: m j,i L DQN WQ ( τ j , θ i ) 14: end for 15: end for 16: end for
Revised: m j,i L DQN WQ ( τ j , θ i ) 14: end for 15: end for 16: end for

Original: Bootstrap with random initialization .
Revised: Bootstrap with random initialization .

Original: Formally, we consider an ensemble of N Q-functions, i.e., { Q θ i } Ni =1 , where θ i denotes the parameters of the i -th Q-function.
Revised: Formally, we consider an ensemble of N Q-functions, i.e., { Q θ i } Ni =1 , where θ i denotes the parameters of the i -th Q-function.

Original: 3 To train the ensemble of
Revised: 3 To train the ensemble of

Original: Q-functions, we use the bootstrap with random initialization (Efron, 1982; Osband et al., 2016a), which enforces the diversity between Q-functions through two simple ideas: First, we initialize the model parameters of all Q-functions with random parameter values for inducing an initial diversity in the models.
Revised: Q-functions, we use the bootstrap with random initialization (Efron, 1982; Osband et al., 2016a), which enforces the diversity between Q-functions through two simple ideas: First, we initialize the model parameters of all Q-functions with random parameter values for inducing an initial diversity in the models.

Original: Second, we apply different samples to train each Q-function.
Revised: Second, we apply different samples to train each Q-function.

Original: Speciﬁcally, for each
Revised: Speciﬁcally, for each

Original: Q-function i in each timestep t , we draw the binary masks m t,i from the Bernoulli distribution with parameter β ∈ (0 , 1] , and store them in the replay buffer.
Revised: Q-function i in each timestep t , we draw the binary masks m t,i from the Bernoulli distribution with parameter β ∈ (0 , 1] , and store them in the replay buffer.

Original: Then, when updating the model parameters of Q-functions, we multiply the bootstrap mask to each objective function.
Revised: Then, when updating the model parameters of Q-functions, we multiply the bootstrap mask to each objective function.

Original: Weighted Bellman backup .
Revised: Weighted Bellman backup .

Original: Since conventional Q-learning is based on the Bellman backup in equation 7, it can be affected by error propagation.
Revised: Since conventional Q-learning is based on the Bellman backup in equation 7, it can be affected by error propagation.

Original: I.e., error in the target Q-function Q ¯ θ ( s t +1 , a t +1 ) gets propagated into the Q-function Q
Revised: I.e., error in the target Q-function Q ¯ θ ( s t +1 , a t +1 ) gets propagated into the Q-function Q

Original: θ ( s t , a t ) at the current state.
Revised: θ ( s t , a t ) at the current state.

Original: Recently, Kumar et al.
Revised: Recently, Kumar et al.

Original: (2020) showed that this error propagation can cause inconsistency and unstable convergence.
Revised: (2020) showed that this error propagation can cause inconsistency and unstable convergence.

Original: To mitigate this issue, for each Q-function i , we consider a weighted Bellman backup as follows:
Revised: To mitigate this issue, for each Q-function i , we consider a weighted Bellman backup as follows:

Original: [Equation]
Revised: [Equation]

Original: where τ t =
Revised: where τ t =

Original: ( s t , a t , r t , s t +1 ) is a transition, and w ( s ) is a conﬁdence weight based on ensemble of target Q-functions:
Revised: ( s t , a t , r t , s t +1 ) is a transition, and w ( s ) is a conﬁdence weight based on ensemble of target Q-functions:

Original: [Equation]
Revised: [Equation]

Original: where T > 0 is a temperature, σ is the sigmoid function, and ¯ Q std ( s ) is the empirical standard deviation of all target Q-functions { max a Q ¯ θ i ( s, a ) } N i =1 .
Revised: where T > 0 is a temperature, σ is the sigmoid function, and ¯ Q std ( s ) is the empirical standard deviation of all target Q-functions { max a Q ¯ θ i ( s, a ) } N i =1 .

Original: Note that the conﬁdence weight is bounded in [0 . 5 , 1 . 0] because standard deviation is always positive.
Revised: Note that the conﬁdence weight is bounded in [0 . 5 , 1 . 0] because standard deviation is always positive.

Original: The proposed objective L DQN WQ downweights the sample transitions with high variance across target Q-functions, resulting in a loss function for the Q-updates that has a better signal-to-noise ratio.
Revised: The proposed objective L DQN WQ downweights the sample transitions with high variance across target Q-functions, resulting in a loss function for the Q-updates that has a better signal-to-noise ratio.

Original: Note that we combine the proposed
Revised: Note that we combine the proposed

Original: UCB exploration .
Revised: UCB exploration .

Original: The ensemble can also be leveraged for efﬁcient exploration (Chen et al., 2017;
Revised: The ensemble can also be leveraged for efﬁcient exploration (Chen et al., 2017;

Original: Osband et al., 2016a) because it can express higher uncertainty on unseen samples.
Revised: Osband et al., 2016a) because it can express higher uncertainty on unseen samples.

Original: Motivated by this, by following the idea of Chen et al.
Revised: Motivated by this, by following the idea of Chen et al.

Original: (2017), we consider an optimism-based exploration that chooses the action that maximizes
Revised: (2017), we consider an optimism-based exploration that chooses the action that maximizes

Original: [Equation]
Revised: [Equation]

Original: where Q mean ( s, a ) and Q std ( s, a ) are the empirical mean and standard deviation of all Q-functions { Q θ i } Ni =1 , and the λ > 0 is a hyperparameter.
Revised: where Q mean ( s, a ) and Q std ( s, a ) are the empirical mean and standard deviation of all Q-functions { Q θ i } Ni =1 , and the λ > 0 is a hyperparameter.

Original: This inference method can encourage exploration by adding an exploration bonus (i.e., standard deviation Q std ) for visiting unseen state-action pairs similar to the UCB algorithm (Auer et al., 2002).
Revised: This inference method can encourage exploration by adding an exploration bonus (i.e., standard deviation Q std ) for visiting unseen state-action pairs similar to the UCB algorithm (Auer et al., 2002).

Original: This inference method was originally proposed in Chen et al.
Revised: This inference method was originally proposed in Chen et al.

Original: (2017) for efﬁcient exploration in DQN,
Revised: (2017) for efﬁcient exploration in DQN,

Original: but we further extend it to Rainbow DQN.
Revised: but we further extend it to Rainbow DQN.

Original: For evaluation, we approximate the maximum a posterior action by choosing the action maximizes the mean of Q-functions, i.e., a t = max a { Q mean ( s t , a ) } .
Revised: For evaluation, we approximate the maximum a posterior action by choosing the action maximizes the mean of Q-functions, i.e., a t = max a { Q mean ( s t , a ) } .

Original: The full procedure is summarized in
Revised: The full procedure is summarized in

Original: Algorithm 3.
Revised: Algorithm 3.

Original: C I MPLEMENTATION DETAILS FOR TOY REGRESSION TASKS
Revised: C I MPLEMENTATION DETAILS FOR TOY REGRESSION TASKS

Original: We evaluate the quality of uncertainty estimates from an ensemble of neural networks on a toy regression task.
Revised: We evaluate the quality of uncertainty estimates from an ensemble of neural networks on a toy regression task.

Original: To this end, we generate twenty training samples drawn as y
Revised: To this end, we generate twenty training samples drawn as y

Original: = x 3 + (cid:15) , where (cid:15) ∼ N (0 , 3 2 ) , and train ten ensembles of regression networks using bootstrap with random initialization.
Revised: = x 3 + (cid:15) , where (cid:15) ∼ N (0 , 3 2 ) , and train ten ensembles of regression networks using bootstrap with random initialization.

Original: The regression network is as fully-connected neural networks with 2 hidden layers and 50 rectiﬁed linear units in each layer.
Revised: The regression network is as fully-connected neural networks with 2 hidden layers and 50 rectiﬁed linear units in each layer.

Original: For bootstrap, we draw the binary masks from the Bernoulli distribution with mean β = 0 . 3 .
Revised: For bootstrap, we draw the binary masks from the Bernoulli distribution with mean β = 0 . 3 .

Original: As uncertainty estimates, we measure the empirical variance of the networks’ predictions.
Revised: As uncertainty estimates, we measure the empirical variance of the networks’ predictions.

Original: As shown in Figure 1(b), the ensemble can produce well-calibrated uncertainty estimates (i.e., variance) on unseen samples.
Revised: As shown in Figure 1(b), the ensemble can produce well-calibrated uncertainty estimates (i.e., variance) on unseen samples.

Original: D E XPERIMENTAL SETUPS AND RESULTS : O PEN AI G YM
Revised: D E XPERIMENTAL SETUPS AND RESULTS : O PEN AI G YM

Original: Environments .
Revised: Environments .

Original: We evaluate the performance of SUNRISE on four complex environments based on the standard bench-marking environments 5 from OpenAI Gym (Brockman et al., 2016).
Revised: We evaluate the performance of SUNRISE on four complex environments based on the standard bench-marking environments 5 from OpenAI Gym (Brockman et al., 2016).

Original: Note that we do not use a modiﬁed Cheetah environments from PETS (Chua et al., 2018)
Revised: Note that we do not use a modiﬁed Cheetah environments from PETS (Chua et al., 2018)

Original: (dented as Cheetah in POPLIN (Wang & Ba, 2020)) because it includes additional information in observations.
Revised: (dented as Cheetah in POPLIN (Wang & Ba, 2020)) because it includes additional information in observations.

Original: Training details .
Revised: Training details .

Original: We consider a combination of SAC and SUNRISE using the publicly released implementation repository ( https://github.com/vitchyr/rlkit ) without any modiﬁcations on hyperparameters and architectures.
Revised: We consider a combination of SAC and SUNRISE using the publicly released implementation repository ( https://github.com/vitchyr/rlkit ) without any modiﬁcations on hyperparameters and architectures.

Original: For our method, the temperature for weighted Bellman backups is chosen from T ∈ { 10 , 20 , 50 } , the mean of the Bernoulli distribution is chosen from β ∈ { 0 . 5 , 1 . 0 } , the penalty parameter is chosen from λ ∈ { 1 , 5 , 10 } , and we train ﬁve ensemble agents.
Revised: For our method, the temperature for weighted Bellman backups is chosen from T ∈ { 10 , 20 , 50 } , the mean of the Bernoulli distribution is chosen from β ∈ { 0 . 5 , 1 . 0 } , the penalty parameter is chosen from λ ∈ { 1 , 5 , 10 } , and we train ﬁve ensemble agents.

Original: The optimal parameters are chosen to achieve the best performance on training environments.
Revised: The optimal parameters are chosen to achieve the best performance on training environments.

Original: Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., β = 1 ) usually achieves the best performance in most cases similar to
Revised: Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., β = 1 ) usually achieves the best performance in most cases similar to

Original: Osband et al.
Revised: Osband et al.

Original: (2016a)
Revised: (2016a)

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: We expect that this is because splitting samples can reduce the sample-efﬁciency.
Revised: We expect that this is because splitting samples can reduce the sample-efﬁciency.

Original: Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.
Revised: Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.

Original: Learning curves .
Revised: Learning curves .

Original: Figure 4 shows the learning curves on all environments.
Revised: Figure 4 shows the learning curves on all environments.

Original: One can note that
Revised: One can note that

Original: SUNRISE consistently improves the performance of SAC by a large margin.
Revised: SUNRISE consistently improves the performance of SAC by a large margin.

Original: Effects of ensembles .
Revised: Effects of ensembles .

Original: Figure 5 shows the learning curves of SUNRISE with varying values of ensemble size on all environments.
Revised: Figure 5 shows the learning curves of SUNRISE with varying values of ensemble size on all environments.

Original: The performance can be improved by increasing the ensemble size, but the improvement is saturated around N = 5 .
Revised: The performance can be improved by increasing the ensemble size, but the improvement is saturated around N = 5 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E E XPERIMENTAL SETUPS AND RESULTS : NOISY REWARD
Revised: E E XPERIMENTAL SETUPS AND RESULTS : NOISY REWARD

Original: DisCor .
Revised: DisCor .

Original: DisCor (Kumar et al., 2020) was proposed to prevent the error propagation issue in Qlearning.
Revised: DisCor (Kumar et al., 2020) was proposed to prevent the error propagation issue in Qlearning.

Original: In addition to a standard Q-learning, DisCor trains an error model ∆ ψ ( s, a ) , which approximates the cumulative sum of discounted Bellman errors over the past iterations of training.
Revised: In addition to a standard Q-learning, DisCor trains an error model ∆ ψ ( s, a ) , which approximates the cumulative sum of discounted Bellman errors over the past iterations of training.

Original: Then, using the error model, DisCor reweights the Bellman backups based on a conﬁdence weight deﬁned as follows:
Revised: Then, using the error model, DisCor reweights the Bellman backups based on a conﬁdence weight deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where γ is a discount factor and T is a temperature.
Revised: where γ is a discount factor and T is a temperature.

Original: By following the setups in Kumar et al.
Revised: By following the setups in Kumar et al.

Original: (2020), we take a network with 1 extra hidden layer than the corresponding Q-network as an error model, and chose T = 10 for all experiments.
Revised: (2020), we take a network with 1 extra hidden layer than the corresponding Q-network as an error model, and chose T = 10 for all experiments.

Original: We update the temperature via a moving average and use the learning rate of 0 . 0003 .
Revised: We update the temperature via a moving average and use the learning rate of 0 . 0003 .

Original: We use the SAC algorithm as the RL objective coupled with DisCor and build on top of the publicly released implementation repository ( https://github.com/ vitchyr/rlkit ).
Revised: We use the SAC algorithm as the RL objective coupled with DisCor and build on top of the publicly released implementation repository ( https://github.com/ vitchyr/rlkit ).

Original: [Table]
Revised: [Table]

Original: F E XPERIMENTAL SETUPS AND RESULTS : D EEP M IND C ONTROL S UITE
Revised: F E XPERIMENTAL SETUPS AND RESULTS : D EEP M IND C ONTROL S UITE

Original: Training details .
Revised: Training details .

Original: We consider a combination of RAD and SUNRISE using the publicly released implementation repository ( https://github.com/MishaLaskin/rad ) with a full list of hyperparameters in Table 4.
Revised: We consider a combination of RAD and SUNRISE using the publicly released implementation repository ( https://github.com/MishaLaskin/rad ) with a full list of hyperparameters in Table 4.

Original: Similar to Laskin et al.
Revised: Similar to Laskin et al.

Original: (2020), we use the same encoder architecture as in (Yarats et al., 2019)
Revised: (2020), we use the same encoder architecture as in (Yarats et al., 2019)

Original: [Table]
Revised: [Table]

Original: and the actor and critic share the same encoder to embed image observations.
Revised: and the actor and critic share the same encoder to embed image observations.

Original: For our method, the temperature for weighted Bellman backups is chosen from T ∈ { 10 , 100 } , the mean of the Bernoulli distribution is chosen from β ∈ { 0 . 5 , 1 . 0 } , the penalty parameter is chosen from λ ∈ { 1 , 5 , 10 }
Revised: For our method, the temperature for weighted Bellman backups is chosen from T ∈ { 10 , 100 } , the mean of the Bernoulli distribution is chosen from β ∈ { 0 . 5 , 1 . 0 } , the penalty parameter is chosen from λ ∈ { 1 , 5 , 10 }

Original: [Table]
Revised: [Table]

Original: and we train ﬁve ensemble agents.
Revised: and we train ﬁve ensemble agents.

Original: The optimal parameters are chosen to achieve the best performance on training environments.
Revised: The optimal parameters are chosen to achieve the best performance on training environments.

Original: Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., β = 1 ) usually achieves the best performance in most cases similar to Osband et al.
Revised: Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., β = 1 ) usually achieves the best performance in most cases similar to Osband et al.

Original: (2016a)
Revised: (2016a)

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: We expect that this is because training samples can reduce the sample-efﬁciency.
Revised: We expect that this is because training samples can reduce the sample-efﬁciency.

Original: Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.
Revised: Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.

Original: Learning curves .
Revised: Learning curves .

Original: Figure 6(g), 6(h), 6(i), 6(j), 6(k), and 6(l) show the learning curves on all environments.
Revised: Figure 6(g), 6(h), 6(i), 6(j), 6(k), and 6(l) show the learning curves on all environments.

Original: Since RAD already achieves the near optimal performances and the room for improvement is small, we can see a small but consistent gains from SUNRISE.
Revised: Since RAD already achieves the near optimal performances and the room for improvement is small, we can see a small but consistent gains from SUNRISE.

Original: To verify the effectiveness of SUNRISE more clearly, we consider a combination of SAC and SUNRISE in Figure 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f), where the gain from SUNRISE is more signiﬁcant.
Revised: To verify the effectiveness of SUNRISE more clearly, we consider a combination of SAC and SUNRISE in Figure 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f), where the gain from SUNRISE is more signiﬁcant.

Original: G E XPERIMENTAL SETUPS AND RESULTS : A TARI GAMES
Revised: G E XPERIMENTAL SETUPS AND RESULTS : A TARI GAMES

Original: Training details .
Revised: Training details .

Original: We consider a combination of sample-efﬁcient versions of Rainbow DQN and SUNRISE using the publicly released implementation repository ( https://github.com/
Revised: We consider a combination of sample-efﬁcient versions of Rainbow DQN and SUNRISE using the publicly released implementation repository ( https://github.com/

Original: Kaixhin/Rainbow ) without any modiﬁcations on hyperparameters and architectures.
Revised: Kaixhin/Rainbow ) without any modiﬁcations on hyperparameters and architectures.

Original: For our method, the temperature for weighted Bellman backups is chosen from T ∈ { 10 , 40 } , the mean of the Bernoulli distribution is chosen from β ∈ { 0 . 5 , 1 . 0 } , the penalty parameter is chosen from λ ∈ { 1 , 10 } , and we train ﬁve ensemble agents.
Revised: For our method, the temperature for weighted Bellman backups is chosen from T ∈ { 10 , 40 } , the mean of the Bernoulli distribution is chosen from β ∈ { 0 . 5 , 1 . 0 } , the penalty parameter is chosen from λ ∈ { 1 , 10 } , and we train ﬁve ensemble agents.

Original: The optimal parameters are chosen to achieve the best performance on training environments.
Revised: The optimal parameters are chosen to achieve the best performance on training environments.

Original: Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., β = 1 ) usually achieves the best performance in most cases similar to Osband et al.
Revised: Here, we remark that training ensemble agents using same training samples but with different initialization (i.e., β = 1 ) usually achieves the best performance in most cases similar to Osband et al.

Original: (2016a)
Revised: (2016a)

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: We expect that this is because splitting samples can reduce the sample-efﬁciency.
Revised: We expect that this is because splitting samples can reduce the sample-efﬁciency.

Original: Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.
Revised: Also, initial diversity from random initialization can be enough because each Q-function has a unique target Q-function, i.e., target value is also different according to initialization.

Original: [Figure]
Revised: [Figure]

Original: RADSUNRISE
Revised: RADSUNRISE

Original: [Figure]
Revised: [Figure]

Original: Learning curves .
Revised: Learning curves .

Original: Figure 7, Figure 8 and Figure 9 show the learning curves on all environments.
Revised: Figure 7, Figure 8 and Figure 9 show the learning curves on all environments.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]