Original: M ASTERING V ISUAL C ONTINUOUS C ONTROL : I MPROVED D ATA -A UGMENTED R EINFORCEMENT
Revised: M ASTERING V ISUAL C ONTINUOUS C ONTROL : I MPROVED D ATA -A UGMENTED R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control.
Revised: We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control.

Original: DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels.
Revised: DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels.

Original: We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite.
Revised: We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite.

Original: Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL.
Revised: Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL.

Original: DrQ-v2 is conceptually simple, easy to implement and provides signiﬁcantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU.
Revised: DrQ-v2 is conceptually simple, easy to implement and provides signiﬁcantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU.

Original: Finally, DrQ-v2’s implementation is publicly released to provide RL practitioners with a strong and computationally efﬁcient baseline
Revised: Finally, DrQ-v2’s implementation is publicly released to provide RL practitioners with a strong and computationally efﬁcient baseline

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Creating sample-efﬁcient continuous control methods that observe high-dimensional images has been a long standing challenge in reinforcement learning (RL) .
Revised: Creating sample-efﬁcient continuous control methods that observe high-dimensional images has been a long standing challenge in reinforcement learning (RL) .

Original: Over the last three years, the RL community has made signiﬁcant headway on this problem, improving sample-efﬁciency signiﬁcantly.
Revised: Over the last three years, the RL community has made signiﬁcant headway on this problem, improving sample-efﬁciency signiﬁcantly.

Original: The key insight to solving visual control is the learning of better low-dimensional representations, either through autoencoders (Yarats et al., 2019;
Revised: The key insight to solving visual control is the learning of better low-dimensional representations, either through autoencoders (Yarats et al., 2019;

Original: Finn et al., 2015), variational inference (Hafner et al., 2018; 2019; Lee et al., 2019), contrastive learning (Srinivas et al., 2020; Yarats et al., 2021a), self-prediction (Schwarzer et al., 2020b), or data augmentations (Yarats et al., 2021b;
Revised: Finn et al., 2015), variational inference (Hafner et al., 2018; 2019; Lee et al., 2019), contrastive learning (Srinivas et al., 2020; Yarats et al., 2021a), self-prediction (Schwarzer et al., 2020b), or data augmentations (Yarats et al., 2021b;

Original: Laskin et al., 2020).
Revised: Laskin et al., 2020).

Original: However, current state-of-the-art model-free methods are still limited in three ways.
Revised: However, current state-of-the-art model-free methods are still limited in three ways.

Original: First, they are unable to solve the more challenging visual control problems such as quadruped and humanoid locomotion.
Revised: First, they are unable to solve the more challenging visual control problems such as quadruped and humanoid locomotion.

Original: Second, they often require signiﬁcant computational resources, i e. lengthy training times using distributed multi-GPU infrastructure.
Revised: Second, they often require signiﬁcant computational resources, i e. lengthy training times using distributed multi-GPU infrastructure.

Original: Lastly, it is often unclear how different design choices affect overall system performance.
Revised: Lastly, it is often unclear how different design choices affect overall system performance.

Original: [Figure]
Revised: [Figure]

Original: In this paper we present DrQ-v2, a simple model-free algorithm that builds on the idea of using data augmentations (Yarats et al., 2021b;
Revised: In this paper we present DrQ-v2, a simple model-free algorithm that builds on the idea of using data augmentations (Yarats et al., 2021b;

Original: Laskin et al., 2020) to solve hard visual control problems.
Revised: Laskin et al., 2020) to solve hard visual control problems.

Original: Most notably, it is the ﬁrst model-free method that solves complex humanoid tasks directly from pixels.
Revised: Most notably, it is the ﬁrst model-free method that solves complex humanoid tasks directly from pixels.

Original: Compared to previous state-of-the-art model-free methods, DrQ-v2 provides signiﬁcant improvements in sample efﬁciency across tasks from the DeepMind Control Suite (Tassa et al., 2018).
Revised: Compared to previous state-of-the-art model-free methods, DrQ-v2 provides signiﬁcant improvements in sample efﬁciency across tasks from the DeepMind Control Suite (Tassa et al., 2018).

Original: Conceptually simple, DrQ-v2 is also computationally efﬁcient, which allows solving most tasks in
Revised: Conceptually simple, DrQ-v2 is also computationally efﬁcient, which allows solving most tasks in

Original: DeepMind Control Suite in just 8 hours on a single GPU (see Figure 1).
Revised: DeepMind Control Suite in just 8 hours on a single GPU (see Figure 1).

Original: Recently, a model-based method, DreamerV2 (Hafner et al., 2020) was also shown to solve visual continuous control problems and it was ﬁrst to solve the humanoid locomotion problem from pixels.
Revised: Recently, a model-based method, DreamerV2 (Hafner et al., 2020) was also shown to solve visual continuous control problems and it was ﬁrst to solve the humanoid locomotion problem from pixels.

Original: While our model-free
Revised: While our model-free

Original: DrQ-v2 matches DreamerV2 in terms sample efﬁciency and performance, it does so 4 ⇥ faster in terms of wall-clock time to train.
Revised: DrQ-v2 matches DreamerV2 in terms sample efﬁciency and performance, it does so 4 ⇥ faster in terms of wall-clock time to train.

Original: We believe this makes DrQ-v2 a more accessible approach to support research in visual continuous control and it reinforces the question on whether model-free or model-based is the more suitable approach to solve this type of tasks.
Revised: We believe this makes DrQ-v2 a more accessible approach to support research in visual continuous control and it reinforces the question on whether model-free or model-based is the more suitable approach to solve this type of tasks.

Original: DrQ-v2, which is detailed in Section 3, improves upon DrQ (Yarats et al., 2021b) by making several algorithmic changes: (i) switching the base RL algorithm from SAC (Haarnoja et al., 2018b) to DDPG (Lillicrap et al., 2015a), (ii) this allows us straightforwardly incorporating multi-step return, (iii) adding bilinear interpolation to the random shift image augmentation, (iv) introducing an exploration schedule, (v) selecting better hyper-parameters including a larger capacity of the replay buffer.
Revised: DrQ-v2, which is detailed in Section 3, improves upon DrQ (Yarats et al., 2021b) by making several algorithmic changes: (i) switching the base RL algorithm from SAC (Haarnoja et al., 2018a) to DDPG (Lillicrap et al., 2015a) with clipped double Q-learning from TD3 (Fujimoto et al., 2018), (ii) this allows us straightforwardly incorporating multi-step return, (iii) adding bilinear interpolation to the random shift image augmentation, (iv) introducing an exploration schedule, (v) selecting better hyper-parameters including a larger capacity of the replay buffer.

Original: A careful ablation study of these design choices is presented in Section 4.4.
Revised: A careful ablation study of these design choices is presented in Section 4.4.

Original: Furthermore, we re-examine the original implementation of DrQ and identify several computational bottlenecks such as replay buffer management, data augmentation processing, batch size, and frequency of learning updates (see Section 3.2).
Revised: Furthermore, we re-examine the original implementation of DrQ and identify several computational bottlenecks such as replay buffer management, data augmentation processing, batch size, and frequency of learning updates (see Section 3.2).

Original: To remedy these, we have developed a new implementation that both achieves better performance and trains around 3 .
Revised: To remedy these, we have developed a new implementation that both achieves better performance and trains around 3 .

Original: 5 times faster with respect to wall-clock time than the previous implementation on the same hardware with an increase in environment frame throughput (FPS) from 28 to 96 (i.e., it takes 10 6 / 96 / 3600 ⇡ 2 . 9 hours to train for 1M environment steps).
Revised: 5 times faster with respect to wall-clock time than the previous implementation on the same hardware with an increase in environment frame throughput (FPS) from 28 to 96 (i.e., it takes 10 6 / 96 / 3600 ⇡ 2 . 9 hours to train for 1M environment steps).

Original: DrQv2’s implementation is available at https://anonymous.4open.science/r/drqv2 .
Revised: DrQ-v2’s implementation is available at https://anonymous.4open.science/r/drqv2 .

Original:  B ACKGROUND 2. R EINFORCEMENT L EARNING FROM I MAGES
Revised:  B ACKGROUND 2. R EINFORCEMENT L EARNING FROM I MAGES

Original: We formulate image-based control as an inﬁnite-horizon Markov Decision Process (MDP) (Bellman, 1957).
Revised: We formulate image-based control as an inﬁnite-horizon Markov Decision Process (MDP) (Bellman, 1957).

Original: Generally, in such a setting, an image rendering of the system is not sufﬁcient to perfectly describe the system’s underlying state.
Revised: Generally, in such a setting, an image rendering of the system is not sufﬁcient to perfectly describe the system’s underlying state.

Original: To this end and per common practice (Mnih et al., 2013), we approximate the current state of the system by stacking three consecutive prior observations.
Revised: To this end and per common practice (Mnih et al., 2013), we approximate the current state of the system by stacking three consecutive prior observations.

Original: With this in mind, such MDP can be described as a tuple ( X , A , P, R,  , d 0 ) , where X is the state space (a three-stack of image observations), A is the action space, P : X ⇥ A !
Revised: With this in mind, such MDP can be described as a tuple ( X , A , P, R,  , d 0 ) , where X is the state space (a three-stack of image observations), A is the action space, P : X ⇥ A !

Original:  ( X ) is the transition function 1 that deﬁnes a probability distribution over the next state given the current state and action,
Revised:  ( X ) is the transition function 1 that deﬁnes a probability distribution over the next state given the current state and action,

Original: R : X ⇥ A !
Revised: R : X ⇥ A !

Original: [0 , 1] is the reward function,  2 [0 , 1) is a discount factor, and d 0 2  ( X ) is the distribution of the initial state x 0 .
Revised: [0 , 1] is the reward function,  2 [0 , 1) is a discount factor, and d 0 2  ( X ) is the distribution of the initial state x 0 .

Original: The goal is to ﬁnd a policy ⇡ : X !
Revised: The goal is to ﬁnd a policy ⇡ : X !

Original:  ( A ) that maximizes the expected discounted sum of rewards E ⇡ [
Revised:  ( A ) that maximizes the expected discounted sum of rewards E ⇡ [

Original: P 1 t =

Revised: P 1 t =

Original:  t r t ] , where x 0 ⇠ d 0 , and 8 t we have a t ⇠ ⇡ ( ·| x t ) , x t +1 ⇠ P ( ·| x t , a t ) , and r t =
Revised:  t r t ] , where x 0 ⇠ d 0 , and 8 t we have a t ⇠ ⇡ ( ·| x t ) , x t +1 ⇠ P ( ·| x t , a t ) , and r t =

Original: R ( x t , a t ) .
Revised: R ( x t , a t ) .

Original:  D EEP D ETERMINISTIC P OLICY G RADIENT
Revised: 2. D EEP D ETERMINISTIC P OLICY G RADIENT

Original: Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015a) is an actor-critic algorithm for continuous control that concurrently learns a Q-function Q ✓ and a deterministic policy ⇡  .
Revised: Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015a) is an actor-critic algorithm for continuous control that concurrently learns a Q-function Q ✓ and a deterministic policy ⇡  .

Original: For this,
Revised: For this,

Original: DDPG uses Q-learning (Watkins and Dayan, 1992) to learn Q ✓ by minimizing the one-step Bellman residual J ✓ ( D ) =
Revised: DDPG uses Q-learning (Watkins and Dayan, 1992) to learn Q ✓ by minimizing the one-step Bellman residual J ✓ ( D ) =

Original: E ( x t , a t ,r t , x t +1 )
Revised: E ( x t , a t ,r t , x t +1 )

Original: ⇠ D [( Q ✓ ( x t , a t )  r t   Q ¯ ✓ ( x t +1 , ⇡  ( x t +1 )) 2 ] .
Revised: ⇠ D [( Q ✓ ( x t , a t )  r t   Q ¯ ✓ ( x t +1 , ⇡  ( x t +1 )) 2 ] .

Original: The policy ⇡  is learned by employing Deterministic Policy Gradient (DPG) (Silver et al., 2014) and maximizing
Revised: The policy ⇡  is learned by employing Deterministic Policy Gradient (DPG) (Silver et al., 2014) and maximizing

Original: J  ( D ) =
Revised: J  ( D ) =

Original: E x t ⇠ D [ Q ✓ ( x t , ⇡  ( x t ))] , so ⇡  ( x t ) approximates argmax a Q ✓ ( x t , a ) .
Revised: E x t ⇠ D [ Q ✓ ( x t , ⇡  ( x t ))] , so ⇡  ( x t ) approximates argmax a Q ✓ ( x t , a ) .

Original: Here, D is a replay buffer of environment transitions and ¯ ✓ is an exponential moving average of the weights.
Revised: Here, D is a replay buffer of environment transitions and ¯ ✓ is an exponential moving average of the weights.

Original: DDPG is amenable to incorporate n -step returns (Watkins, 1989; eng and Williams, 1996) when estimating
Revised: DDPG is amenable to incorporate n -step returns (Watkins, 1989; eng and Williams, 1996) when estimating

Original: TD error beyond a single step (Barth-Maron et al., 2018).
Revised: TD error beyond a single step (Barth-Maron et al., 2018).

Original: In practice, n -step returns allow for faster reward propagation and has been previously used in policy gradient and Q-learning methods (Mnih et al., 2016b; Barth-Maron et al., 2018; Hessel et al., 2017).
Revised: In practice, n -step returns allow for faster reward propagation and has been previously used in policy gradient and Q-learning methods (Mnih et al., 2016b; Barth-Maron et al., 2018; Hessel et al., 2017).

Original: 2. D ATA A UGMENTATION IN R EINFORCEMENT L EARNING
Revised: 2. D ATA A UGMENTATION IN R EINFORCEMENT L EARNING

Original: Recently, it has been shown that data augmentation techniques, commonplace in Computer Vision, are also important for achieving the state-of-the-art performance in image-based RL (Yarats et al., 2021b;
Revised: Recently, it has been shown that data augmentation techniques, commonplace in Computer Vision, are also important for achieving the state-of-the-art performance in image-based RL (Yarats et al., 2021b;

Original: Laskin et al., 2020).
Revised: Laskin et al., 2020).

Original: For example, the state-of-the-art algorithm for visual RL, DrQ (Yarats et al., 2021b) builds on top of Soft Actor-Critic (Haarnoja et al., 2018b), a model-free actor-critic algorithm, by adding a convolutional encoder and data augmentation in the form of random shifts.
Revised: For example, the state-of-the-art algorithm for visual RL, DrQ (Yarats et al., 2021b) builds on top of Soft Actor-Critic (Haarnoja et al., 2018a), a model-free actor-critic algorithm, by adding a convolutional encoder and data augmentation in the form of random shifts.

Original: The use of such data augmentations now forms an essential component of several recent visual RL algorithms (Srinivas et al., 2020; Raileanu et al., 2020;
Revised: The use of such data augmentations now forms an essential component of several recent visual RL algorithms (Srinivas et al., 2020; Raileanu et al., 2020;

Original: Yarats et al., 2021a; Stooke et al., 2020;
Revised: Yarats et al., 2021a; Stooke et al., 2020;

Original: Hansen and Wang, 2021; Schwarzer et al., 2020b).
Revised: Hansen and Wang, 2021; Schwarzer et al., 2020b).

Original:  D R Q- V 2: I MPROVED D ATA -A UGMENTED R EINFORCEMENT L EARNING
Revised:  D R Q- V 2: I MPROVED D ATA -A UGMENTED R EINFORCEMENT L EARNING

Original: In this section, we describe DrQ-v2, a simple model-free actor-critic RL algorithm for image-based continuous control, that builds upon DrQ.
Revised: In this section, we describe DrQ-v2, a simple model-free actor-critic RL algorithm for image-based continuous control, that builds upon DrQ.

Original: 3. A LGORITHMIC D ETAILS
Revised: 3. A LGORITHMIC D ETAILS

Original: Image Augmentation
Revised: Image Augmentation

Original: As in DrQ we apply random shifts image augmentation to pixel observations of the environment.
Revised: As in DrQ we apply random shifts image augmentation to pixel observations of the environment.

Original: In the settings of visual continuous control by DMC, this augmentation can be instantiated by ﬁrst padding each side of 84 ⇥ 84 observation rendering by 4 pixels (by repeating boundary pixels), and then selecting a random 84 ⇥ 84 crop, yielding the original image shifted by ± 4 pixels.
Revised: In the settings of visual continuous control by DMC, this augmentation can be instantiated by ﬁrst padding each side of 84 ⇥ 84 observation rendering by 4 pixels (by repeating boundary pixels), and then selecting a random 84 ⇥ 84 crop, yielding the original image shifted by ± 4 pixels.

Original: We also ﬁnd it useful to apply bilinear interpolation on top of the shifted image (i.e, we replace each pixel value with the average of the four nearest pixel values).
Revised: We also ﬁnd it useful to apply bilinear interpolation on top of the shifted image (i.e, we replace each pixel value with the average of the four nearest pixel values).

Original: In our experiments, this modiﬁcation provides an additional performance boost across the board.
Revised: In our experiments, this modiﬁcation provides an additional performance boost across the board.

Original: Image Encoder
Revised: Image Encoder

Original: The augmented image observation is then embedded into a low-dimensional latent vector by applying a convolutional encoder.
Revised: The augmented image observation is then embedded into a low-dimensional latent vector by applying a convolutional encoder.

Original: We use the same encoder architecture as in DrQ, which ﬁrst was introduced introduced in SAC-AE (Yarats et al., 2019).
Revised: We use the same encoder architecture as in DrQ, which ﬁrst was introduced introduced in SAC-AE (Yarats et al., 2019).

Original: This process can be succinctly summarized as h =
Revised: This process can be succinctly summarized as h =

Original: f ⇠ (aug( x )) , where f ⇠ is the encoder, aug is the random shifts augmentation, and x is the original image observation.
Revised: f ⇠ (aug( x )) , where f ⇠ is the encoder, aug is the random shifts augmentation, and x is the original image observation.

Original: Actor-Critic Algorithm
Revised: Actor-Critic Algorithm

Original: We use DDPG (Lillicrap et al., 2015a) as a backbone actor-critic RL algorithm and, similarly to Barth-Maron et al.
Revised: We use DDPG (Lillicrap et al., 2015a) as a backbone actor-critic RL algorithm and, similarly to Barth-Maron et al.

Original: (2018), augment it with n -step returns to estimate TD error.
Revised: (2018), augment it with n -step returns to estimate TD error.

Original: This results into faster reward propagation and overall learning progress (Mnih et al., 2016a).
Revised: This results into faster reward propagation and overall learning progress (Mnih et al., 2016a).

Original: While some methods (Hafner et al., 2020) employ more sophisticated techniques such as TD(  ) or
Revised: While some methods (Hafner et al., 2020) employ more sophisticated techniques such as TD(  ) or

Original: Retrace(  ) (Munos et al., 2016), they are often computationally demanding when n is large.
Revised: Retrace(  ) (Munos et al., 2016), they are often computationally demanding when n is large.

Original: We ﬁnd that using simple n -step returns, without an importance sampling correction, strikes a good balance between performance and efﬁciency.
Revised: We ﬁnd that using simple n -step returns, without an importance sampling correction, strikes a good balance between performance and efﬁciency.

Original: We also employ clipped double Q-learning (Fujimoto et al., 2018) to reduce overestimation bias in the target value.
Revised: We also employ clipped double Q-learning (Fujimoto et al., 2018) to reduce overestimation bias in the target value.

Original: Practically, this requires training two Qfunctions Q ✓ 1 and Q ✓ 2 .
Revised: Practically, this requires training two Qfunctions Q ✓ 1 and Q ✓ 2 .

Original: For this, we sample a mini-batch of transitions ⌧ = ( x t , a t , r t : t + n  1 , x t + n ) from the replay buffer D and compute the following two losses:
Revised: For this, we sample a mini-batch of transitions ⌧ = ( x t , a t , r t : t + n  1 , x t + n ) from the replay buffer D and compute the following two losses:

Original: [Equation]
Revised: [Equation]

Original: with the TD target y deﬁned as:
Revised: with the TD target y deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where h t = f
Revised: where h t = f

Original: ⇠ (aug( x t )) , h t + n = f ⇠ (aug( x t + n )) , a t + n = ⇡  ( h t + n ) + ✏ , ¯ ✓ 1 and ¯ ✓ 2 are the slowmoving weights for the Q target networks.
Revised: ⇠ (aug( x t )) , h t + n = f ⇠ (aug( x t + n )) , a t + n = ⇡  ( h t + n ) + ✏ , ¯ ✓ 1 and ¯ ✓ 2 are the slowmoving weights for the Q target networks.

Original: We note, that in contrast to DrQ, we do not employ a target network for the encoder f
Revised: We note, that in contrast to DrQ, we do not employ a target network for the encoder f

Original: ⇠ and always use the most recent weights ⇠ to embed x t and x t + n .
Revised: ⇠ and always use the most recent weights ⇠ to embed x t and x t + n .

Original: The exploration noise ✏ is sampled from clip( N (0 ,  2 ) ,  c, c ) similar to TD3 (Fujimoto et al., 2018), with the exception of decaying  , which we describe below.
Revised: The exploration noise ✏ is sampled from clip( N (0 ,  2 ) ,  c, c ) similar to TD3 (Fujimoto et al., 2018), with the exception of decaying  , which we describe below.

Original: Finally, we train the deterministic actor ⇡  using DPG with the following loss:
Revised: Finally, we train the deterministic actor ⇡  using DPG with the following loss:

Original: [Equation]
Revised: [Equation]

Original: where h t = f
Revised: where h t = f

Original: ⇠ (aug( x t )) , a t = ⇡  ( h t ) + ✏ , and ✏ ⇠ clip( N (0 ,  2 ) ,  c, c ) .
Revised: ⇠ (aug( x t )) , a t = ⇡  ( h t ) + ✏ , and ✏ ⇠ clip( N (0 ,  2 ) ,  c, c ) .

Original: Similar to DrQ, we do not use actor’s gradients to update the encoder’s parameters ⇠ .
Revised: Similar to DrQ, we do not use actor’s gradients to update the encoder’s parameters ⇠ .

Original: Scheduled Exploration Noise Empirically, we observe that it is helpful to have different levels of exploration at different stages of learning.
Revised: Scheduled Exploration Noise Empirically, we observe that it is helpful to have different levels of exploration at different stages of learning.

Original: At the beginning of training we want the agent to be more stochastic and explore the environment more effectively, while at the later stages of training, when the agent has already identiﬁed promising behaviors, it is better to be more deterministic and master those behaviors.
Revised: At the beginning of training we want the agent to be more stochastic and explore the environment more effectively, while at the later stages of training, when the agent has already identiﬁed promising behaviors, it is better to be more deterministic and master those behaviors.

Original: Similar to Amos et al.
Revised: Similar to Amos et al.

Original: (2020), we instantiate this idea by using linear decay  ( t ) for the variance  2 of the exploration noise deﬁned as:
Revised: (2020), we instantiate this idea by using linear decay  ( t ) for the variance  2 of the exploration noise deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where  init and  ﬁnal are the initial and ﬁnal values for standard deviation, and T is the decay horizon.
Revised: where  init and  ﬁnal are the initial and ﬁnal values for standard deviation, and T is the decay horizon.

Original: Key Hyper-Parameters We conduct an extensive hyper-parameter search and identify several hyper-parameter changes compared to DrQ. The three most important hyper-parameters are: (i) the size of the replay buffer, (ii) mini-batch size, and (iii) learning rate.
Revised: Key Hyper-Parameters We conduct an extensive hyper-parameter search and identify several hyper-parameter changes compared to DrQ. The three most important hyper-parameters are: (i) the size of the replay buffer, (ii) mini-batch size, and (iii) learning rate.

Original: Speciﬁcally, we use a 10 times larger replay buffer than DrQ. We also use a smaller mini-batch size of 256 without any noticeable performance degradation.
Revised: Speciﬁcally, we use a 10 times larger replay buffer than DrQ. We also use a smaller mini-batch size of 256 without any noticeable performance degradation.

Original: This is in contrast to CURL (Srinivas et al., 2020) and DrQ (Yarats et al., 2021b) that both use a larger batch size of 512 to attain more stable training in the expense of computational efﬁciency.
Revised: This is in contrast to CURL (Srinivas et al., 2020) and DrQ (Yarats et al., 2021b) that both use a larger batch size of 512 to attain more stable training in the expense of computational efﬁciency.

Original: Finally, we ﬁnd that using smaller learning rate of 1 ⇥ 10  4 , rather than
Revised: Finally, we ﬁnd that using smaller learning rate of 1 ⇥ 10  4 , rather than

Original: DrQ’s learning rate of 1 ⇥ 10  3 , results into more stable training without any loss in learning speed.
Revised: DrQ’s learning rate of 1 ⇥ 10  3 , results into more stable training without any loss in learning speed.

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: Faster Image Augmentation We replace DrQ’s random shifts augmentation (i.e., kornia.augmentation.
Revised: Faster Image Augmentation We replace DrQ’s random shifts augmentation (i.e., kornia.augmentation.

Original: RandomCrop ) by a custom implementation that uses ﬂowﬁeld image sampling provided in PyTorch (i.e., grid_sample ).
Revised: RandomCrop ) by a custom implementation that uses ﬂowﬁeld image sampling provided in PyTorch (i.e., grid_sample ).

Original: This is done for two reasons.
Revised: This is done for two reasons.

Original: First, we noticed that Kornia’s implementation does not fully utilize GPU pipelining since it has some intermediate CPU to GPU data transferring which breaks the computational ﬂow.
Revised: First, we noticed that Kornia’s implementation does not fully utilize GPU pipelining since it has some intermediate CPU to GPU data transferring which breaks the computational ﬂow.

Original: Second, using grid_sample allows straightforward addition of bilinear interpolation.
Revised: Second, using grid_sample allows straightforward addition of bilinear interpolation.

Original: Our custom random shifts augmentation improves training throughput by a factor of 2 .
Revised: Our custom random shifts augmentation improves training throughput by a factor of 2 .

Original: Faster Replay Buffer Another computational bottleneck of DrQ was the replay buffer.
Revised: Faster Replay Buffer Another computational bottleneck of DrQ was the replay buffer.

Original: The speciﬁc implementation had poor memory management which resulted in slow CPU to GPU data transfer, which also restricted the number of image-based transitions that could be stored.
Revised: The speciﬁc implementation had poor memory management which resulted in slow CPU to GPU data transfer, which also restricted the number of image-based transitions that could be stored.

Original: We reimplemented the replay buffer to address these issues which led to a ten-fold increase in storage capacity and faster data transfer.
Revised: We reimplemented the replay buffer to address these issues which led to a ten-fold increase in storage capacity and faster data transfer.

Original: More details are available in our open-source release.
Revised: More details are available in our open-source release.

Original: We note that the improved training speed of DrQ-v2 was key to solving humanoid tasks as it enabled much faster experimentation.
Revised: We note that the improved training speed of DrQ-v2 was key to solving humanoid tasks as it enabled much faster experimentation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we provide empirical evaluation of DrQ-v2 on an extensive set of visual continuous control tasks from DMC (Tassa et al., 2018).
Revised: In this section we provide empirical evaluation of DrQ-v2 on an extensive set of visual continuous control tasks from DMC (Tassa et al., 2018).

Original: We ﬁrst present comparison to prior methods, both model-free and model-based, in terms of sample efﬁciency and wall-clock time.
Revised: We ﬁrst present comparison to prior methods, both model-free and model-based, in terms of sample efﬁciency and wall-clock time.

Original: We then present a large scale ablation study that guided the ﬁnal version of DrQ-v2.
Revised: We then present a large scale ablation study that guided the ﬁnal version of DrQ-v2.

Original: 4. S ETUP
Revised: 4. S ETUP

Original: Environments We consider a set of MuJoCo tasks (Todorov et al., 2012) provided by DMC (Tassa et al., 2018), a widely used benchmark for continous control.
Revised: Environments We consider a set of MuJoCo tasks (Todorov et al., 2012) provided by DMC (Tassa et al., 2018), a widely used benchmark for continous control.

Original: DMC offers environments of various difﬁculty, ranging from the simple control problems such as the single degree of freedom (DOF) pendulum and cartpool, to the control of complex multi-joint bodies such as the humanoid (21 DOF).
Revised: DMC offers environments of various difﬁculty, ranging from the simple control problems such as the single degree of freedom (DOF) pendulum and cartpool, to the control of complex multi-joint bodies such as the humanoid (21 DOF).

Original: We consider learning from pixels.
Revised: We consider learning from pixels.

Original: In this setting, environment observations are stacks of 3 consecutive
Revised: In this setting, environment observations are stacks of 3 consecutive

Original: RGB images of size 84 ⇥ 84 , stacked along the channel dimension to enable inference of dynamic information like velocity and acceleration.
Revised: RGB images of size 84 ⇥ 84 , stacked along the channel dimension to enable inference of dynamic information like velocity and acceleration.

Original: In total, we consider 24 different tasks, which we group into three buckets, easy , medium , and hard , according to the sample complexity to reach near-optimal performance (see Appendix B).
Revised: In total, we consider 24 different tasks, which we group into three buckets, easy , medium , and hard , according to the sample complexity to reach near-optimal performance (see Appendix B).

Original: Our motivation for this is to encourage RL practitioners to focus on the medium and hard tasks and stop using the easy tasks for evaluation, as they are mostly solved at this point and may no longer provide any valuable signal in comparing different methods.
Revised: Our motivation for this is to encourage RL practitioners to focus on the medium and hard tasks and stop using the easy tasks for evaluation, as they are mostly solved at this point and may no longer provide any valuable signal in comparing different methods.

Original: Training Details For all tasks in the suite an episode corresponds to 1000 steps, where a per-step reward is in the unit interval [0 , 1] .
Revised: Training Details For all tasks in the suite an episode corresponds to 1000 steps, where a per-step reward is in the unit interval [0 , 1] .

Original: This upper bounds the episode return to 1000 making it easier to compute aggregated performance measures across tasks.
Revised: This upper bounds the episode return to 1000 making it easier to compute aggregated performance measures across tasks.

Original: To facilitate fair wall-clock time comparison all algorithms are trained on the same hardware (i.e., a single NVIDIA V100 GPU machine) and evaluated with the same periodicity of 20000 environment steps.
Revised: To facilitate fair wall-clock time comparison all algorithms are trained on the same hardware (i.e., a single NVIDIA V100 GPU machine) and evaluated with the same periodicity of 20000 environment steps.

Original: Each evaluation query averages episode returns over 10 episodes.
Revised: Each evaluation query averages episode returns over 10 episodes.

Original: Per common practice (Hafner et al., 2019), we employ action repeat of 2 and measure sample complexity in the environment steps, rather than the actor steps.
Revised: Per common practice (Hafner et al., 2019), we employ action repeat of 2 and measure sample complexity in the environment steps, rather than the actor steps.

Original: In all the ﬁgures we plot the mean performance over 10 seeds together with the shaded regions which represent 95% conﬁdence intervals.
Revised: In all the ﬁgures we plot the mean performance over 10 seeds together with the shaded regions which represent 95% conﬁdence intervals.

Original: A full list of hyper-parameters can be found in Appendix E.
Revised: A full list of hyper-parameters can be found in Appendix E.

Original: Comparison Axes
Revised: Comparison Axes

Original: In many real-world applications, taking a step in the environment incurs signiﬁcant computational cost making sample efﬁciency a critical feature of an RL algorithm.
Revised: In many real-world applications, taking a step in the environment incurs signiﬁcant computational cost making sample efﬁciency a critical feature of an RL algorithm.

Original: It is hence important to compare RL algorithms in terms of their sample efﬁciency.
Revised: It is hence important to compare RL algorithms in terms of their sample efﬁciency.

Original: We facilitate this comparison by computing an algorithm’s performance measured by episode return with respect to environment steps.
Revised: We facilitate this comparison by computing an algorithm’s performance measured by episode return with respect to environment steps.

Original: On the other end, striving low sample complexity often comes at the cost of a poor computational efﬁciency.
Revised: On the other end, striving low sample complexity often comes at the cost of a poor computational efﬁciency.

Original: Unfortunately, recent deep RL literature has paid very little attention to this important axis, which has led to skyrocketing hardware requirements.
Revised: Unfortunately, recent deep RL literature has paid very little attention to this important axis, which has led to skyrocketing hardware requirements.

Original: Such a trend has made it virtually impossible for an RL practitioner with modest hardware capacity to contribute to advancements in image-based RL, leaving research in this area to a few well-equipped labs.
Revised: Such a trend has made it virtually impossible for an RL practitioner with modest hardware capacity to contribute to advancements in image-based RL, leaving research in this area to a few well-equipped labs.

Original: To democratize research in visual RL, we additionally propose to compare the agents in terms of wall-clock training time given the same single GPU hardware.
Revised: To democratize research in visual RL, we additionally propose to compare the agents in terms of wall-clock training time given the same single GPU hardware.

Original: We note that it is possible to adapt DrQ-v2 to a distributed setup, as has been done for DDPG in prior work (Barth-Maron et al., 2018; Hoffman et al., 2020).
Revised: We note that it is possible to adapt DrQ-v2 to a distributed setup, as has been done for DDPG in prior work (Barth-Maron et al., 2018; Hoffman et al., 2020).

Original: 4. C OMPARISON TO M ODEL -F REE M ETHODS
Revised: 4. C OMPARISON TO M ODEL -F REE M ETHODS

Original: Baselines We compare our method to several state-of-the-art model-free algorithms for visual RL including CURL (Srinivas et al., 2020), DrQ (Yarats et al., 2021b), and vanilla SAC (Haarnoja et al., 2018b) augmented with the convolutional encoder from SAC-AE (Yarats et al., 2019).
Revised: Baselines We compare our method to several state-of-the-art model-free algorithms for visual RL including CURL (Srinivas et al., 2020), DrQ (Yarats et al., 2021b), and vanilla SAC (Haarnoja et al., 2018a) augmented with the convolutional encoder from SAC-AE (Yarats et al., 2019).

Original: Vanilla SAC is a weak baseline and only included as a ground point to showcase the recent progress in visual RL. 
Revised: Vanilla SAC is a weak baseline and only included as a ground point to showcase the recent progress in visual RL. [Figure]

Original: Sample Efﬁciency Axis
Revised: Sample Efﬁciency Axis

Original: We present results on several medium and hard tasks in Figure 3a.
Revised: We present results on several medium and hard tasks in Figure 3a.

Original: Full results can be found in Appendix (Figure 6, Figure 8, and Figure 10).
Revised: Full results can be found in Appendix (Figure 6, Figure 8, and Figure 10).

Original: Our empirical study reveals that DrQ-v2 outperforms prior model-free methods in terms of sample efﬁciency across the three benchmarks with different levels of difﬁculty.
Revised: Our empirical study reveals that DrQ-v2 outperforms prior model-free methods in terms of sample efﬁciency across the three benchmarks with different levels of difﬁculty.

Original: Importantly, DrQ-v2’s advantage is more pronounced on harder tasks (i.e., acrobot, quadruped, and humanoid), where exploration is especially challenging.
Revised: Importantly, DrQ-v2’s advantage is more pronounced on harder tasks (i.e., acrobot, quadruped, and humanoid), where exploration is especially challenging.

Original: Finally, DrQ-v2 solves the DMC humanoid locomotion tasks directly from pixels, which, to the best of our knowledge, is the ﬁrst successful demonstration of such feat by a model-free method.[Figure]
Revised: Finally, DrQ-v2 solves the DMC humanoid locomotion tasks directly from pixels, which, to the best of our knowledge, is the ﬁrst successful demonstration of such feat by a model-free method.

Original: Compute Efﬁciency Axis To facilitate a fair comparison in terms of sheer wall-clock training time, besides employee the identical training protocol (see Section 4.1), we also use the same mini-batch size of 256 for each agent.
Revised: Compute Efﬁciency Axis To facilitate a fair comparison in terms of sheer wall-clock training time, besides employee the identical training protocol (see Section 4.1), we also use the same mini-batch size of 256 for each agent.

Original: In Figure 13, we evaluate DrQ-v2 on a subset of DMC tasks for the sake of brevity only, and note that the demonstrated results can be easily extrapolated to the other tasks given the linear dependency between training time and sample complexity.
Revised: In Figure 13, we evaluate DrQ-v2 on a subset of DMC tasks for the sake of brevity only, and note that the demonstrated results can be easily extrapolated to the other tasks given the linear dependency between training time and sample complexity.

Original: In our benchmarks, DrQ-v is able to achieve a throughput of 96 FPS, which favorably compares to DrQ’s 28 FPS (a 3 . 4 ⇥ increase), and CURL’s 

Revised: In our benchmarks, DrQ-v is able to achieve a throughput of 96 FPS, which favorably compares to DrQ’s 28 FPS (a 3 . 4 ⇥ increase), and CURL’s

Original: FPS (a 6 ⇥ increase) throughputs.
Revised: FPS (a 6 ⇥ increase) throughputs.

Original: Practically, DrQ-v2 solves easy, medium, and hard tasks within 2 . 9 , 8 . 6 , and 86 hours respectively.
Revised: Practically, DrQ-v2 solves easy, medium, and hard tasks within 2 . 9 , 8 . 6 , and 86 hours respectively.

Original: Full results can be found in Appendix (Figure 7, Figure 9, and Figure 11).
Revised: Full results can be found in Appendix (Figure 7, Figure 9, and Figure 11).

Original: 4. C OMPARISON TO M ODEL -B ASED M ETHODS
Revised: 4. C OMPARISON TO M ODEL -B ASED M ETHODS

Original: Baseline To see how DrQ-v2 stacks up against model-based methods, which tend to achieve better sample complexity in expense of a larger computational footprint, we also compare to recent and  unpublished 2 improvements to Dreamer-v2 (Hafner et al., 2020), a leading model-based approach for visual continuous control.
Revised: Baseline To see how DrQ-v2 stacks up against model-based methods, which tend to achieve better sample complexity in expense of a larger computational footprint, we also compare to recent and [Figure] unpublished 2 improvements to Dreamer-v2 (Hafner et al., 2020), a leading model-based approach for visual continuous control.

Original: The recent update shows that the model-based approach can solve the
Revised: The recent update shows that the model-based approach can solve the

Original: DMC humanoid tasks directly from pixel inputs.
Revised: DMC humanoid tasks directly from pixel inputs.

Original: The open-source implementation of Dreamer-v ( https://github.com/danijar/dreamerv2 ) only provides learning curves for Humanoid
Revised: The open-source implementation of Dreamer-v ( https://github.com/danijar/dreamerv2 ) only provides learning curves for Humanoid

Original: Walk .
Revised: Walk .

Original: For this reason we run their code to obtain results on other DMC tasks.
Revised: For this reason we run their code to obtain results on other DMC tasks.

Original: To limit hardware[Figure] requirements of compute-expensive Dreamer-v2, we only run it on a subset of 12 out of 24 considered tasks.
Revised: To limit hardware requirements of compute-expensive Dreamer-v2, we only run it on a subset of 12 out of 24 considered tasks.

Original: This subset, however, overlaps with all the three (i.e. easy, medium, and hard) benchmarks.
Revised: This subset, however, overlaps with all the three (i.e. easy, medium, and hard) benchmarks.

Original: Sample Efﬁciency Axis
Revised: Sample Efﬁciency Axis

Original: Our empirical study in Figure 4a reveals that in many cases, DrQ-v2, despite being a model-free method, can rival sample efﬁciency of state-of-the-art model-based
Revised: Our empirical study in Figure 4a reveals that in many cases, DrQ-v2, despite being a model-free method, can rival sample efﬁciency of state-of-the-art model-based

Original: Dreamer-v2.
Revised: Dreamer-v2.

Original: We note, however, that on several tasks (for example Acrobot Swingup ) Dreamer-v outperforms DrQ-v2.
Revised: We note, however, that on several tasks (for example Acrobot Swingup ) Dreamer-v outperforms DrQ-v2.

Original: We leave investigation of such discrepancy for future work.
Revised: We leave investigation of such discrepancy for future work.

Original: Full results are provided in Appendix D (Figure 12).
Revised: Full results are provided in Appendix D (Figure 12).

Original: Compute Efﬁciency Axis
Revised: Compute Efﬁciency Axis

Original: A different picture emerges if comparison is done with respect to wallclock training time.
Revised: A different picture emerges if comparison is done with respect to wallclock training time.

Original: Dreamer-v2, being a model-based method, performs signiﬁcantly more ﬂoating point operations to reach its sample efﬁciency.
Revised: Dreamer-v2, being a model-based method, performs signiﬁcantly more ﬂoating point operations to reach its sample efﬁciency.

Original: In our benchmarks, Dreamer-v2 records a throughput of 24 FPS, which is 4 ⇥ less than DrQ-v2’s throughput of 96 FPS, measured on the same hardware.
Revised: In our benchmarks, Dreamer-v2 records a throughput of 24 FPS, which is 4 ⇥ less than DrQ-v2’s throughput of 96 FPS, measured on the same hardware.

Original: In Figure 4b we plot learning curves against wall-clock time and observe that DrQ-v2 takes less time to solve the tasks.
Revised: In Figure 4b we plot learning curves against wall-clock time and observe that DrQ-v2 takes less time to solve the tasks.

Original: Full results can be found in Appendix (Figure 13).
Revised: Full results can be found in Appendix (Figure 13).

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: In this section we present an extensive ablation study that guided us to the ﬁnal version of DrQ-v2.
Revised: In this section we present an extensive ablation study that guided us to the ﬁnal version of DrQ-v2.

Original: Here, for brevity we only discuss experiments that were most impactful and omit others that did not pan out.
Revised: Here, for brevity we only discuss experiments that were most impactful and omit others that did not pan out.

Original: For computational reasons, we only ablate on 3 different control tasks of various difﬁculty levels.
Revised: For computational reasons, we only ablate on 3 different control tasks of various difﬁculty levels.

Original: Our ﬁndings are summarized in Figure 5 and detailed below.
Revised: Our ﬁndings are summarized in Figure 5 and detailed below.

Original: Switching from SAC to DDPG DrQ (Yarats et al., 2021b) leverages SAC (Haarnoja et al., 2018b) as the backbone RL algorithm.
Revised: Switching from SAC to DDPG DrQ (Yarats et al., 2021b) leverages SAC (Haarnoja et al., 2018a) as the backbone RL algorithm.

Original: While it has been demonstrated by many works, including the original manuscripts (Haarnoja et al., 2018b;a) that SAC is superior to DDPG (Lillicrap et al., 2015b), our careful examination identiﬁes two shortcomings that preclude SAC (within DrQ) to solve hard exploration-wise image-based tasks.
Revised: While it has been demonstrated by many works, including the original manuscripts (Haarnoja et al., 2018a;b) that SAC is superior to DDPG (Lillicrap et al., 2015b), our careful examination identiﬁes two shortcomings that preclude SAC (within DrQ) to solve hard exploration-wise image-based tasks.

Original: First, the automatic entropy adjustment strategy, introduced in Haarnoja et al.
Revised: First, the automatic entropy adjustment strategy, introduced in Haarnoja et al.

Original: (2018a), is inadequate and in some cases leads to a premature entropy collapse. 
Revised: (2018b), is inadequate and in some cases leads to a premature entropy collapse. [Equation] [Equation] [Equation]

Revised: [Equation]

Original: This prevents the agent from ﬁnding more optimal behaviors due to the insufﬁcient exploration.
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: This prevents the agent from ﬁnding more optimal behaviors due to the insufﬁcient exploration.

Original: In Figure 5a, we empirically verify our intuition and, indeed, observe that DDPG demonstrates better exploration properties than SAC.
Revised: In Figure 5a, we empirically verify our intuition and, indeed, observe that DDPG demonstrates better exploration properties than SAC.

Original: Here, DDPG uses constant  = 0 . 2 for the exploration noise.
Revised: Here, DDPG uses constant  = 0 . 2 for the exploration noise.

Original: N -step Returns The second issue concerns the inability of soft Q-learning to incorporate n -step returns to estimate TD error in a straightforward manner.
Revised: N -step Returns The second issue concerns the inability of soft Q-learning to incorporate n -step returns to estimate TD error in a straightforward manner.

Original: The reason for this is that computing a target value for soft Q-function requires estimating per-step entropy of the policy, which is challenging [Equation] [Equation] [Equation]
Revised: The reason for this is that computing a target value for soft Q-function requires estimating per-step entropy of the policy, which is challenging to do for large n in the off-policy regime.

Original: [Equation]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] to do for large n in the off-policy regime. In contrast, DDPG does not require estimating per-step entropy to compute targets and is more amenable for n -step returns.
Revised: In contrast, DDPG does not require estimating per-step entropy to compute targets and is more amenable for n -step returns.

Original: In Figure 5b we demonstrate that estimating TD error with n -step returns improves sample efﬁciency over vanilla DDPG.
Revised: In Figure 5b we demonstrate that estimating TD error with n -step returns improves sample efﬁciency over vanilla DDPG.

Original: We select 3 -step returns as a sensible choice for our method.
Revised: We select 3 -step returns as a sensible choice for our method.

Original: Replay Buffer Size We hypothesize that a larger replay buffer plays an important role in circumventing the catastrophic forgetting problem (Fedus et al., 2020).
Revised: Replay Buffer Size We hypothesize that a larger replay buffer plays an important role in circumventing the catastrophic forgetting problem (Fedus et al., 2020).

Original: This issue is especially prominent in tasks with more diverse initial state distributions (i.e., reacher or humanoid tasks), where the vast variety of possible behaviors requires signiﬁcantly larger memory.
Revised: This issue is especially prominent in tasks with more diverse initial state distributions (i.e., reacher or humanoid tasks), where the vast variety of possible behaviors requires signiﬁcantly larger memory.

Original: We conﬁrm this intuition by ablating the size of the replay buffer in Figure 5c, where we observe that a buffer size of 1M helps to improve performance on Reacher Hard considerably.
Revised: We conﬁrm this intuition by ablating the size of the replay buffer in Figure 5c, where we observe that a buffer size of 1M helps to improve performance on Reacher Hard considerably.

Original: Scheduled Exploration Noise Finally, we demonstrate that it is useful to decay the variance of the exploration noise over the course of training according to Equation (3).
Revised: Scheduled Exploration Noise Finally, we demonstrate that it is useful to decay the variance of the exploration noise over the course of training according to Equation (3).

Original: In Figure 5d, we compare two versions of our algorithm, where the ﬁrst variant uses a ﬁxed standard deviation of  = 0 .
Revised: In Figure 5d, we compare two versions of our algorithm, where the ﬁrst variant uses a ﬁxed standard deviation of  = 0 .

Original: 2 , while the second variant employes the decaying schedule  ( t ) , with parameters  init = 1 . 0 ,
Revised: 2 , while the second variant employes the decaying schedule  ( t ) , with parameters  init = 1 . 0 ,

Original:  ﬁnal = 0 . 1 , and
Revised:  ﬁnal = 0 . 1 , and

Original: T = 500000 .
Revised: T = 500000 .

Original: Having the exploration noise to decay linearly over time turns out to be helpful and provide an additional performance boost, which was especially useful for solving humanoid tasks.
Revised: Having the exploration noise to decay linearly over time turns out to be helpful and provide an additional performance boost, which was especially useful for solving humanoid tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Visual Reinforcement Learning Successes of visual representation learning in computer vision (Vincent et al., 2008; Doersch et al., 2015; Wang and Gupta, 2015; Noroozi and Favaro, 2016; Zhang et al., 2017; Gidaris et al., 2018) has inspired successes in visual RL, where coherent representations are learned alongside RL.
Revised: Visual Reinforcement Learning Successes of visual representation learning in computer vision (Vincent et al., 2008; Doersch et al., 2015; Wang and Gupta, 2015; Noroozi and Favaro, 2016; Zhang et al., 2017; Gidaris et al., 2018) has inspired successes in visual RL, where coherent representations are learned alongside RL.

Original: Works such as SAC-AE (Yarats et al., 2019), PlaNet (Hafner et al., 2018), and SLAC (Lee et al., 2019), demonstrated how auto-encoders (Finn et al., 2015) could improve visual RL.
Revised: Works such as SAC-AE (Yarats et al., 2019), PlaNet (Hafner et al., 2018), and SLAC (Lee et al., 2019), demonstrated how auto-encoders (Finn et al., 2015) could improve visual RL.

Original: Following this, other self-supervised objectives such as contrastive learning in CURL (Srinivas et al., 2020) and ATC (Stooke et al., 2020), self-prediction in SPR (Schwarzer et al., 2020a), contrastive cluster assignment in Proto-RL (Yarats et al., 2021a), and augmented data in DrQ (Yarats et al., 2021b) and RAD (Laskin et al., 2020), have signiﬁcantly bridged the gap between state-based and image-based RL.
Revised: Following this, other self-supervised objectives such as contrastive learning in CURL (Srinivas et al., 2020) and ATC (Stooke et al., 2020), self-prediction in SPR (Schwarzer et al., 2020a), contrastive cluster assignment in Proto-RL (Yarats et al., 2021a), and augmented data in DrQ (Yarats et al., 2021b) and RAD (Laskin et al., 2020), have signiﬁcantly bridged the gap between state-based and image-based RL.

Original: Future prediction objectives (Hafner et al., 2018; 2019; Yan et al., 2020; Finn et al., 2015; Pinto et al., 2016; Agrawal et al., 2016) and other auxiliary objectives (Jaderberg et al., 2016; Zhan et al., 2020; Young et al., 2020; Chen et al., 2020) have shown improvements on a variety of problems ranging from gameplay, continuous control, and robotics.
Revised: Future prediction objectives (Hafner et al., 2018; 2019; Yan et al., 2020; Finn et al., 2015; Pinto et al., 2016; Agrawal et al., 2016) and other auxiliary objectives (Jaderberg et al., 2016; Zhan et al., 2020; Young et al., 2020; Chen et al., 2020) have shown improvements on a variety of problems ranging from gameplay, continuous control, and robotics.

Original: In the context of visual control settings, clever use of augmented data (Yarats et al., 2021b; Laskin et al., 2020) currently produces state-of-the-art results on visual tasks from DMC (Tassa et al., 2018).
Revised: In the context of visual control settings, clever use of augmented data (Yarats et al., 2021b; Laskin et al., 2020) currently produces state-of-the-art results on visual tasks from DMC (Tassa et al., 2018).

Original: Humanoid 
Revised: Humanoid Control

Original: Control The humanoid control problem ﬁrst presented in Tassa et al.
Revised: The humanoid control problem ﬁrst presented in Tassa et al.

Original: (2012), has been studied as one of the hardest control problems due to its large state and action spaces.
Revised: (2012), has been studied as one of the hardest control problems due to its large state and action spaces.

Original: The earliest solutions to this problem use ideas in model-based optimal control to generate policies given an accurate model of the humanoid .
Revised: The earliest solutions to this problem use ideas in model-based optimal control to generate policies given an accurate model of the humanoid .

Original: Subsequent works in RL have shown that model-free policies can solve the humanoid control problem given access to proprioceptive state observations.
Revised: Subsequent works in RL have shown that model-free policies can solve the humanoid control problem given access to proprioceptive state observations.

Original: However, solving such a problem from visual observations has been a challenging problem, with leading RL algorithms making little progress to solve the task (Tassa et al., 2018).
Revised: However, solving such a problem from visual observations has been a challenging problem, with leading RL algorithms making little progress to solve the task (Tassa et al., 2018).

Original: Recently, Hafner et al.
Revised: Recently, Hafner et al.

Original: (2020) was able to solve this problem through a model-based technique in around 30M environment steps and 340 hours of training on a single GPU machine.
Revised: (2020) was able to solve this problem through a model-based technique in around 30M environment steps and 340 hours of training on a single GPU machine.

Original: DrQ-v2, presented in this paper, marks the ﬁrst model-free RL method that can solve humanoid control from visual observations, taking also around 30M steps and 86 hours of training on the same hardware.
Revised: DrQ-v2, presented in this paper, marks the ﬁrst model-free RL method that can solve humanoid control from visual observations, taking also around 30M steps and 86 hours of training on the same hardware.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have introduced a conceptually simple model-free actor-critic RL agent for image-based continuous control – DrQ-v2.
Revised: We have introduced a conceptually simple model-free actor-critic RL agent for image-based continuous control – DrQ-v2.

Original: Our method provides signiﬁcantly better computational footprint and masters tasks from DMC directly from pixels, most notably the humanoid locomotion tasks that were previously unsolved by model-free approaches.
Revised: Our method provides signiﬁcantly better computational footprint and masters tasks from DMC directly from pixels, most notably the humanoid locomotion tasks that were previously unsolved by model-free approaches.

Original: To support our empirical results and inspire further research in visual RL we provide an efﬁcient PyTorch implementation of DrQ-v2 at https://anonymous.4open.science/r/drqv2 .
Revised: To support our empirical results and inspire further research in visual RL we provide an efﬁcient PyTorch implementation of DrQ-v2 at https://anonymous.4open.science/r/drqv2 .