Original: O N THE D ATA -E FFICIENCY WITH C ONTRASTIVE I MAGE T RANSFORMATION IN R EINFORCEMENT L EARNING
Revised: O N THE D ATA -E FFICIENCY WITH C ONTRASTIVE I MAGE T RANSFORMATION IN R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Data-efficiency has always been an essential issue in pixel-based reinforcement learning (RL).
Revised: Data-efficiency has always been an essential issue in pixel-based reinforcement learning (RL).

Original: As the agent not only learnsthe decision-making but also meaningful representations from images.
Revised: As The agent not only learns decision-making but also meaningful representations from images.

Original: The line of reinforcement learning with data augmentation shows significant improvements in sample-efficiency.
Revised: The line of reinforcement learning with data augmentation shows significant improvements in sample-efficiency.

Original: However, it is challenging to guarantee the optimality invariant transformation, that is, the augmented data are readily recognized as a completely different state by the agent.
Revised: However, it is challenging to guarantee the optimality invariant transformation, that is, the augmented data are readily recognized as a completely different state by the agent.

Original: In the end, we propose a contrastive invariant transformation (CoIT), a simple yet promising learnable data augmentation combined with standard model-free algorithms to improve sample-efficiency.
Revised: In the end, we propose a contrastive invariant transformation (CoIT), a simple yet promising learnable data augmentation combined with standard model-free algorithms to improve sample-efficiency.

Original: Concretely, the differentiable CoIT leverages original samples with augmented samples and hastens the state encoder for a contrastive invariant embedding.
Revised: Concretely, the differentiable CoIT leverages original samples with augmented samples and hastens the state encoder for a contrastive invariant embedding.

Original: We evaluate our approach on DeepMind Control
Revised: We evaluate our approach on DeepMind Control

Original: Suite and Atari100K. Empirical results verify advances using CoIT, enabling it to outperform the new state-of-the-art on various tasks.
Revised: Suite and Atari100K. Empirical results verify advances using CoIT, enabling it to outperform the new state-of-the-art on various tasks.

Original: Source code is available at https://github.com/Kamituna/CoIT .
Revised: Source code is available at https://github.com/Kamituna/CoIT .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Improving data-efficiency to accomplishthe sequential decisionsmaking has always been a crucial problem in pixel-based reinforcement learning.
Revised: Improving data-efficiency to accomplish sequential decisions has always been a crucial problem in pixel-based reinforcement learning.

Original: As the agent has to learn an optimal policy with a meaningful information abstraction from observations parallel.
Revised: As the agent has to learn an optimal policy with a meaningful information abstraction from observations parallel.

Original: Unlike supervised representation learning with strong supervised high-dimensional signals, the training process in RL is fragile.
Revised: Unlike supervised representation learning with strong supervised high-dimensional signals, the training process in RL is fragile.

Original: It could be harmful to the training process and cause performance degradation consequently using inappropriate manners.
Revised: It could be harmful to the training process and cause performance degradation consequently using inappropriate manners.

Original: Hence, it is an urgent request to seek subtle representation learning methods for visual RL.
Revised: Hence, it is an urgent request to seek subtle representation learning methods for visual RL.

Original: Previous works have been proposed in the literature to demonstrate that introducing auxiliary loss functions such as pixel reconstruction (47) and contrastive learning (27) alleviates this issue.
Revised: Previous works have been proposed in the literature to demonstrate that introducing auxiliary loss functions such as pixel reconstruction (47) and contrastive learning (27) alleviates this issue.

Original: In particular, data augmentations have already provento be beneficial to the data efficiency.
Revised: In particular, data augmentations have already proven beneficial to data-efficiency.

Original: RAD (26) performs an extension of experiments, and widely analyzes the impact of various techniques in data augmentation.
Revised: RAD (26) performs an extension of experiments and widely analyzes the impact of various techniques in data augmentation.

Original: DrQ (48) and DrQ-v2 (49) make use of appropriate image augmentation with great successes.
Revised: DrQ (48) and DrQ-v2 (49) make use of appropriate image augmentation with great success.

Original: Also, previous works have carried out the potential of data augmentation in terms of generalization (19; 37; 55; 18; 14).
Revised: Also, previous works have carried out the potential of data augmentation in terms of generalization (19; 37; 55; 18; 14).

Original: Despite the mentioned efforts, it is fairly difficult to guarantee that the augmented representations are sufficiently diverse yet semantically consistent.
Revised: Despite the mentioned efforts, it is pretty hard to guarantee that the augmented representations are sufficiently diverse yet semantically consistent.

Original: To the end, we explore the underlying condition for representation learning in RL.
Revised: To this end, we explore the underlying condition for representation learning in RL.

Original: It is rationale to hypothesize that there is an optimal transformation enabling an encoder to abstract informative latent space.
Revised: It is rational to hypothesize that there is an optimal transformation enabling an encoder to abstract informative latent space.

Original: This line of works belongs to the regime of state abstraction (13; 54; 42; 45), which derives from grouping similar world-states for descriptions of the environment (12; 2; 8).
Revised: This line of works belongs to the regime of state abstraction (13; 54; 42; 45), which derives from grouping similar world-states for descriptions of the environment (12; 2; 8).

Original: Inspired by spatial transformer networks (STN) (23), a data augmentation model in  vision domain, we hold the point that merging the parameterized transformation with visual
Revised: Inspired by spatial transformer networks (STN) (23), a data augmentation model in the vision domain, we consider that merging the parameterized transformation with visual

Original: RL could be beneficial.
Revised: RL could be beneficial.

Original: The designed transformation not only discovers the optimality of state abstraction but also produces diverse virtual samples for the agent.
Revised: The designed transformation not only discovers the optimality of state abstraction but also produces diverse virtual samples for the agent.

Original: To do so, we enforce a learnable data augmentation whose parametersare updated along with the RL objective.
Revised: To do so, we enforce a learnable data augmentation that updates its parameters along with the RL objective.

Original: To understandthe parameterized augmentation and its relation tothe representation learning in RL, we focus on the basic data manipulation by generating augmented data from a learnable Gaussian distribution.
Revised: To understand parameterized augmentation and its relation to representation learning in RL, we focus on fundamental data manipulation by generating augmented data from a learnable Gaussian distribution.

Original: To be clearer, we present this TRANSFORM operation to control the margin of the augmentation under an RL training-friendly data distribution.
Revised: To be clear, we present the image transformation to control the margin of the augmentation under an RL training-friendly data distribution.

Original: Since changed data distributions meanwhile being controlled by learning algorithms would be helpful in high-dimensional cases (4).
Revised: Since changed data distributions meanwhile being controlled by learning algorithms would be helpful in high-dimensional cases (4).

Original: Here we raise our idea:
Revised: Here we raise our idea:

Original: Can we parameterize the data augmentation by sampling from a dynamic distribution to obtain a training-friendly state distribution along with model-free RL?
Revised: Can we parameterize the data augmentation by sampling from a dynamic distribution to obtain a training-friendly state distribution along with model-free RL?

Original: In light of this challenge, we present a contrastive invariant transformation (CoIT), a novel contrastive learning to ameliorate the data-efficiency for visual RL.
Revised: In light of this challenge, we present a contrastive invariant transformation (CoIT), a novel contrastive learning to ameliorate the data-efficiency for visual RL.

Original: CoIT integrates a learnable transformation for model-free methods with minimal changes to the architecture and training pipeline.
Revised: CoIT integrates a learnable transformation for model-free methods with minimal modification to the architecture and training pipeline.

Original: Specifically, we parameterize the mean and variance of a Gaussian distribution for transforming data and update the parameters altogether with RLtraining, by using constraints to urge faster algorithm convergence empirically.
Revised: Specifically, we parameterize the mean and variance of a Gaussian distribution for transforming data and update the parameters together with RL by using constraints to urge faster algorithm convergence empirically.

Original: As the learning goes on, the agent approximates the TRANSFORM distribution that is optimal for the task at hand to solve the task.
Revised: As the learning goes on, the agent approximates the TRANSFORM distribution that is optimal for the task at hand to solve the task.

Original: In addition, we evaluate
Revised: In addition, we evaluate

Original: CoIT on DeepMind Control Suite and Atari100K, and experimental results demonstrate that the learnable transformation outperforms the current SOTA methods such as DrQ-v2 and CURL (1).
Revised: CoIT on DeepMind Control Suite and Atari100K, and experimental results demonstrate that the learnable transformation outperforms the current SOTA methods.

Original: Besides, our method does not claim any custom architecture choices which is essential for reproducing endto-end training.
Revised: Besides, our method does not claim any custom architecture choices and is essential for reproducing end-to-end training.

Original: Through these results, we demonstrate that a learnable transformation is effective to improve data-efficiency for visual RL.
Revised: Based on these results, we demonstrate that a learnable transformation improves dataefficiency effectively for visual RL.

Original: Key Contributions: (i) We present CoIT, a simple yet effective framework with a learnable image transformation that integrates invariant representations with model-free RL to improve data-efficiency.
Revised: Key Contributions: (i) We present CoIT, a simple yet effective framework with a learnable image transformation that integrates invariant representations with model-free RL to improve data-efficiency.

Original: (ii)
Revised: (ii)

Original: We propose a theoretical analysis of how our method can approximate a stationary distribution over the transformed data by the optimal invariant metric, thus learning better representations.
Revised: We propose a theoretical analysis of how our method can approximate a stationary distribution over the transformed data by the optimal invariant metric, thus learning better representations.

Original: (iii)
Revised: (iii)

Original: We evaluate CoIT on popular benchmarks and show that our method outperforms prior state-of-the-art methods onboth data-efficiency and stability.
Revised: We evaluate CoIT on popular benchmarks and show that our method outperforms previous state-of-the-art methods on data-efficiency and stability.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Several concurrent methods have been proposed for improving data-efficiency whose common ingredients containing data augmentation and self-supervised learning are listed.
Revised: Several concurrent methods have been proposed for improving data-efficiency whose common ingredients containing data augmentation and self-supervised learning are listed.

Original: Data augmentation in RL.
Revised: Data augmentation in RL.

Original: Like the success of data augmentation in computer vision (57; 11; 52; 56), these methods have played a key role in improving the data-efficiency of visual RL problems(33; 47; 17; 28).
Revised: Like the success of data augmentation in computer vision (57; 11; 52; 56), these methods have played a key role in improving the data-efficiency of visual RL problems (33; 47; 17; 28).

Original: RAD (26) conducted mounts of experiments and finds  that different data augmentations lead to various results.
Revised: RAD (26) conducted mounts of experiments and finds out that different data augmentations lead to entirely different results.

Original: This provides a broader perspective for the follow-up study of data augmentation in RL.
Revised: It provides a broader perspective for the follow-up study of data augmentation in RL.

Original: DrQ (48) proposed an effective augmentation method called random shift and introduced a regularization term for Q-learning.
Revised: DrQ (48) proposed an effective augmentation method called random shift and introduced a regularization term for Q-learning.

Original: Based on DrQ, the DrQ-v2 (49) conducted minimal changes and demonstrate that merely a simple augmentation method could match the state-of-the-art model-based algorithm on both data-efficiency and performance.
Revised: Based on DrQ, the DrQ-v2 (49) conducted minimal changes and demonstrate that merely a simple augmentation method could match the state-of-the-art model-based algorithm on both data-efficiency and performance.

Original: Self-supervised learning in RL.
Revised: Self-supervised learning in RL.

Original: Motivated by the breakthrough in self-supervised learning (9; 20; 7; 16; 10), it is natural to combine these methods with visual RL to learn rich representations.
Revised: Motivated by the breakthrough in self-supervised learning (9; 20; 7; 16; 10), it is natural to combine these methods with visual RL to learn rich representations.

Original: CURL (27) introduces an framework similar to SimCLR (10) and MoCo (20) into visual RL.
Revised: CURL (27) introduces an framework similar to SimCLR (10) and MoCo (20) into visual RL.

Original: ReLIC (32) and CoBERL (5) also tried to minimize the consistent between positive samples by semanticpreserving data augmentation.
Revised: ReLIC (32) and CoBERL (5) also tried to minimize the consistent between positive samples by semanticpreserving data augmentation.

Original: Besides, STDIM (31) and PI-SAC (29) maximize the temporal mutual information (MI) between the nearby states.
Revised: Besides, STDIM (31) and PI-SAC (29) maximize the temporal mutual information (MI) between the nearby states.

Original: SPR (38) and PlayVirtual (51) follow their idea, but they utilize the dynamics model to predict nearby states in latent space.
Revised: SPR (38) and PlayVirtual (51) follow their idea, but they utilize the dynamics model to predict nearby states in latent space.

Original: DBC (54) and PSM (1) focus on learning task-relevant information.
Revised: DBC (54) and PSM (1) focus on learning task-relevant information.

Original: They utilize signals in the environment to achieve an invariant representation learning and thereby generalize the agent to unseen environments.
Revised: They utilize signals in the environment to achieve an invariant representation learning and thereby generalize the agent to unseen environments.

Original:  P RELIMINARIES 3. R EINFORCEMENT L EARNING FROM O BSERVATIONS
Revised:  P RELIMINARIES 3. R EINFORCEMENT L EARNING FROM O BSERVATIONS

Original: Visual RL control is formulated as an infinite-horizon Markov Decision Process (MDP) (6), as the rendered observations can not fully describe the underlying state.
Revised: Visual RL control is formulated as an infinite-horizon Markov Decision Process (MDP) (6), as the rendered observations can not fully describe the underlying state.

Original: To address this problem, we stack multiple consecutive frames together to represent the current underlying state s (33).
Revised: To address this problem, we stack multiple consecutive frames together to represent the current underlying state s (33).

Original: In this mind, the
Revised: In this mind, the

Original: MDP M is a 5-tuple ⟨O , S , A , r, γ ⟩ .
Revised: MDP M is a 5-tuple ⟨O , S , A , r, γ ⟩ .

Original: Here O generally consists of a multiple-stack of observations to represent the observation space.
Revised: Here O generally consists of a multiple-stack of observations to represent the observation space.

Original: The state space S is either observable or unobservable (40; 53).
Revised: The state space S is either observable or unobservable (40; 53).

Original: And the agent use observations O to sample actions from the action space A .
Revised: And the agent use observations O to sample actions from the action space A .

Original: Every time the agent interacts with the environment, it obtains a reward r .
Revised: Every time the agent interacts with the environment, it obtains a reward r .

Original: The end goal is to train an agent to maximize the cumulative reward R .
Revised: The end goal is to train an agent to maximize the cumulative reward R .

Original: The policy evaluation used as estimating the performance of the policy π ϕ is normally defined by rewards in infinite-horizon tasks,
Revised: The policy evaluation used as estimating the performance of the policy π ϕ is normally defined by rewards in infinite-horizon tasks,

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ [0 , 1) is the discount factor and r t denotes the reward at time t .
Revised: where γ ∈ [0 , 1) is the discount factor and r t denotes the reward at time t .

Original: 3. Q L EARNING
Revised: 3. Q L EARNING

Original: The state-action value function Q θ is trained by minimizing the Bellman error to estimate the cumulative reward at the current state:
Revised: The state-action value function Q θ is trained by minimizing the Bellman error to estimate the cumulative reward at the current state:

Original: [Equation]
Revised: [Equation]

Original: where e is a transition from the replay buffer D .
Revised: where e is a transition from the replay buffer D .

Original: And ¯ θ denotes an exponential moving average of θ .
Revised: And ¯ θ denotes an exponential moving average of θ .

Original: For the continuous control tasks, we utilize an actor-critic algorithm called Deep Deterministic Policy
Revised: For the continuous control tasks, we utilize an actor-critic algorithm called Deep Deterministic Policy

Original: Gradient (DDPG) (39; 30) which consists of the aforementioned state-action value function Q θ and a deterministic policy π ϕ .
Revised: Gradient (DDPG) (39; 30) which consists of the aforementioned state-action value function Q θ and a deterministic policy π ϕ .

Original: The policy π ϕ aims at maximizing J ϕ
Revised: The policy π ϕ aims at maximizing J ϕ

Original: ( D ) =
Revised: ( D ) =

Original: E D [( Q θ ( s t , π ϕ ( s t ))] .
Revised: E D [( Q θ ( s t , π ϕ ( s t ))] .

Original: Various effective improvements have also been lead to DDPG.
Revised: Various effective improvements have also been lead to DDPG.

Original: The Q-learning process incorporates n -step returns (46; 35).
Revised: The Q-learning process incorporates n -step returns (46; 35).

Original: The scheduled exploration noise is produced by a linear decay ˜ σ ( t ) for the variance ˜ σ 2 which provides different levels of exploration at different training steps: ˜ σ ( t ) =
Revised: The scheduled exploration noise is produced by a linear decay ˜ σ ( t ) for the variance ˜ σ 2 which provides different levels of exploration at different training steps: ˜ σ ( t ) =

Original: ˜ σ init + (1 − min( t/T, 1))(˜ σ final − ˜ σ init ) .
Revised: ˜ σ init + (1 − min( t/T, 1))(˜ σ final − ˜ σ init ) .

Original: The initial and final value for standard deviation are defined by ˜ σ init and ˜ σ final , and the decay horizon T is related to the total training steps of the environment.
Revised: The initial and final value for standard deviation are defined by ˜ σ init and ˜ σ final , and the decay horizon T is related to the total training steps of the environment.

Original: For the discrete control, we use the data-efficient Rainbow DQN (44) which applied multiple improvements on top of the original Nature DQN (34).
Revised: For the discrete control, we use the data-efficient Rainbow DQN (44) which applied multiple improvements on top of the original Nature DQN (34).

Original: 3. S TATE A BSTRACTION
Revised: 3. S TATE A BSTRACTION

Original: While visual RL has achieved many successes in simulated tasks, it remains challenging to learn robust representations from real vision, where images reveal detailed scenes of a complex and unstructured world (54; 1; 45).
Revised: While visual RL has achieved many successes in simulated tasks, it remains challenging to learn robust representations from real vision, where images reveal detailed scenes of a complex and unstructured world (54; 1; 45).

Original: Therefore, abstracting meaningful elements from the visual scene to present the underlying state is significantly important for visual RL.
Revised: Therefore, abstracting meaningful elements from the visual scene to present the underlying state is significantly important for visual RL.

Original: We follow the Block Markov Decision Process (BMDP) (13), which refers to episodic learning tasks via an unobservable latent space S and an observable context space X .
Revised: We follow the Block Markov Decision Process (BMDP) (13), which refers to episodic learning tasks via an unobservable latent space S and an observable context space X .

Original: The environment generates a context by x ∼ p ( ·| s ) .
Revised: The environment generates a context by x ∼ p ( ·| s ) .

Original: They present a fundamental assumption as: each observation x uniquely determines its generating state s .
Revised: They present a fundamental assumption as: each observation x uniquely determines its generating state s .

Original: Similarly, in model-free RL without modeling dynamics, the manipulated context x can be conditioned on a certain probability given an environment transition e which is x ∼ p ( ·| e ) .
Revised: Similarly, in model-free RL without modeling dynamics, the manipulated context x can be conditioned on a certain probability given an environment transition e which is x ∼ p ( ·| e ) .

Original:  T HE C O IT 4. L EARNABLE I NVARIANT T RANSFORMATION
Revised:  T HE C O IT 4. L EARNABLE I NVARIANT T RANSFORMATION

Original: Following the motivation of smoothing training experiences to stabilize the target Q network (33; 30), the transformed x ′ is required to satisfy x ′ ∼ p ( ·| e ) , where environment transition e is ideally in the replay distribution D .
Revised: Following the motivation of smoothing training experiences to stabilize the target Q network (33; 30), the transformed x ′ is required to satisfy x ′ ∼ p ( ·| e ) , where environment transition e is ideally in the replay distribution D .

Original: Note that e is a random variables.
Revised: Note that e is a random variables.

Original: Formally, we are ready to introduce the optimal invariant metric to reach the stationary distribution D over the augmented context x ′ , through the definition,
Revised: Formally, we are ready to introduce the optimal invariant metric to reach the stationary distribution D over the augmented context x ′ , through the definition,

Original: Definition 4.1.
Revised: Definition 4.1.

Original: (Optimal Invariant Metric).
Revised: (Optimal Invariant Metric).

Original: Given a transition distribution D for tuples in the replay buffer, suppose the block structure assumption holds, the shift between transitions x and its context x ′ can be measured by a conditional divergence:
Revised: Given a transition distribution D for tuples in the replay buffer, suppose the block structure assumption holds, the shift between transitions x and its context x ′ can be measured by a conditional divergence:

Original: [Equation]
Revised: [Equation]

Original: where d KL ( ·||· ) is the Kullback-Leibler (KL) divergence.
Revised: where d KL ( ·||· ) is the Kullback-Leibler (KL) divergence.

Original: It indicates the expected distance between x and x ′ conditioning on e .
Revised: It indicates the expected distance between x and x ′ conditioning on e .

Original: One may argue that the dynamics cannot be assumed as a fixed distribution when it comes to new observations, especially after data manipulation.
Revised: One may argue that the dynamics cannot be assumed as a fixed distribution when it comes to new observations, especially after data manipulation.

Original: Nevertheless, it generally claims that the experiences are uniformly sampled in a replay memory (33).
Revised: Nevertheless, it generally claims that the experiences are uniformly sampled in a replay memory (33).

Original: Also, the fact that the given conditions of the transition are consistent for the observed data as well as the transformed data, makes the above definition reasonable.
Revised: Also, the fact that the given conditions of the transition are consistent for the observed data as well as the transformed data, makes the above definition reasonable.

Original: Next, we will show why the conditional divergence defined in
Revised: Next, we will show why the conditional divergence defined in

Original: Eq.(3) is an optimal invariant metric from theoretical perspectives.
Revised: Eq.(3) is an optimal invariant metric from theoretical perspectives.

Original: We employ the Bayes’ rule on the conditionally distribution p ( x | e ) =
Revised: We employ the Bayes’ rule on the conditionally distribution p ( x | e ) =

Original: p ( e | x ) p ( x ) /p
Revised: p ( e | x ) p ( x ) /p

Original: ( e ) , ∀ x ∈
Revised: ( e ) , ∀ x ∈

Original: O , e ∈ D .
Revised: O , e ∈ D .

Original: Then the transition operator p ( e | x ) can be further divided as p ( e | s ) p ( s | x ) for any x ∈ O , if e and x are conditional independent given s 1 . Eq.(3) is rewritten as:
Revised: Then the transition operator p ( e | x ) can be further divided as p ( e | s ) p ( s | x ) for any x ∈ O , if e and x are conditional independent given s 1 . Eq.(3) is rewritten as:

Original: [Equation]
Revised: [Equation]

Original: Therefore, minimizing the conditional divergence leads to encoding the observation x and the transformed context x ′ into an invariant latent state space S .
Revised: Therefore, minimizing the conditional divergence leads to encoding the observation x and the transformed context x ′ into an invariant latent state space S .

Original: As a consequence, the learnable pixel transformation is an optimality invariant combining a qualified encoder.
Revised: As a consequence, the learnable pixel transformation is an optimality invariant combining a qualified encoder.

Original: Now we have the observation encoder g : O → S mapping from the observed state O to the latent state S by a non-trivial function g such that g ( x ) =
Revised: Now we have the observation encoder g : O → S mapping from the observed state O to the latent state S by a non-trivial function g such that g ( x ) =

Original: p ( s | x ) , ∀ x .
Revised: p ( s | x ) , ∀ x .

Original: Traditionally, there is only one encoder dubbed feature backbone in RL models.
Revised: Traditionally, there is only one encoder dubbed feature backbone in RL models.

Original: Since the pixel transformation could drift away, e.g., supported by different components with those supporting x (13).
Revised: Since the pixel transformation could drift away, e.g., supported by different components with those supporting x (13).

Original: To enforce the invariant hidden states, another state encoder g ′ that can map x ′ to s should exist, which is g ′ ( x ′ )
Revised: To enforce the invariant hidden states, another state encoder g ′ that can map x ′ to s should exist, which is g ′ ( x ′ )

Original: =
Revised: =

Original: p ( s | ν ( x , · )) , ∀ x ′ , ν ( x , · ) is the transformation.
Revised: p ( s | ν ( x , · )) , ∀ x ′ , ν ( x , · ) is the transformation.

Original: So far, the goal of learning the optimal transformed data and encoders boils down to minimizing the distance between representations g ( x ) and g ′ ( x ′ ) .
Revised: So far, the goal of learning the optimal transformed data and encoders boils down to minimizing the distance between representations g ( x ) and g ′ ( x ′ ) .

Original: To tackle the issue, we first provide two definitions to introduce measurements as follows,
Revised: To tackle the issue, we first provide two definitions to introduce measurements as follows,

Original: Definition 4.2.
Revised: Definition 4.2.

Original: ( ϵ -Approximation).
Revised: ( ϵ -Approximation).

Original: Given a distance metric d : O×S → R + satisfies
Revised: Given a distance metric d : O×S → R + satisfies

Original: d
Revised: d

Original: ( s , s )
Revised: ( s , s )

Original: = 0 , ∀ s , and let g, g ′ :
Revised: = 0 , ∀ s , and let g, g ′ :

Original: O → S be two functions.
Revised: O → S be two functions.

Original: Let ϵ ≥ 0 , given a distribution ˆ D on O , then g and g ′ are ϵ - approximate w.r.t.
Revised: Let ϵ ≥ 0 , given a distribution ˆ D on O , then g and g ′ are ϵ - approximate w.r.t.

Original: ( d, D ) if E x ∼ ˆ D
Revised: ( d, D ) if E x ∼ ˆ D

Original: [ d ( g ( x ) , g ′ ( x ))] ≤ ϵ .
Revised: [ d ( g ( x ) , g ′ ( x ))] ≤ ϵ .

Original: Definition 4.3.
Revised: Definition 4.3.

Original: ( β -Similarity).
Revised: ( β -Similarity).

Original: Given a distance metric d : O × S → R + satisfies
Revised: Given a distance metric d : O × S → R + satisfies

Original: d ( s , s ) = 0 , ∀ s .
Revised: d ( s , s ) = 0 , ∀ s .

Original: There exists g : O → S .
Revised: There exists g : O → S .

Original: Let ϵ ≥ 0 , given distributions ˆ D and ˆ D ′ on O , then x and x ′ are β - similar if E x ∼ ˆ D , x ′ ∼ ˆ D ′
Revised: Let ϵ ≥ 0 , given distributions ˆ D and ˆ D ′ on O , then x and x ′ are β - similar if E x ∼ ˆ D , x ′ ∼ ˆ D ′

Original: [ d ( g ( x ) , g ( x ′ ))] ≤ β .
Revised: [ d ( g ( x ) , g ( x ′ ))] ≤ β .

Original: Without loss of generality, the distance between the encoded states g ( x ) and g ′ ( x ′ ) can be expressed as the following triangular inequality.
Revised: Without loss of generality, the distance between the encoded states g ( x ) and g ′ ( x ′ ) can be expressed as the following triangular inequality.

Original: To obtain a metric, Kullback-Leibler divergence is rewritten in a form of the square root of Jensen-Shannon divergence.
Revised: To obtain a metric, Kullback-Leibler divergence is rewritten in a form of the square root of Jensen-Shannon divergence.

Original: Therefore, we have,
Revised: Therefore, we have,

Original: [Equation]
Revised: [Equation]

Original: From the view of invariant learning, minimizing the right side of the inequality can upper bound our problem.
Revised: From the view of invariant learning, minimizing the right side of the inequality can upper bound our problem.

Original: The first term on the right side is the so-called ϵ -approximation to measure the functional similarity after state abstraction, whereas the second term exists based on the procedure of data augmentation.
Revised: The first term on the right side is the so-called ϵ -approximation to measure the functional similarity after state abstraction, whereas the second term exists based on the procedure of data augmentation.

Original: Thus, we learn the encoders and shifted data simultaneously through the upper bound.
Revised: Thus, we learn the encoders and shifted data simultaneously through the upper bound.

Original: 4. O PTIMAL S TATE A BSTRACTION
Revised: 4. O PTIMAL S TATE A BSTRACTION

Original: To restrict the functional similarity of Eq.(5) from the perspective of learning a good encoding function with consistent semantics, the approaches formulating the main and momentum feature learning are utilized, motivated by contrastive learning (20).
Revised: To restrict the functional similarity of Eq.(5) from the perspective of learning a good encoding function with consistent semantics, the approaches formulating the main and momentum feature learning are utilized, motivated by contrastive learning (20).

Original: In particular, we enforce the encoding functions with exactly the same architecture, and use ¯ ξ t
Revised: In particular, we enforce the encoding functions with exactly the same architecture, and use ¯ ξ t

Original: =
Revised: =

Original: (1 − τ m )¯ ξ t − 1 + τ m ξ t at timestep t to update the parameters of momentum function g ¯ ξ with g ξ , where τ m ∈ [0 , 1] is the updating rate.
Revised: (1 − τ m )¯ ξ t − 1 + τ m ξ t at timestep t to update the parameters of momentum function g ¯ ξ with g ξ , where τ m ∈ [0 , 1] is the updating rate.

Original: Furthermore, we design a projection that is f : S → Y using a ReLU network (36) to upper bound the divergence by minimizing the distance in the projected space Y .
Revised: Furthermore, we design a projection that is f : S → Y using a ReLU network (36) to upper bound the divergence by minimizing the distance in the projected space Y .

Original: Previously, the projection has been proposed by Chen et al.
Revised: Previously, the projection has been proposed by Chen et al.

Original: (9), while the theoretical guarantees of the underlying mechanism with momentum updating for model-free RL are explained in this work.
Revised: (9), while the theoretical guarantees of the underlying mechanism with momentum updating for model-free RL are explained in this work.

Original: Suppose the Markov chain O g −→ S f −→ Y holds.
Revised: Suppose the Markov chain O g −→ S f −→ Y holds.

Original: For two functions g and f in the compatible ranges, we use f ◦ g to denote the function composition f
Revised: For two functions g and f in the compatible ranges, we use f ◦ g to denote the function composition f

Original: ( g ( · )) .
Revised: ( g ( · )) .

Original: Before showing the proposed data transformed method, we introduce technical lemmas to take advantage of the designable projection function by leveraging the convexity.
Revised: Before showing the proposed data transformed method, we introduce technical lemmas to take advantage of the designable projection function by leveraging the convexity.

Original: The momentum updating paradigm is capable of turning into momentum feature updating through a convex function or an equivalence of the convex function.
Revised: The momentum updating paradigm is capable of turning into momentum feature updating through a convex function or an equivalence of the convex function.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: Assume that h : R |S| → R |Y| can be written as h ( ξ ) =
Revised: Assume that h : R |S| → R |Y| can be written as h ( ξ ) =

Original: f ( < ξ, s > ) , for some s ∈ R |S| , and f : R |Y| → R |Y| with parameter ξ .
Revised: f ( < ξ, s > ) , for some s ∈ R |S| , and f : R |Y| → R |Y| with parameter ξ .

Original: Then, convexity of f implies the convexity of h .
Revised: Then, convexity of f implies the convexity of h .

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: Given the dynamical updating: ¯ ξ t = (1 − τ m )¯ ξ t − 1 + τ m ξ t .
Revised: Given the dynamical updating: ¯ ξ t = (1 − τ m )¯ ξ t − 1 + τ m ξ t .

Original: By Lemma 4.1, f ξ = f ¯ ξ holds after convergence.
Revised: By Lemma 4.1, f ξ = f ¯ ξ holds after convergence.

Original: As a result, the problem of min E x [ ∥
Revised: As a result, the problem of min E x [ ∥

Original: f ξ ◦ g ξ ( x ) − f ¯ ξ ◦ g ¯ ξ ( x ) ∥ ] is equivalent to the problem of min E x [ ∥ g ξ ( x ) − g ¯ ξ ( x ) ∥ ] .
Revised: f ξ ◦ g ξ ( x ) − f ¯ ξ ◦ g ¯ ξ ( x ) ∥ ] is equivalent to the problem of min E x [ ∥ g ξ ( x ) − g ¯ ξ ( x ) ∥ ] .

Original: To meet the requirement of a small upper bound, we state a theorem that provides some insights into why it is necessary to learn optimal transformed data together with the encoders.
Revised: To meet the requirement of a small upper bound, we state a theorem that provides some insights into why it is necessary to learn optimal transformed data together with the encoders.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: (CoIT)
Revised: (CoIT)

Original: Suppose that Lipschitzness holds for functions g ξ , g ¯ ξ , f ξ and f ¯ ξ , respectively.
Revised: Suppose that Lipschitzness holds for functions g ξ , g ¯ ξ , f ξ and f ¯ ξ , respectively.

Original: The updating dynamics is:
Revised: The updating dynamics is:

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: 1 + τ m ξ t , τ m ∈ [0 , 1] .
Revised: 1 + τ m ξ t , τ m ∈ [0 , 1] .

Original: For any input x ∼ ˆ D and transformed x ′ obtained via the transform operator ν ( x , · ) , optimizing the conditional divergence in
Revised: For any input x ∼ ˆ D and transformed x ′ obtained via the transform operator ν ( x , · ) , optimizing the conditional divergence in

Original: Definition 4.1 means to minimize the upper bound as follows,
Revised: Definition 4.1 means to minimize the upper bound as follows,

Original: [Equation]
Revised: [Equation]

Original: where ρ = L g ( CL f + ∥ ξ f ∥ ) ,
Revised: where ρ = L g ( CL f + ∥ ξ f ∥ ) ,

Original: C = 1+ τ 1 − τ , τ = 1 − τ m are constants.
Revised: C = 1+ τ 1 − τ , τ = 1 − τ m are constants.

Original: L f and L g are Lipschitz constants of the functions f ( s ) and g ( x ) , respectively.
Revised: L f and L g are Lipschitz constants of the functions f ( s ) and g ( x ) , respectively.

Original: 1: Initialization: Draw distribution G 0 ∼ G |O| with given mean µ 0 and variance σ 20 ;
Revised: 1: Initialization: Draw distribution G 0 ∼ G |O| with given mean µ 0 and variance σ 20 ;

Original: Set cumulative reward R = 0 .
Revised: Set cumulative reward R = 0 .

Original: 2: Training: 3: for each timestep t in 0 , · · · , T do 4: x ′ t =
Revised: 2: Training: 3: for each timestep t in 0 , · · · , T do 4: x ′ t =

Original: TRANSFORM ( x t , G t ) 5:
Revised: TRANSFORM ( x t , G t ) 5:

Original: R = R + γ t r ( x ′ t , a t ) 6:
Revised: R = R + γ t r ( x ′ t , a t ) 6:

Original: Adjust to an optimal G t ( µ t , σ t ) .
Revised: Adjust to an optimal G t ( µ t , σ t ) .

Original: 7: end for ∗ Details about learning µ t and σ t are given in Algo- rithm 2 in Appendix B
Revised: 7: end for ∗ Details about learning µ t and σ t are given in Algo- rithm 2 in Appendix B.

Original: The upper bound of the right side measures the margin of augmentation between original and transformed data.
Revised: The upper bound of the right side measures the margin of augmentation between original and transformed data.

Original: The left side measures the distribution changes in the latent space.
Revised: The left side measures the distribution changes in the latent space.

Original: Since both transformed data x ′ and encoder g are updating, incorporating augmentation directly cannot well meet the basic stationary environment.
Revised: Since both transformed data x ′ and encoder g are updating, incorporating augmentation directly cannot well meet the basic stationary environment.

Original: Hereby, the theorem suggests us that the automatic transformation is used to bound the representation learning so that the abstracted states enhance the stationary distribution of tuples e and facilitate efficient training.
Revised: Hereby, the theorem suggests us that the automatic transformation is used to bound the representation learning so that the abstracted states enhance the stationary distribution of tuples e and facilitate efficient training.

Original: Proofs are given in Appendix A.
Revised: Proofs are given in Appendix A.

Original: The empirical comparisons of the projection network are presented in Appendix D.
Revised: The empirical comparisons of the projection network are presented in Appendix D.

Original: 4. P ARAMETERIZABLE O BSERVATION
Revised: 4. P ARAMETERIZABLE O BSERVATION

Original: From the Theorem 4.1, we know the relation between the parameterized augmentation and the learnable latent state.
Revised: From the Theorem 4.1, we know the relation between the parameterized augmentation and the learnable latent state.

Original: That is, image transformation needs to be constrained by the distance between an observation and its associated augmentation.
Revised: That is, image transformation needs to be constrained by the distance between an observation and its associated augmentation.

Original: The optimal embedding can be obtained by minimizing of Eq.(5).
Revised: The optimal embedding can be obtained by minimizing of Eq.(5).

Original: Particularly, we parameterize the transformed data x ′ as ν ( x , G ) , where G is a Gaussian distribution dynamically changed along with the RL training.
Revised: Particularly, we parameterize the transformed data x ′ as ν ( x , G ) , where G is a Gaussian distribution dynamically changed along with the RL training.

Original: In Algorithm 1, it defines the MDP M with Gaussian random variables G 0 ∼ G |O| for initialization.
Revised: In Algorithm 1, it defines the MDP M with Gaussian random variables G 0 ∼ G |O| for initialization.

Original: The TRANSFORM operator is fulfilled by the aforementioned pixel transformation ν which is a shift subject following Gaussian distribution G t ( µ t , σ t ) on the top of data interpolation.
Revised: The TRANSFORM operator is fulfilled by the aforementioned pixel transformation ν which is a shift subject following Gaussian distribution G t ( µ t , σ t ) on the top of data interpolation.

Original: The transformation x ′ =
Revised: The transformation x ′ =

Original: ν ( x , G ) is normalized according to both x and learned distribution G , and thereby contributes to the cumulative reward maximization in an interactive way.
Revised: ν ( x , G ) is normalized according to both x and learned distribution G , and thereby contributes to the cumulative reward maximization in an interactive way.

Original: We use the scope to the observation sampling for an optimal state abstraction.
Revised: We use the scope to the observation sampling for an optimal state abstraction.

Original: It can be regarded as data sampling from the replay buffer to reach a training-friendly distribution, dubbed Contrastive Invariant Transformation .
Revised: It can be regarded as data sampling from the replay buffer to reach a training-friendly distribution, dubbed Contrastive Invariant Transformation .

Original: S TABILIZING R EWARD F UNCTION
Revised: S TABILIZING R EWARD F UNCTION

Original: It identifies one of the key impacts that CoIT in the RL training procedure as parametrizing the underlying invariant optimization to smooth the distribution D in the replay buffer.
Revised: It identifies one of the key impacts that CoIT in the RL training procedure as parametrizing the underlying invariant optimization to smooth the distribution D in the replay buffer.

Original: To further stabilize the reward function, we propose a mixed CoIT that samples multiple transformed data from the learned distribution G ( µ, σ ) , and then mix up the learned hidden state s ′ .
Revised: To further stabilize the reward function, we propose a mixed CoIT that samples multiple transformed data from the learned distribution G ( µ, σ ) , and then mix up the learned hidden state s ′ .

Original: Similarly, we provide the invariant learning guarantee by optimizing the right side transformation in Theorem 4.2.
Revised: Similarly, we provide the invariant learning guarantee by optimizing the right side transformation in Theorem 4.2.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: (Mixed CoIT)
Revised: (Mixed CoIT)

Original: Suppose that Lipschitzness holds for functions g ξ , g ¯ ξ , f ξ and f ¯ ξ , respectively.
Revised: Suppose that Lipschitzness holds for functions g ξ , g ¯ ξ , f ξ and f ¯ ξ , respectively.

Original: The updating dynamics is: ¯ ξ t = (1 − τ m )¯ ξ t − 1 + τ m ξ t .
Revised: The updating dynamics is: ¯ ξ t = (1 − τ m )¯ ξ t − 1 + τ m ξ t .

Original: For any input x ∼ ˆ D and shifted x ′ ∼ G , the divergence with mixed transformed states can be bound by,
Revised: For any input x ∼ ˆ D and shifted x ′ ∼ G , the divergence with mixed transformed states can be bound by,

Original: [Equation]
Revised: [Equation]

Original: where ρ =
Revised: where ρ =

Original: L g ( CL f + ∥ ξ f ∥ ) , C = 1+ τ 1 − τ , τ = 1 − τ m .
Revised: L g ( CL f + ∥ ξ f ∥ ) , C = 1+ τ 1 − τ , τ = 1 − τ m .

Original: L f and L g are Lipschitz constants of the functions f ( s ) and g ( x ) respectively.
Revised: L f and L g are Lipschitz constants of the functions f ( s ) and g ( x ) respectively.

Original: The proof of Theorem 4.2 is straightforward based on Theorem 4.1, and the details are presented in the supplement.
Revised: The proof of Theorem 4.2 is straightforward based on Theorem 4.1, and the details are presented in the supplement.

Original: 4. L EARNING C ONTRASTIVE I NVARIANT T RANSFORMATION
Revised: 4. L EARNING C ONTRASTIVE I NVARIANT T RANSFORMATION

Original: With theoretical analysis of invariant transformations, we presented a new framework with normalization variants to ensure above discussed learning guarantees by optimizing parameters.
Revised: With theoretical analysis of invariant transformations, we presented a new framework with normalization variants to ensure above discussed learning guarantees by optimizing parameters.

Original: We initialize a distribution G t ( µ, σ ) and use the TRANSFORM operator to produce different views of x t (Algorithm 1).
Revised: We initialize a distribution G t ( µ, σ ) and use the TRANSFORM operator to produce different views of x t (Algorithm 1).

Original: The transformed data x ′ t is viewed as the positive pair of x t .
Revised: The transformed data x ′ t is viewed as the positive pair of x t .

Original: We also utilize a similarity metric d to learn contrastive invariant transformation for the encoder g ξ ( · ) .
Revised: We also utilize a similarity metric d to learn contrastive invariant transformation for the encoder g ξ ( · ) .

Original: We first apply the bilinear interpolation to x t and sample shift terms from G t to produce multiple positive samples x 1 t , x 2 t , .., x nt and mix them together as x ′ t following Theorem 4.2.
Revised: We first apply the bilinear interpolation to x t and sample shift terms from G t to produce multiple positive samples x 1 t , x 2 t , .., x nt and mix them together as x ′ t following Theorem 4.2.

Original: To prevent the distribution of transformed data from shifting too far away from the replay buffer D , we borrow a similar idea from (50) to regularize and smooth the distribution shift between the transformed and overall data.
Revised: To prevent the distribution of transformed data from shifting too far away from the replay buffer D , we borrow a similar idea from (50) to regularize and smooth the distribution shift between the transformed and overall data.

Original: We use the statistical data stored in the BatchNorm layers to approximate the distribution of the overall data.
Revised: We use the statistical data stored in the BatchNorm layers to approximate the distribution of the overall data.

Original: In this way, the distribution shift between the transformed and overall data can be estimated by the following formulation,
Revised: In this way, the distribution shift between the transformed and overall data can be estimated by the following formulation,

Original: [Equation]
Revised: [Equation]

Original: where ˜ µ ( x ′ t ) and ˜ σ 2 ( x ′ t ) are the mean and variance of the transformed data and ω represents parameter collection { µ t , σ t } of the Gaussian distribution G t ( µ, σ ) .
Revised: where ˜ µ ( x ′ t ) and ˜ σ 2 ( x ′ t ) are the mean and variance of the transformed data and ω represents parameter collection { µ t , σ t } of the Gaussian distribution G t ( µ, σ ) .

Original: The expectation terms E (˜ µ l ( x ) |O ) and E (˜ σ 2 l ( x ) |O ) respectively denote the estimation of the batch-wise mean and variance for the feature map corresponding to the l -th convolution layer, and O is the given observations.
Revised: The expectation terms E (˜ µ l ( x ) |O ) and E (˜ σ 2 l ( x ) |O ) respectively denote the estimation of the batch-wise mean and variance for the feature map corresponding to the l -th convolution layer, and O is the given observations.

Original: Second, we utilize the similarity metric d proposed by Chen et al.
Revised: Second, we utilize the similarity metric d proposed by Chen et al.

Original: (9) for learning the encoder g ξ ( · ) which maps high-dimensional observation to embeddings to meet the invariant transformation in
Revised: (9) for learning the encoder g ξ ( · ) which maps high-dimensional observation to embeddings to meet the invariant transformation in

Original: Eq.5.
Revised: Eq.5.

Original: Given a positive observation pair ( x t , x + t ′ ) , the loss is given by
Revised: Given a positive observation pair ( x t , x + t ′ ) , the loss is given by

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] Next, we update the critic network Q θ with transformed data x ′ t and x ′ t + n to minimize the TD error for n -steps returns.
Revised: Next, we update the critic network Q θ with transformed data x ′ t and x ′ t + n to minimize the TD error for n -steps returns.

Original: This is regarded as a regularized Q learning by Yarats et al.
Revised: This is regarded as a regularized Q learning by Yarats et al.

Original: (48) where the regularized representation learning is beneficial for optimal action taking (53).
Revised: (48) where the regularized representation learning is beneficial for optimal action taking (53).

Original: [Equation]
Revised: [Equation]

Original: Eventually, we give the unified objective function as the full version of the CoIT,
Revised: Eventually, we give the unified objective function as the full version of the CoIT,

Original: [Equation]
Revised: [Equation]

Original: where α and λ are hyper-parameters.
Revised: where α and λ are hyper-parameters.

Original: We replace the vanilla Q-learning by J θ,ξ,ω ( D ) during training and the entire algorithm is presented in Algorithm 2 in Appendix B.
Revised: We replace the vanilla Q-learning by J θ,ξ,ω ( D ) during training and the entire algorithm is presented in Algorithm 2 in Appendix B.

Original: Then, we evaluate CoIT on popular benchmarks to demonstrate the benefits of this invariant transformation.
Revised: Then, we evaluate CoIT on popular benchmarks to demonstrate the benefits of this invariant transformation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we benchmark our method on the DeepMind control suite and Atari100K.
Revised: In this section, we benchmark our method on the DeepMind control suite and Atari100K.

Original: We compare
Revised: We compare

Original: CoIT with prior model-free methods first, then we present ablation studies to show the details of our method.
Revised: CoIT with prior model-free methods first, then we present ablation studies to show the details of our method.

Original: Implementation details can be found in Appendix C
Revised: Implementation details can be found in Appendix C

Original: 5. E NVIRONMENTS
Revised: 5. E NVIRONMENTS

Original: DMControl.
Revised: DMControl.

Original: DeepMind control suite (41) is a widely used benchmark with several robot control tasks.
Revised: DeepMind control suite (41) is a widely used benchmark with several robot control tasks.

Original: Each episode is set to be 1 , 000 frames and we use the total experienced frames to measure the data-efficiency.
Revised: Each episode is set to be 1 , 000 frames and we use the total experienced frames to measure the data-efficiency.

Original: The per-frame reward is in the unit interval [0 , 1] , so each episode contains a total reward of no more than 1 , 000 .
Revised: The per-frame reward is in the unit interval [0 , 1] , so each episode contains a total reward of no more than 1 , 000 .

Original: Considering the different difficulties depending on tasks, we refer to setting more episodes with hard tasks for better evaluation.
Revised: Considering the different difficulties depending on tasks, we refer to setting more episodes with hard tasks for better evaluation.

Original: Atari100K. There have been numbers of prior papers that have benchmarked data-efficiency on the
Revised: Atari100K. There have been numbers of prior papers that have benchmarked data-efficiency on the

Original: Atari 2600 Games for discrete control.
Revised: Atari 2600 Games for discrete control.

Original: Van Hasselt et al.
Revised: Van Hasselt et al.

Original: (44) and Kielak (25) propose data-efficient version of Rainbow DQN (21) compared with human performance (24) within 100K time steps (400K frames, frame skip of 4).
Revised: (44) and Kielak (25) propose data-efficient version of Rainbow DQN (21) compared with human performance (24) within 100K time steps (400K frames, frame skip of 4).

Original: This sample-constrained evaluation is so-called Atari100K
Revised: This sample-constrained evaluation is so-called Atari100K

Original: and we benchmark CoIT on all the 26 Atari Games.
Revised: and we benchmark CoIT on all the 26 Atari Games.

Original: 5. B ASELINES
Revised: 5. B ASELINES

Original: DMControl.
Revised: DMControl.

Original: For continuous control we present several baselines, including methods of using data augmentation and contrastive learning to improve data-efficiency: (i) DrQ-v2 (49), (ii) CURL (27),
Revised: For continuous control we present several baselines, including methods of using data augmentation and contrastive learning to improve data-efficiency: (i) DrQ-v2 (49), (ii) CURL (27),

Original: [Table]
Revised: [Table]

Original: Vanilla SAC and DDPG training directly from pixels, All methods are evaluated with the same periodicity of frames and average over 10 episodes return for evaluation query.
Revised: Vanilla SAC and DDPG training directly from pixels, All methods are evaluated with the same periodicity of frames and average over 10 episodes return for evaluation query.

Original: Atari100K.
Revised: Atari100K.

Original: To benchmark the data-efficiency of CoIT for discrete control tasks, we compare our method to (i) DrQ (48), (ii) CURL (27), (iii) SPR (38), (iv) Random Agent and Human Performance (24), All the algorithms are evaluated within 100K time steps for interaction.
Revised: To benchmark the data-efficiency of CoIT for discrete control tasks, we compare our method to (i) DrQ (48), (ii) CURL (27), (iii) SPR (38), (iv) Random Agent and Human Performance (24), All the algorithms are evaluated within 100K time steps for interaction.

Original: We average CoIT’s performance over 10 random seeds and report the best score for each game following prior works.
Revised: We average CoIT’s performance over 10 random seeds and report the best score for each game following prior works.

Original: 5. M AIN R ESULTS
Revised: 5. M AIN R ESULTS

Original: DMControl.
Revised: DMControl.

Original: We choose 8 complex tasks from the DMC for evaluation and present the results in Figure 3 and Table 2 in Appendix C. We also report the percentage (%) of score solved in the
Revised: We choose 8 complex tasks from the DMC for evaluation and present the results in Figure 3 and Table 2 in Appendix C. We also report the percentage (%) of score solved in the

Original: DMControl for baselines and CoIT in 500K and 100K steps in Figure 1.
Revised: DMControl for baselines and CoIT in 500K and 100K steps in Figure 1.

Original: Below are key findings: (i)
Revised: Below are key findings: (i)

Original: CoIT outperforms vanilla DDPG and SAC in a wide range.
Revised: CoIT outperforms vanilla DDPG and SAC in a wide range.

Original: (ii) We also compare CoIT with DrQ-v2, a remarkable method for continuous control, to better demonstrates the data-efficiency of our method.
Revised: (ii) We also compare CoIT with DrQ-v2, a remarkable method for continuous control, to better demonstrates the data-efficiency of our method.

Original: (iii) From general trends of the learning curves, CoIT improves or keeps the data-efficiency in a more stable manner which is not trivial on DM control tasks.
Revised: (iii) From general trends of the learning curves, CoIT improves or keeps the data-efficiency in a more stable manner which is not trivial on DM control tasks.

Original: Atari100K. We present results for Atari100K in Table 1 and below are key findings: (i) CoIT achieves top-performance on 10 of 26 games while still being competitive in the rest.
Revised: Atari100K. We present results for Atari100K in Table 1 and below are key findings: (i) CoIT achieves top-performance on 10 of 26 games while still being competitive in the rest.

Original: (ii) CoIT surpasses superhuman performance on 6 games .
Revised: (ii) CoIT surpasses superhuman performance on 6 games .

Original: (iii)
Revised: (iii)

Original: We also report the mean and std of the scores achieved by CoIT in 10 of 26 games which are top-performance.
Revised: We also report the mean and std of the scores achieved by CoIT in 10 of 26 games which are top-performance.

Original: We present the results of two versions of CoIT (mixed & no-mixed) in Figure 7 in the appendix.
Revised: We present the results of two versions of CoIT (mixed & no-mixed) in Figure 7 in the appendix.

Original: From the histogram, we find that
Revised: From the histogram, we find that

Original: CoIT has much better stability, which is similar to the observation in the DMControl.
Revised: CoIT has much better stability, which is similar to the observation in the DMControl.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: We first visualize the TRANSFORM operator to demonstrate that there exists an invariant transformation for each task.
Revised: We first visualize the TRANSFORM operator to demonstrate that there exists an invariant transformation for each task.

Original: We initialize the Gaussian distribution G t ( µ, σ ) based on the range of pixel shifts and plot the curves of the mean µ and std σ during training in the DMC in Figure 4.
Revised: We initialize the Gaussian distribution G t ( µ, σ ) based on the range of pixel shifts and plot the curves of the mean µ and std σ during training in the DMC in Figure 4.

Original: According to the curves below, we find that the mean and std converge to an interval as the training goes on.
Revised: According to the curves below, we find that the mean and std converge to an interval as the training goes on.

Original: These results demonstrate that the gaussian distribution proposed in CoIT could automatically find a TRANSFORM to smooth the distribution shift between the different views of the same observation, therefore be beneficial to the representation learning.
Revised: These results demonstrate that the gaussian distribution proposed in CoIT could automatically find a TRANSFORM to smooth the distribution shift between the different views of the same observation, therefore be beneficial to the representation learning.

Original: Then, we study the effects of different components in Eq.(11).
Revised: Then, we study the effects of different components in Eq.(11).

Original: This object function is composed of two parts: R ω ( x ′ t ) for regularization and L ξ, ¯ ξ,ω for similarity metric.
Revised: This object function is composed of two parts: R ω ( x ′ t ) for regularization and L ξ, ¯ ξ,ω for similarity metric.

Original: On this basis, we divide
Revised: On this basis, we divide

Original: CoIT into 4 versions: (i) Critic .
Revised: CoIT into 4 versions: (i) Critic .

Original: Transformation is only updated with the critic.
Revised: Transformation is only updated with the critic.

Original: (ii) X-stats & Critic .
Revised: (ii) X-stats & Critic .

Original: Transformation is updated by critic and R ω ( x ′ t ) together.
Revised: Transformation is updated by critic and R ω ( x ′ t ) together.

Original: (iii) H-dist & Critic .
Revised: (iii) H-dist & Critic .

Original: Transformation is updated by critic and L ξ, ¯ ξ,ω together.
Revised: Transformation is updated by critic and L ξ, ¯ ξ,ω together.

Original: (iv) Unified Objective .
Revised: (iv) Unified Objective .

Original: We evaluate all of these versions onrepresentative tasks from the DMC and present the results in Figure 5 in the appendix.
Revised: We evaluate all of these versions onrepresentative tasks from the DMC and present the results in Figure 5 in the appendix.

Original: Compared Critic with other variants, we demonstrate that both of the components are beneficial to the performance.
Revised: Compared Critic with other variants, we demonstrate that both of the components are beneficial to the performance.

Original: Though Critic is data-efficient on most tasks, it may fall into trivial solutions.
Revised: Though Critic is data-efficient on most tasks, it may fall into trivial solutions.

Original: To solve this issue, we utilize the regularization in Eq.(8) with similarity metric in Eq.(9) to meet the invariant transformation.
Revised: To solve this issue, we utilize the regularization in Eq.(8) with similarity metric in Eq.(9) to meet the invariant transformation.

Original: Thus the Unified Objective ’s performance leads ahead of all tasks.
Revised: Thus the Unified Objective ’s performance leads ahead of all tasks.

Original: See
Revised: See

Original: Appendix C for extra ablation studies.
Revised: Appendix C for extra ablation studies.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: A novel pixel transformation CoIT under model-free RL algorithms that significantly improves the data-efficiency and stability for visual tasks is introduced in this work.
Revised: A novel pixel transformation CoIT under model-free RL algorithms that significantly improves the data-efficiency and stability for visual tasks is introduced in this work.

Original: We theoretically analyze how the learnable transformation constrains the distribution of transformed data, and dissect its benefits to representation learning.
Revised: We theoretically analyze how the learnable transformation constrains the distribution of transformed data, and dissect its benefits to representation learning.

Original: CoIT is no need for any additional modifications to the backbone RL algorithm and is easy to implement.
Revised: CoIT is no need for any additional modifications to the backbone RL algorithm and is easy to implement.

Original: We compare CoIT to SOTA methods on popular benchmarks and certify that it gains promising performance with advanced stability.
Revised: We compare CoIT to SOTA methods on popular benchmarks and certify that it gains promising performance with advanced stability.

Original: Hopefully, contrastive invariant transformation can lead to a new branch for representation learning in RL
Revised: Hopefully, contrastive invariant transformation can lead to a new branch for representation learning in RL