Original: Palm up: Playing in the Latent Manifold for Unsupervised Pretraining
Revised: Palm up: Playing in the Latent Manifold for Unsupervised Pretraining

Original: Abstract
Revised: Abstract

Original: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence.
Revised: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence.

Original: Intelligent creatures, however, learn by interactingwith the environment, which changes the input sensory signals and the state of theenvironment.
Revised: Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment.

Original: In this work, we aim to bring the best of both worlds and proposean algorithm that exhibits an exploratory behavior whilst it utilizes large diversedatasets.
Revised: In this work, we aim to bring the best of both worlds and propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets.

Original: Our key idea is to leverage deep generative models that are pretrained onstatic datasets and introduce a dynamic model in the latent space.
Revised: Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space.

Original: The transitiondynamics simply mixes an action and a random sampled latent.
Revised: The transition dynamics simply mixes an action and a random sampled latent.

Original: It then appliesan exponential moving average for temporal persistency, the resulting latent isdecoded to image using pretrained generator.
Revised: It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator.

Original: We then employ an unsupervisedreinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.
Revised: We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.

Original: We further leverage thetemporal information of this data to pair data points as a natural supervision for representation learning.
Revised: We further leverage the temporal information of this data to pair data points as a natural supervision for representation learning.

Original: Our experiments suggest that the learned representations canbe successfully transferred to downstream tasks in both vision and reinforcementlearning domains.
Revised: Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains.

Original:  Introduction
Revised:  Introduction

Original: Large and diverse datasets have been the cornerstones of many impressive successes at the frontier ofartificial intelligence, such as protein folding [66, 28], image recognition [57, 13] and understandingnatural language [6, 12].
Revised: Large and diverse datasets have been the cornerstones of many impressive successes at the frontier of artificial intelligence, such as protein folding [64, 28], image recognition [55, 13] and understanding natural language [6, 12].

Original: Training machine learning models on diverse datasets that cover a breadthof human-written text and natural images often dramatically improves performance and enablesimpressive generalization capabilities [16, 6, 57, 12].
Revised: Training machine learning models on diverse datasets that cover a breadth of human-written text and natural images often dramatically improves performance and enables impressive generalization capabilities [16, 6, 55, 12].

Original: These models are learned from a fixed set ofimages, videos or languages, guided by supervision that comes from ground-truth labels [13, 57],self-supervised contrastive learning [10], or masked token prediction [6, 12] among other approaches.
Revised: These models are learned from a fixed set of images, videos, or languages, guided by supervision that comes from ground-truth labels [13, 55], self-supervised contrastive learning [10] or masked token prediction [6, 12] among other approaches.

Original: In contrast, intelligent creatures learn to perform interactions that actively change the input sensorysignals and the state of the environment towards a desired configuration.
Revised: In contrast, intelligent creatures learn to perform interactions that actively change the input sensory signals and the state of the environment towards a desired configuration.

Original: For example, in psychology,it has been shown that interaction with the environment is vital for developing flexible and generalizeintelligence [23, 76, 17].
Revised: For example, in psychology, it has been shown that interaction with the environment is vital for developing flexible and generalize intelligence [23, 74, 17].

Original: During the first few months of interactions, infants develop meaningfulunderstandings about objects [69] and prefer to look at exemplars from a novel class (e.g., dogs) afterobserving exemplars from a different class (e.g., cats) [56].
Revised: During the first few months of interactions, infants develop meaningful understandings about objects [67] and prefer to look at exemplars from a novel class (e.g., dogs) after observing exemplars from a different class (e.g., cats) [54].

Original: We hypothesis that by situating the agentin a high semantic complexity environment, the agent can develop interesting cognition abilities.
Revised: We hypothesis that by situating the agent in a high semantic complexity environment, the agent can develop interesting cognition abilities.

Original: Given the effectiveness of learning from large and diverse datasets and the significance of interactivelearning behaviors in intelligent creatures, it is therefore extremely important to connect them to buildbetter learning algorithms.
Revised: Given the effectiveness of learning from large and diverse datasets and the significance of interactive learning behaviors in intelligent creatures, it is therefore extremely important to connect them to build better learning algorithms.

Original: To bridge the gap, perhaps one straightforward solution is learning in the diverse real world [1, 55],however, teaching a robot to interact in the physical world is both time consuming and resourceintensive, and it is also difficult to scale up.
Revised: To bridge the gap, perhaps one straightforward solution is learning in the diverse real world [1, 53], however, teaching a robot to interact in the physical world is both time consuming and resource intensive, and it is also difficult to scale up.

Original: An alternative is learning in simulated environmentswhich have got surged interest recently.
Revised: An alternative is learning in simulated environments which have got surged interest recently.

Original: There simulators, such as Habitat [63, 72], RLBench [27],
Revised: There simulators, such as Habitat [61, 70], RLBench [27],

Original: House3D [79], and AI2THOR [36] enable the agent to interact with its environment.
Revised: House3D [77], and AI2THOR [36] enable the agent to interact with its environment.

Original: However, thevisual complexity of these simulated environments is far from matching the intricate real world.
Revised: However, the visual complexity of these simulated environments is far from matching the intricate real world.

Original: Thekey limitation is that making hand-designed simulation that is close enough to what a camera in thereal-world would capture is both challenging and tedious.
Revised: The key limitation is that making hand-designed simulation that is close enough to what a camera in the real-world would capture is both challenging and tedious.

Original: To remedy the issue, we propose a conceptually simple yet effective method that leverages existingdiverse datasets, builds an environment with high semantic complexity from them, and then performsinteractive learning in this environment.
Revised: To remedy the issue, we propose a conceptually simple yet effective method that leverages existing diverse datasets, builds an environment with high semantic complexity from them, and then performs interactive learning in this environment.

Original: We do so by leveraging deep generative models that aretrained in static datasets and introduce transition dynamics in the latent space of the generative model.
Revised: We do so by leveraging deep generative models that are trained in static datasets and introduce transition dynamics in the latent space of the generative model.

Original: Specifically, at each time step, the transition dynamics simply mix action and a random sampledlatent.
Revised: Specifically, at each time step, the transition dynamics simply mix action and a random sampled latent.

Original: It then applies an exponential moving average for temporal persistency, imitating the prevalenttemporal persistency in the real world.
Revised: It then applies an exponential moving average for temporal persistency, imitating the prevalent temporal persistency in the real world.

Original: Finally, the resulting latent is decoded to an image using atrained generator.
Revised: Finally, the resulting latent is decoded to an image using a trained generator.

Original: The generator is a conditional generative model which is conditioned on a prompte.g. a class label sampled at the beginning of an episode for achieving further temporal persistency.
Revised: The generator is a conditional generative model which is conditioned on a prompt e.g. a class label sampled at the beginning of an episode for achieving further temporal persistency.

Original: For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity,although our method is not restricted to it, and can be also applied to other generative models such aslanguage conditioned model Dalle [58].
Revised: For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity, although our method is not restricted to it, and can be also applied to other generative models such as language conditioned model Dalle [56].

Original: We employ unsupervised reinforcement learning (RL) [42] to explore this environment motivatedby imitating how intelligent creatures acquire perception and action skills by curiosity [23, 35,62].
Revised: We employ unsupervised reinforcement learning (RL) [41] to explore this environment motivated by imitating how intelligent creatures acquire perception and action skills by curiosity [23, 35, 60].

Original: Specifically, we use the nonparametric entropy maximization method named APT [45] whichencourages the agent to actively explore the environment to seek novel and unseen observations.
Revised: Specifically, we use the nonparametric entropy maximization method named APT [44] which encourages the agent to actively explore the environment to seek novel and unseen observations.

Original: Similar to other pixel-based unsupervised RL methods, APT learns an abstract representation byusing off-the-shelf data augmentation and contrastive learning techniques from vision [37, 41, 40].
Revised: Similar to other pixel-based unsupervised RL methods, APT learns an abstract representation by using off-the-shelf data augmentation and contrastive learning techniques from vision [37, 40, 39].

Original: While effective, designing these techniques requires domain knowledge.
Revised: While effective, designing these techniques requires domain knowledge.

Original: We show that by simplyleveraging the temporal nature, representation can be effectively learned.
Revised: We show that by simply leveraging the temporal nature, representation can be effectively learned.

Original: We do so by maximizingthe similarity between representations of current and next observations based on siamese network [5]without needing to use domain knowledge or data augmentation.
Revised: We do so by maximizing the similarity between representations of current and next observations based on siamese network [5] without needing to use domain knowledge or data augmentation.

Original: Our method is named as p l a ying inthe l atent m anifold for unsupervised pretraining (PALM).
Revised: Our method is named as p l a ying in the l atent m anifold for unsupervised pretraining (PALM).

Original: We conduct experiments in CIFAR classification and out-of-distribution detection by transferringour unsupervised exploratory pretrained representations in StyleGAN-based environments.
Revised: We conduct experiments in CIFAR classification and out-of-distribution detection by transferring our unsupervised exploratory pretrained representations in StyleGAN-based environments.

Original: Ourexperiments show that the learned representations achieve competitive results with state-of-theart methods in image recognition and out-of-distribution detection despite being only trained insynthesized data without data augmentation.
Revised: Our experiments show that the learned representations achieve competitive results with state-of-theart methods in image recognition and out-of-distribution detection despite being only trained in synthesized data without data augmentation.

Original: We also train StyleGAN in observation data collectedfrom Atari and apply our method to it.
Revised: We also train StyleGAN in observation data collected from Atari and apply our method to it.

Original: We found that the learned representation helped in maximizingmany Atari game rewards.
Revised: We found that the learned representation helped in maximizing many Atari game rewards.

Original: Our major contributions are summarized below:
Revised: Our major contributions are summarized below:

Original: • We present a surprisingly simple yet effective approach to leverage generative models as an interactive environment for unsupervised RL.
Revised: • We present a surprisingly simple yet effective approach to leverage generative models as an interactive environment for unsupervised RL.

Original: By doing so, we connect vision datasets with RL, and enable learning representation by actively interacting with the environment.
Revised: By doing so, we connect vision datasets with RL, and enable learning representation by actively interacting with the environment.

Original: • We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent to learn representations from a synthetic environment without data augmentations.
Revised: • We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent to learn representations from a synthetic environment without data augmentations.

Original: • We show that PALM matches SOTA self-supervised representation learning methods on CIFAR and out-of-distribution benchmarks.
Revised: • We show that PALM matches SOTA self-supervised representation learning methods on CIFAR and out-of-distribution benchmarks.

Original: • We show that PALM outperforms strong model-free and model-based training from scratchRL.
Revised: • We show that PALM outperforms strong model-free and model-based training from scratch RL.

Original: It also achieves competitive scores as SOTA exploratory pre-training RL and offline-datapretraining RL methods.
Revised: It also achieves competitive scores as SOTA exploratory pre-training RL and offline-data pretraining RL methods.

Original:  Related work
Revised:  Related work

Original: Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on thetarget task has been explored in reinforcement learning to improve downstream task performance.
Revised: Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on the target task has been explored in reinforcement learning to improve downstream task performance.

Original: One common approach has been to allow the agent a period of fully-unsupervised interaction with theenvironment, during which the agent is trained to maximize a surrogate exploration-based task suchas the diversity of the states it encounters [45, 44, 80].
Revised: One common approach has been to allow the agent a period of fully-unsupervised interaction with the environment, during which the agent is trained to maximize a surrogate exploration-based task such as the diversity of the states it encounters [44, 43, 78].

Original: Others have proposed to use self-supervisedobjectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of aninverse dynamics model [54, 7].
Revised: Others have proposed to use self-supervised objectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of an inverse dynamics model [52, 7].

Original: SGI [65] combines forward predictive representation learning [64]with inverse dynamics model [54] and demonstrate the power of representation pretraining for downstream RL tasks.
Revised: SGI [63] combines forward predictive representation learning [62] with inverse dynamics model [52] and demonstrate the power of representation pretraining for downstream RL tasks.

Original: Massive-scale unsupervised pretraining has shown strong results [8].
Revised: Massive-scale unsupervised pretraining has shown strong results [8].

Original: Laskin et al.[42] conducted a comparison of different unsupervised pretraining reinforcement learning algorithms.
Revised: Laskin et al. [41] conducted a comparison of different unsupervised pretraining reinforcement learning algorithms.

Original: Finally, Chaplot et al.
Revised: Finally, Chaplot et al.

Original: [9], Weihs et al.
Revised: [9], Weihs et al.

Original: [77] studied training RL agent in game simulators and transferring its representation to various vision tasks.
Revised: [75] studied training RL agent in game simulators and transferring its representation to various vision tasks.

Original: Their environments are equipped with carefully chosendomain-specific reward function to guide the learning of RL agent, and the architectures of their RLagents are fairly complicated.
Revised: Their environments are equipped with carefully chosen domain-specific reward function to guide the learning of RL agent, and the architectures of their RL agents are fairly complicated.

Original: Our work differs in that we do not rely on hand-crafted simulators and renderers which require a hugeamount of domain knowledge and effort to build, instead we leverage generative models as renders.
Revised: Our work differs in that we do not rely on hand-crafted simulators and renderers which require a huge amount of domain knowledge and effort to build, instead we leverage generative models as renders.

Original: Unlike many prior work in unsupervised pretraining RL, our work does not focus on improvingtransfer performance to downstream RL tasks although it can be used for this purpose.
Revised: Unlike many prior work in unsupervised pretraining RL, our work does not focus on improving transfer performance to downstream RL tasks although it can be used for this purpose.

Original: Training with synthetic data Using deep generative models as a source of synthetic data forrepresentation learning has been studied in prior work [61, 26, 33].
Revised: Training with synthetic data Using deep generative models as a source of synthetic data for representation learning has been studied in prior work [59, 26, 33].

Original: These generative models are fitto real image datasets and produce realistic-looking images as samples.
Revised: These generative models are fit to real image datasets and produce realistic-looking images as samples.

Original: Baradad et al.
Revised: Baradad et al.

Original: [2] studiedusing data sampled from random initialized generative models to train contrastive representations.
Revised: [2] studied using data sampled from random initialized generative models to train contrastive representations.

Original: Gowal et al.
Revised: Gowal et al.

Original: [20] studied combining data sampled from pretrained generative models with real datafor adversarial training and demonstrated improved results in robustness.
Revised: [20] studied combining data sampled from pretrained generative models with real data for adversarial training and demonstrated improved results in robustness.

Original: The use of synthesized datahas been explored in reinforcement learning under the heading of domain randomization [74], where3D synthetic data is rendered under a variety of lighting conditions to transfer to real environmentswhere the lighting may be unknown.
Revised: The use of synthesized data has been explored in reinforcement learning under the heading of domain randomization [72], where 3D synthetic data is rendered under a variety of lighting conditions to transfer to real environments where the lighting may be unknown.

Original: Our approach does away with the hand crafted simulationengine entirely by making the training data diverse through unsupervised exploration.
Revised: Our approach does away with the hand crafted simulation engine entirely by making the training data diverse through unsupervised exploration.

Original: Different from them, our work focus on leveraging a generative model as an interactive environmentand eliminating the need for data augmentation.
Revised: Different from them, our work focus on leveraging a generative model as an interactive environment and learn representation without using data augmentation.

Original: Temporal persistent representation Using temporal persistent information for representationlearning has been proposed in the past with similar motivations as ours.
Revised: Temporal persistent representation Using temporal persistent information for representation learning has been proposed in the past with similar motivations as ours.

Original: It has been used in learningrepresentation from videos [3, 78, 49, 19, 14, 53] by minimizing different metrics of representationdifference over a temporal segment.
Revised: It has been used in learning representation from videos [3, 76, 48, 19, 14, 51] by minimizing different metrics of representation difference over a temporal segment.

Original: Learning persistent representation has been explored in reinforcement learning, and has been demonstrated to improve data efficiency [52, 67, 64, 81] and improvedownstream task performance [70, 65].
Revised: Learning persistent representation has been explored in reinforcement learning, and has been demonstrated to improve data efficiency [50, 65, 62, 79] and improve downstream task performance [68, 63].

Original: In relation to these prior efforts, our work studies visual representation learning from interactedexperiences based on real-world data.
Revised: In relation to these prior efforts, our work studies visual representation learning from interacted experiences based on real-world data.

Original:  Preliminary
Revised:  Preliminary

Original: Unsupervised reinforcement learning Reinforcement learning considers the problem of findingan optimal policy for an agent that interacts with an uncertain environment and collects reward peraction [71].
Revised: Unsupervised reinforcement learning Reinforcement learning considers the problem of finding an optimal policy for an agent that interacts with an uncertain environment and collects reward per action [69].

Original: The agent maximizes its cumulative reward by interacting with its environment.
Revised: The agent maximizes its cumulative reward by interacting with its environment.

Original: Formally, this problem can be viewed as a Markov decision process (MDP) defined by( S , A , T , ρ 0 , r, γ ) where S ⊆ R n s is a set of n s -dimensional states, A ⊆ R n a is a set of n adimensional actions, T : S ×
Revised: Formally, this problem can be viewed as a Markov decision process (MDP) defined by ( S , A , T , ρ 0 , r, γ ) where S ⊆ R n s is a set of n s -dimensional states, A ⊆ R n a is a set of n adimensional actions, T : S ×

Original: A × S → [0 , 1] is the state transition probability distribution.ρ 0 : S → [0 , 1] is the distribution over initial states, r : S ×
Revised: A × S → [0 , 1] is the state transition probability distribution. ρ 0 : S → [0 , 1] is the distribution over initial states, r : S ×

Original: A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.
Revised: A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.

Original: At environment states s ∈ S , the agent take actions a ∈ A ,in the (unknown) environment dynamics defined by the transition probability T ( s ′ | s, a ) , and thereward function yields a reward immediately following the action a t performed in state s t .
Revised: At environment states s ∈ S , the agent take actions a ∈ A , in the (unknown) environment dynamics defined by the transition probability T ( s ′ | s, a ) , and the reward function yields a reward immediately following the action a t performed in state s t .

Original: In valuebased reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a,state-action value function Q π ( s t , a t ) =
Revised: In valuebased reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a, state-action value function Q π ( s t , a t ) =

Original: E s t
Revised: E s t

Original: +1 ,a t +1 ,... (cid:2)(cid:80) ∞
Revised: +1 ,a t +1 ,... (cid:2)(cid:80) ∞

Original: l =

Revised: l =

Original: γ l r ( s t + l , a t + l ) (cid:3) .
Revised: γ l r ( s t + l , a t + l ) (cid:3) .

Original: A new policy can bederived from value function by acting ϵ -greedily with respect to the action values (discrete) or byusing policy gradient to maximize the value function (continuous).
Revised: A new policy can be derived from value function by acting ϵ -greedily with respect to the action values (discrete) or by using policy gradient to maximize the value function (continuous).

Original: In unsupervised reinforcement learning, the reward function is defined as some form of intrinsicreward that is agnostic to standard task-specific reward function r := r intrinsic .
Revised: In unsupervised reinforcement learning, the reward function is defined as some form of intrinsic reward that is agnostic to standard task-specific reward function r := r intrinsic .

Original: The intrinsic rewardfunction is usually constructed for a better exploration and is computed using states and actionscollected by the agent.
Revised: The intrinsic reward function is usually constructed for a better exploration and is computed using states and actions collected by the agent.

Original: Generative adversarial model Generative Adversarial Networks (GANs) [18] consider the problem of generating photo realistic images.
Revised: Generative adversarial model Generative Adversarial Networks (GANs) [18] consider the problem of generating photo realistic images.

Original: The StyleGAN [30, 32, 31] architecture is one of thestate-of-the-art in high-resolution image generation for a multitude of different natural image categories such as faces, buildings, and animals.
Revised: The StyleGAN [30, 32, 31] architecture is one of the state-of-the-art in high-resolution image generation for a multitude of different natural image categories such as faces, buildings, and animals.

Original: To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generatorarchitecture which consists of a mapping network and synthesis network.
Revised: To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generator architecture which consists of a mapping network and synthesis network.

Original: The mapping networkconverts a latent vector z ∈ Z with Z ∈ R n into an intermediate latent space w ∈ W with W ∈ R n .
Revised: The mapping network converts a latent vector z ∈ Z with Z ∈ R n into an intermediate latent space w ∈ W with W ∈ R n .

Original: The mapping network is implemented using a multilayer perceptron that typically consists of 8 layers.
Revised: The mapping network is implemented using a multilayer perceptron that typically consists of 8 layers.

Original: The resulting vector w in that intermediate latent space is then transformed using learned affinetransformations and used as an input to a synthesis network.
Revised: The resulting vector w in that intermediate latent space is then transformed using learned affine transformations and used as an input to a synthesis network.

Original: The synthesis network consists of multiple blocks that each takes three inputs.
Revised: The synthesis network consists of multiple blocks that each takes three inputs.

Original: First, they take afeature map that contains the current content information of the image that is to be generated.
Revised: First, they take a feature map that contains the current content information of the image that is to be generated.

Original: Second,each block takes a transformed representation of the vector w as an input to its style parts, followedby a normalization of the feature map.
Revised: Second, each block takes a transformed representation of the vector w as an input to its style parts, followed by a normalization of the feature map.

Original:  Method
Revised:  Method

Original: Our objective is to leverage pretrained deep generative models G :
Revised: Our objective is to leverage pretrained deep generative models G :

Original: Z × C
Revised: Z × C

Original: → S where Z denotelatents, C denote prompt or labels and S denote observations to build an interactive environment andtrain an unsupervised reinforcement learning agent in such environment for representation pretraining.
Revised: → S where Z denote latents, C denote prompt or labels and S denote observations to build an interactive environment and train an unsupervised reinforcement learning agent in such environment for representation pretraining.

Original: 4.1 Latent environment dynamics
Revised: 4.1 Latent environment dynamics

Original: The transition dynamics are designed in the latent space of StyleGAN.
Revised: The transition dynamics are designed in the latent space of StyleGAN.

Original: At the beginning of an episode,a class label or prompt c is randomly sampled, at each time step, c and a latent z t that depends on theaction and previous latent z t =
Revised: At the beginning of an episode, a class label or prompt c is randomly sampled, at each time step, c and a latent z t that depends on the action and previous latent z t =

Original: T ( a, z t − 1 ) are transformed by the synthesis network of StyleGANinto an image which serves as an observation s t =
Revised: T ( a, z t − 1 ) are transformed by the synthesis network of StyleGAN into an image which serves as an observation s t =

Original: G ( z t , c ) .
Revised: G ( z t , c ) .

Original: We note that while the environmentconditions on the label, the ground-truth information is not directly used by PALM.
Revised: We note that while the environment conditions on the label, the ground-truth information is not directly used by PALM.

Original: The transition dynamics model consists of two steps.
Revised: The transition dynamics model consists of two steps.

Original: In the first step, it combines a randomly sampledlatent z with the agent’s action to obtain a new latent z ′ =
Revised: In the first step, it combines a randomly sampled latent z with the agent’s action to obtain a new latent z ′ =

Original: T com ( z, a t ) .
Revised: T com ( z, a t ) .

Original: Although more advancedcombination methods can be used, we use simple weighted summation to combine latent and action
Revised: Although more advanced combination methods can be used, we use simple weighted summation to combine latent and action

Original: T com ( z, a t ) =
Revised: T com ( z, a t ) =

Original: αa t
Revised: αa t

Original: +
Revised: +

Original: (1 − α ) z , where a is an action generated by the policy and α ∈ [0 , 1] is ahyperparameter controls the relative importance of action and latent.
Revised: (1 − α ) z , where a is an action generated by the policy and α ∈ [0 , 1] is a hyperparameter controls the relative importance of action and latent.

Original: The second step is a transformation T ema ( z ′ ) which is designed to ensure temporal persistency of theenvironment.
Revised: The second step is a transformation T ema ( z ′ ) which is designed to ensure temporal persistency of the environment.

Original: We use a simple exponential moving average (EMA) to implement T ema .
Revised: We use a simple exponential moving average (EMA) to implement T ema .

Original: Specifically,this module generate new latent by following
Revised: Specifically, this module generate new latent by following

Original: [Equation]
Revised: [Equation]

Original: where z ′ is the combined latent and action, z t is the exponential moving average at timestep t, andβ ∈ [0 , 1] is a hyperparameter that controls how fast the environment changes in respond to agent.
Revised: where z ′ is the combined latent and action, z t is the exponential moving average at timestep t, and β ∈ [0 , 1] is a hyperparameter that controls how fast the environment changes in respond to agent.

Original: We used β = 0 .
Revised: We used β = 0 .

Original: 95 in most of our experiments per our initial experiments.
Revised: 95 in most of our experiments per our initial experiments.

Original: The high β is motivated by observations in unsupervised RL and linguistics.
Revised: The high β is motivated by observations in unsupervised RL and linguistics.

Original: It has been shown that ahighly random environment hinders the learning of unsupervised RL agents which is also known asthe "Noisy-TV" problem [7].
Revised: It has been shown that a highly random environment hinders the learning of unsupervised RL agents which is also known as the "Noisy-TV" problem [7].

Original: It also has an interesting connection to the uniform information densityhypothesis in linguistics which states that—subject to the constraints of the grammar—humans prefersentences that distribute information equally across the linguistic signal, i.e. , evenly distributedsurprise [43].
Revised: It also has an interesting connection to the uniform information density hypothesis in linguistics which states that—subject to the constraints of the grammar—humans prefer sentences that distribute information equally across the linguistic signal, i.e. , evenly distributed surprise [42].

Original: In addition to the latent z t , we also condition the generator G ( z, c ) on a random sample class labelc sampled at the beginning of an episode, resulting in further persistent consistency.
Revised: In addition to the latent z t , we also condition the generator G ( z, c ) on a random sample class label c sampled at the beginning of an episode, resulting in further persistent consistency.

Original: Finally, theenvironment dynamics can be written as T :
Revised: Finally, the environment dynamics can be written as T :

Original: = G ( T ema ( T com ( z, a )) , c ) .
Revised: = G ( T ema ( T com ( z, a )) , c ) .

Original: 4.2 Temporal persistent representation
Revised: 4.2 Temporal persistent representation

Original: Our goal is to pretrain a representation f : S → H which is a mapping that maps observations s tolower dimensional representations h , using the data (experience replay) collected by the agent.
Revised: Our goal is to pretrain a representation f : S → H which is a mapping that maps observations s to lower dimensional representations h , using the data (experience replay) collected by the agent.

Original: Dataaugmentations are key ingredients of unsupervised representation learning methods [see e.g. 10, 11].
Revised: Data augmentations are key ingredients of unsupervised representation learning methods [see e.g. 10, 11].

Original: However, designing augmentations requires domain knowledge.
Revised: However, designing augmentations requires domain knowledge.

Original: Our representation learning is instead based on temporal information.
Revised: Our representation learning is instead based on temporal information.

Original: Our key idea is that since eachtrajectory is generated by interacting with a deep generative model conditions on the same label andtemporal persistent latent, it provides natural augmentations of visual content under various changingfactors, such as occlusion and illumination.
Revised: Our key idea is that since each trajectory is generated by interacting with a deep generative model conditions on the same label and temporal persistent latent, it provides natural augmentations of visual content under various changing factors, such as occlusion and illumination.

Original: Some rollout trajectories during training are presentedin Figure 2.
Revised: Some rollout trajectories during training are presented in Figure 2.

Original: In addition, the exploratory learning behavior of the agent increases the diversity of thetrajectory.
Revised: In addition, the exploratory learning behavior of the agent increases the diversity of the trajectory.

Original: Specifically, the idea is to train the encoder f ( s ) to produce embeddings that are persistent in time inthe same trajectory, i.e. , two presentations h t = f
Revised: Specifically, the idea is to train the encoder f ( s ) to produce embeddings that are persistent in time in the same trajectory, i.e. , two presentations h t = f

Original: ( s t ) and h t +1 = f
Revised: ( s t ) and h t +1 = f

Original: ( s t +1 ) should be similar.
Revised: ( s t +1 ) should be similar.

Original: To doso, we use the Siamese network [5] and its modern training techniques from SimSiam [11].
Revised: To do so, we use the Siamese network [5] and its modern training techniques from SimSiam [11].

Original: Specially,we maximize the similarity between q ( g ( f ( s ))) and stop_gradient ( g ( f ( s ′ ))) where q and gare multilayer perceptrons.
Revised: Specially, we maximize the similarity between q ( g ( f ( s ))) and stop_gradient ( g ( f ( s ′ ))) where q and g are multilayer perceptrons.

Original: More details can be found in appendix.
Revised: More details can be found in appendix.

Original: 4.3 Unsupervised exploration
Revised: 4.3 Unsupervised exploration

Original: The objective of unsupervised exploration is to maximize the diversity of the data seen by the agent,imitating how intelligent creatures develop recognition.
Revised: The objective of unsupervised exploration is to maximize the diversity of the data seen by the agent, imitating how intelligent creatures develop recognition.

Original: An observation s t is first encoded using the encoder f to get representation h t = f
Revised: An observation s t is first encoded using the encoder f to get representation h t = f

Original: ( s t ) .
Revised: ( s t ) .

Original: The policytakes representation h t as input and generates action a t of the same dimension.
Revised: The policy takes representation h t as input and generates action a t of the same dimension.

Original: The policy distributionis a tanh Gaussian following prior work in unsupervised reinforcement learning [80, 42].
Revised: The policy distribution is a tanh Gaussian following prior work in unsupervised reinforcement learning [78, 41].

Original: We resort to maximizing the nonparametric entropy of the representation space which has been widelyused in unsupervised reinforcement learning [68, 4, 45, 80].
Revised: We resort to maximizing the nonparametric entropy of the representation space which has been widely used in unsupervised reinforcement learning [66, 4, 44, 78].

Original: By explicitly encouraging the agents tovisit diverse states, it effectively encourages the agents to explore environments, and thus learn morediverse behaviors.
Revised: By explicitly encouraging the agents to visit diverse states, it effectively encourages the agents to explore environments, and thus learn more diverse behaviors.

Original: The estimator computes the distance between each particle h i =
Revised: The estimator computes the distance between each particle h i =

Original: f ( s i ) and its k -thnearest neighbor h ⋆i .
Revised: f ( s i ) and its k -th nearest neighbor h ⋆i .

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ 2 is ℓ 2 norm.
Revised: where ∥ · ∥ 2 is ℓ 2 norm.

Original: We associate each transition from experience replay { ( s, a, s ′ ) } with anintrinsic reward given by
Revised: We associate each transition from experience replay { ( s, a, s ′ ) } with an intrinsic reward given by

Original: [Equation]
Revised: [Equation]

Original: Putting everything together, the diagram of our method is shown in Figure 1.
Revised: Putting everything together, the diagram of our method is shown in Figure 1.

Original:  Results
Revised:  Results

Original: It is important to stress that this work focuses on studying how combined exploratory pretraining andstatic datasets contribute to representation learning in artificial agents and not on developing a new,state-of-the-art, methodology for representation learning.
Revised: It is important to stress that this work focuses on studying how combined exploratory pretraining and static datasets contribute to representation learning in artificial agents and not on developing a new, state-of-the-art, methodology for representation learning.

Original: Nevertheless, to better situate our results inthe context of existing work, we provide strong baselines in our experiments.
Revised: Nevertheless, to better situate our results in the context of existing work [Figure] we provide strong baselines in our experiments [Figure]

Original:  We test PALM on image classification tasks using a ResNet-18 as encoderf .
Revised: Image classification We test PALM on image classification tasks using a ResNet-18 as encoder f .

Original: The RL agent interacts with the environment for 50M steps, and trains representation using theprocedure described in the method section.
Revised: The RL agent interacts with the environment for 50M steps, and trains representation using the procedure described in the method section.

Original: The hyperparameters of the representation learning followthe default hyperparameters of SimSiam [11] and more details can be found in the supplementalmaterial.
Revised: The hyperparameters of the representation learning follow the default hyperparameters of SimSiam [11] and more details can be found in the supplemental material.

Original: We compare our method with a series of strong methods in static datasets, including state-of-the-artcontrastive representation learning methods SimCLR [10] and SimSiam, supervised trained model,and CLIP [57] which is pretrained on a large-scale paired image-text datasets.[Figure] R a n [Figure]
Revised: We compare our method with a series of strong methods in static datasets, including state-of-the-art contrastive representation learning methods SimCLR [10] and SimSiam, supervised trained model, and CLIP [55] which is pretrained on a large-scale paired image-text datasets.

Original: o m

Original: P A L M P A L M + d a t a a u g S i m S i a m
Revised: Figure 3 shows the linear classification performance of each method, PALM is surprisingly effective given the lack of data augmentation and real data.

Original: S i m C L

Original: R C L I P S u p e r
Revised: Note that the baselines do not have an option not to use data augmentation.

Original: v i s e d [Figure] P [Figure]
Revised: Additionally, adding the same data augmentation techniques used in SimSiam to our method further improves the results, matching or outperforming current SOTA static representation learning algorithms.

Original: M

Original: [Figure] a t a a u g S i m S i a 
Revised: Compared with the SOTA method SimSiam, PALM + data augmentation actively explore an environment instead of using a static dataset

Original: [Figure]
Revised: [Table]

Original: S i m C L

Original: R

Original: C L I P [Figure] r v i s e d Figure 3 shows the linear classification performance of each method, PALM is surprisingly effectivegiven the lack of data augmentation and real data.
Revised: and the representation learning and data augmentation are applied to synthesized temporal pair instead of two views of the same image

Original: Note that the baselines do not have an option notto use data augmentation. Additionally, adding the same data augmentation techniques used in SimSiam to our method furtherimproves the results, matching or outperforming current SOTA static representation learning algorithms. Compared with the SOTA method SimSiam, PALM + data augmentation actively explore anenvironment instead of using a static dataset, and the representation learning and data augmentationare applied to synthesized temporal pair instead of two views of the same image.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Out-of-distribution detection While some prior work demonstrated that self-supervised learningapproaches significantly improve OOD detection performance [25, 15], their self-supervised pre-trainheavily relies on domain-specific data augmentations, we want to study how PALM performs on
Revised: of-distribution detection While some prior work demonstrated that self-supervised learning approaches significantly improve OOD detection performance [25, 15], their self-supervised pre-train heavily relies on domain-specific data augmentations, we want to study how PALM performs on

Original: OOD benchmarks.
Revised: OOD benchmarks.

Original: We compare PALM with standard supervised model [22, 24] and contrastive representation learningbased methods including SupCLR [34] and CSI [73].
Revised: We compare PALM with standard supervised model [22, 24] and contrastive representation learning based methods including SupCLR [34] and CSI [71].

Original: CSI demonstrates state-of-the-art results bycombining contrastive representation learning and supervised learning.
Revised: CSI demonstrates state-of-the-art results by combining contrastive representation learning and supervised learning.

Original: Table 1 shows the comparisonon multiple datasets.
Revised: Table 1 shows the comparison on multiple datasets.

Original: PALM outperforms CSI in 4 out of 5 out-of-the-distribution datasets, despiteusing only synthesized data and no data augmentation.
Revised: PALM outperforms CSI in 4 out of 5 out-of-the-distribution datasets, despite using only synthesized data and no data augmentation.

Original: We attribute the effectiveness of PALM on
Revised: We attribute the effectiveness of PALM on

Original: OOD detection to active exploration stimulates the model to learn the invariance of data.
Revised: OOD detection to active exploration stimulates the model to learn the invariance of data.

Original: In doing so,the representation is optimized for distinguishing different observations.
Revised: In doing so, the representation is optimized for distinguishing different observations.

Original: Data efficient representation in RL In this experiment, we are interested in testing PALM pretrained model for downstream reinforcement learning tasks.
Revised: Data efficient representation in RL In this experiment, we are interested in testing PALM pre-trained model for downstream reinforcement learning tasks.

Original: To do so, we focus our experimentation on the Atari 100k benchmark introduced by Kaiser et al.
Revised: To do so, we focus our experimentation on the Atari 100k benchmark introduced by Kaiser et al.

Original: [29], in which agents are allowed only 100ksteps of interaction with their environment. 
Revised: [29], in which agents are allowed only 100k steps of interaction with their environment. [Table] [Table]

Original: There has been extensive usage of Atari for representationlearning and exploratory pretraining [8, 45, 65, 21].
Revised: Exploratory [Table] + [Table]

Revised: 250M

Revised: ATC

Revised: [Table]

Revised: [Table]

Revised: 3M

Revised: SGI

Revised: [Table]

Revised: [Table]

Revised: 6M

Revised: Offline-data Pretraining + Finetuning

Revised: ATC 0.

Revised: 3M

Revised: SGI 0.

Revised: 6M

Revised: Synthesized Exploratory Pretraining + Finetuning

Revised: PALM (ours) 0.

Original:  Finetuning after exploratory pretraining is highlyeffective [e.g., 65, 8], providing strong baselines to compare to.
Revised: 50M + 6M There has been extensive usage of Atari for representation learning and exploratory pretraining [8, 44, 63, 21]. Finetuning after exploratory pretraining is highly effective [e.g., 63, 8], providing strong baselines to compare to.

Original: Unlike experiments in CIFAR datasets, here we train an unconditional StyleGAN model in a datasetcollected by running a double DQN [48, 75] agent.
Revised: Unlike experiments in CIFAR datasets, here we train an unconditional StyleGAN model in a dataset collected by running a double DQN [47, 73] agent.

Original: The dataset consists of 50M frames for eachgame.
Revised: The dataset consists of 50M frames for each game.

Original: We use 6M steps for exploratory pretraining in the synthesized environment and we utilize DrQ [37] for online finetuning.[Table]
Revised: We use 6M steps for exploratory pretraining in the synthesized environment and we utilize DrQ [37] for online finetuning.

Original: Table 2 shows the comparison of
Revised: Table 2 shows the comparison of

Original: PALM with baselines.
Revised: PALM with baselines.

Original: Comparingwith exploratory pretraining baselines(APT [45], ATC [70], and SGI [65])that are pretrained in Atari games directly, PALM achieves surprisinglygood results considering that it istrained in a simple synthesized environment.
Revised: Comparing with exploratory pretraining baselines (APT [44], ATC [68], and SGI [63]) that are pretrained in Atari games directly, PALM achieves surprisingly good results considering that it is trained in a simple synthesized environment.

Original: PALM achieves highermedium and mean score than DrQin Atari 100k despite the online finetuning of PALM is the same as DrQ, demonstrating the benefit ofpretrained representation.
Revised: PALM achieves higher medium and mean score than DrQ in Atari 100k despite the online finetuning of PALM is the same as DrQ, demonstrating the benefit of pretrained representation.

Original: Comparedwith offline-data pretraining baselines(ATC and SGI), there is still a gapbetween PALM and the state-of-thearts, which is probably due to PALM’senvironment being trained on observations only while ATC and SGI aretrained on both observations and actions in a model-based style.
Revised: Compared with offline-data pretraining baselines (ATC and SGI), there is still a gap between PALM and the state-of-the-arts, which is probably due to PALM’s environment being trained on observations only while ATC and SGI are trained on both observations and actions in a model-based style.

Original: Worth mentioning that while PALM utilizes 50M offline action-freeexperience for pretraining, the amount is less than the 250M online state-action experience used in APT.
Revised: Worth mentioning that while PALM utilizes 50M offline action-free experience for pretraining, the amount is less than the 250M online action labeled experience used in APT.

Original: We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform DrQ without data augmentation given the same amount of online experience.
Revised: We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform DrQ without data augmentation given the same amount of online experience.

Original:  Analysis
Revised:  Analysis

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Persistent environment and active interactionare important Next, we perform an ablationstudy, to study the importance of a persistentenvironment.
Revised: are important. Next, we perform an ablation study, to study the importance of a persistent environment.

Original: We aim to answer the followingquestion.
Revised: We aim to answer the following question.

Original: How important it is to have action(controlled by α ) and state (controlled by β ) inthe transition dynamics?
Revised: How important it is to have action (controlled by α ) and state (controlled by β ) in the transition dynamics?

Original: When β =0, there iseffectively no state, i.e. , its a bandit.
Revised: When β =0, there is effectively no state, i.e. , its a bandit.

Original: As shownin Figure 4, when decreasing the importance ofstates which is denoted as PALM ( α = 0 .
Revised: As shown in Figure 4, when decreasing the importance of states which is denoted as PALM ( α = 0 .

Original: 5 β =0 . 5 ), the performance decreases.
Revised: 5 β = 0 . 5 ), the performance decreases.

Original: The extremecase β = 0 shows a significant drop in accuracy.
Revised: The extreme case β = 0 shows a significant drop in accuracy.

Original: These results indicate that it is important to havetemporal persistent transition dynamics.
Revised: These results indicate that it is important to have temporal persistent transition dynamics.

Original: When α = 0 , action is ignored in transition dynamics, thisablative baseline performs significantly worse than its default α = 0 .
Revised: When α = 0 , action is ignored in transition dynamics, this ablative baseline performs significantly worse than its default α = 0 .

Original: When replacing the agent’saction with random noise which is denoted as PALM + random action, we also see a significantperformance drop.
Revised: When replacing the agent’s action with random noise which is denoted as PALM + random action, we also see a significant performance drop.

Original: These results show having active interaction is important to the results of PALM.
Revised: These results show having active interaction is important to the results of PALM.

Original: [Figure]
Revised: We present the results of more combinations of α and β in Figure 5, the results show that as α decreases from 0.95 to 0, because of the decreasing importance of action, the accuracy continues to drop. α = 0 . 95 in general performs better than α = 1 . 0 , indicates mixing a small amount of random noise with action helps exploration and improves representation learning.

Original: PALM passive PALM (random) passive PALM (ordered) static PALM
Revised: When α or β is zero, the results are significantly worse. We also see that β ∈ [0 . 5 , 0 . 75] performs significantly better than other values, showing that persistence of dynamics is crucial. [Figure]

Original: Exploratory Learning as Curriculum Learning
Revised: Exploratory Learning as Curriculum Learning.

Original: As we have seen, representations learned fromexploratory learning in a synthesized environment can be surprisingly powerful.
Revised: As we have seen, representations learned from exploratory learning in a synthesized environment can be surprisingly powerful.

Original: We are interested inhaving a perhaps preliminary study of the reasons.
Revised: We are interested in having a perhaps preliminary study of the reasons.

Original: To do so, we compare PALM with its passive version (passive PALM) and static version (static 
Revised: To do so, we compare PALM with its passive version (passive PALM), static version (static PALM), and methods proposed in Jahanian et al.

Revised: In the random variant, the positives pairs are sampled from

Original: PALM).
Revised: G ( z, c = c ′ ) (i.e. fix the class, take two random draws from z). We denote this baseline as Random Jahanian et al.

Revised: In Gaussian Jahanian et al, the positive pairs are given by z and z+w, where w is Gaussian.

Original: Static PALM is not trained in the interactive environment, instead, it is trained with realimages.
Revised: Static PALM is not trained in the interactive environment, instead, it is trained with real images.

Original: Specifically, we randomly sample two images of the same label from a dataset and use themas a temporal pair to train the representation learning of PALM.
Revised: Specifically, we randomly sample two images of the same label from a dataset and use them as a temporal pair to train the representation learning of PALM.

Revised: We note that static PALM is related to SupCon [34], but it differs in that static PALM does not use any data augmentation while SupCon leverages a combination of data augmentations.

Revised: SupCon also uses a large number of negative examples from different categories.

Original: Passive PALM is trained offline on the experience of PALM.
Revised: Passive PALM is trained offline on the experience of PALM.

Original: We first store all the online experience(50M in total) of PALM, then we use the saved experience to train passive PALM without having tointeract with the environment.
Revised: We first store all the online experience (50M in total) of PALM, then we use the saved experience to train passive PALM without having to interact with the environment.

Original: We consider two options of passive PALM.
Revised: We consider two options of passive PALM.

Original: The first one is randompassive PALM which randomly samples consecutive observations from the stored offline experience.
Revised: The first one is random passive PALM which randomly samples consecutive observations from the stored offline experience.

Original: The second one is ordered passive PALM which is trained on the same order of observations as inonline experience.
Revised: The second one is ordered passive PALM which is trained on the same order of observations as in online experience.

Original: Figure 5 shows the comparison results.

Original: With the number of data increasing, PALM keeps improvingwith the amount of training data and significantly outperforms static PALM even though static PALMis trained on real datasets.
Revised: With the number of data increasing, PALM keeps improving with the amount of training data and significantly outperforms static PALM even though static PALM is trained on real datasets.

Original: We believe the reason is that the diverse synthesized data collected by PALM can greatly help to learn representation.
Revised: We believe the reason is that the diverse synthesized data collected by PALM can greatly help to learn representation.

Original: PALM nearly always outperforms random passive
Revised: In the low data regime, carefully designed non-active learning based methods [26] outperforms PALM, possibly due to RL training being sample inefficient. With the increasing number of synthesized samples, PALM continues to improve and significantly outperform all baselines.

Original:  PALM, demonstrating the effectiveness of exploratory learning.
Revised: The results also show that PALM scales better than non active learning based methods. PALM nearly always outperforms random passive PALM, demonstrating the effectiveness of exploratory learning.

Original: Interestingly but not surprisingly,ordered passive PALM performs nearly the same as PALM, indicating the exploratory learningbehavior of PALM might play an important role in curriculum learning.
Revised: Interestingly but not surprisingly, ordered passive PALM performs nearly the same as PALM, indicating the exploratory learning behavior of PALM might play an important role in curriculum learning.

Original: The agent is optimized tosearch for novel samples in terms of their representation, by doing so, the searched samples shouldbecome more difficult to distinguish.
Revised: The agent is optimized to search for novel samples in terms of their representation, by doing so, the searched samples should become more difficult to distinguish.

Original: We believe a comprehensive understanding of the effectivenessof PALM is out of the scope of this work, we tend to leave it as future work. 
Revised: We believe a comprehensive understanding of the effectiveness of PALM is out of the scope of this work, we tend to leave it as future work. [Figure] [Table] Diversity analysis.

Original: Conclusion and discussion
Revised: In order to measure the diversity of the sampled synthetic data, we randomly sample 10K images from the CIFAR-10 train set and generate 10K images from StyleGAN by random sample latents. We also generate 10K images with PALM in latent space. For each sample in each set, we find its closest neighbor in Inception feature space (obtained after the pooling layer). The coverage is defined as the proportion of nearest neighbors that are unique in the train set.

Revised: A better active exploration method would produce samples that are equally likely to be close to any image in the train set.

Revised: PALM achieves a better coverage score than random sampling in latent space.

Revised: Although the Euclidean distance in Inception feature space is inaccurate, the better coverage achieved by PALM indicates that by exploring the latent manifold it gets more diverse data.

Original: Our work presents a method to connect learning from large, diverse static datasets and learning byinteraction, and demonstrates promising results of doing so.
Revised: Our work presents a method to connect learning from large, diverse static datasets and learning by interaction, and demonstrates promising results of doing so.

Original: We believe it opens up an interestingfuture direction for advancing both reinforcement learning and self-supervised visual representationlearning.
Revised: We believe it opens up an interesting future direction for advancing both reinforcement learning and self-supervised visual representation learning.

Original: For limitations, since it is computational expensive to draw samples from large generative model,interacting with such environment is significantly slower than well optimized physical simulationsused in RL, e.g. , Mujoco.
Revised: For limitations, since it is computational expensive to draw samples from large generative model, interacting with such environment is significantly slower than well optimized physical simulations used in RL, e.g. , Mujoco.

Original: Addressing such limitation would make this work more accessible andfacilitate future research.
Revised: Addressing such limitation would make this work more accessible and facilitate future research.

Original: For future work, it would be interesting to design or learn better transition dynamics.
Revised: For future work it would be interesting to design or learn better transition dynamics.

Revised: It would be also interesting to evaluate our method in RL settings where a lot of action-free data are available and won’t have to be collected (Minecraft for example).

Original: It would be alsointeresting to use a language conditioned generative model as an alternative e.g. Dalle [59] and Dallev2 [60], so the environment has more complexity by prompting it with natural language. 
Revised: It would be also interesting to use a language conditioned generative model as an alternative e.g. Dalle [57, 58], so the environment has more complexity by prompting it with natural language. Acknowledgements We would like to thank Guillaume Desjardins for insightful discussion and giving constructive comments. We would also like to thank anonymous reviewers for their helpful feedback

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c) Did you discuss any potential negative societal impacts of your work?[Yes]
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The instructions will be provided in the supplemental material and we plan to release the code and pre-trained models to facilitate further research.
Revised: [Yes] The instructions will be provided in the supplemental material and we plan to release the code and pre-trained models to facilitate further research.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] The full details will be provided in supplementary material.
Revised: [Yes] The full details will be provided in supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Our experiments on classification and OOD detection experiments are not sensitive to random seeds.
Revised: [Yes] Our experiments on classification and OOD detection experiments are not sensitive to random seeds.

Original: For our RL experiments, we run the evaluate multiple times and include the details in experiment section and supplementary material.
Revised: For our RL experiments, we run the evaluate multiple times and include the details in experiment section and supplementary material.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] The full details will be provided in supplementary material.
Revised: [Yes] The full details will be provided in supplementary material.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The datasets we use are public and do not contain personally identifiable information.
Revised: The datasets we use are public and do not contain personally identifiable information.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Figure] (b) Did you describe any potential participant risks, with links to Institutional Review [Figure]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Board ( ) approvals, if applicable?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Figure]
Revised: We follow the same optimization step of SimSiam [11]. The input image size is 32 × 32. We use SGD with base lr =0 .

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]
Revised: 03 and a cosine decay [46] schedule for 800 epochs, weight decay =0 . 0005 , momentum =0 . 9 , and batch size =512 . The encoder f is a

Revised: ResNet-18 [22] with the last fc layer removed.

Revised: We use the same projection and prediction networks in SimSiam.

Revised: Specifically, the projection MLP g has 3 layers, and has BN applied to each fully-connected (fc) layer, including its output fc.

Revised: There is no ReLU for the last fc layer in g and the hidden size is 2048.

Revised: The prediction MLP ( h ) has 2 layers, and has BN applied to its hidden fc layers.

Revised: Its output fc does not have BN or ReLU.

Revised: The dimension of h ’s input and output is d =2048 , and h ’s hidden layer’s dimension is 512 , making h a bottleneck structure.

Revised: After pretraining, the representation encoder f is finetuned for downstream tasks.

Revised: B Representation Learning Loss and Pseudocode

Revised: We use the symmetrized cosine similarity loss from SimSiam.

Revised: Specifically, for two consecutive observations s 1 and s 2 , denoting the two output vectors as p 1 = h

Revised: ( g ( f ( s 1 ))) and z 2 = g

Revised: ( f ( s 2 )) , we minimize their negative cosine similarity:

Revised: [Equation]

Revised: where ∥·∥ 2 is ℓ 2 -norm.

Revised: And the loss function is defined as

Original: [Figure]
Revised: [Equation]

Revised: Algorithm 1 shows the pseudocode of representation learning.

Revised: C Experiment and dataset details

Revised: Datasets CIFAR-10 [38] and CIFAR-100 [38] consist of 50,000 training and 10,000 test images with 10 and 20 (super-class) image classes, respectively.

Revised: For CIFAR-10, out-of-distribution (OOD) samples are as follows: SVHN [49] consists of 26,032 test images with 10 digits, resized LSUN [80] consists of 10,000 test images of 10 different scenes, Interp.

Revised: consists of 10,000 test images of linear interpolation of CIFAR-10 test images.

Revised: CelebA [45], a labeled dataset consisting of over 200 ,face images and each with 40 attribute annotation.

Revised: The Atari observation dataset is collected using double DQN [47, 73] with n-step return (n=3) and 3 frames stacking.

Revised: In total we collected 50M images and each observation is resized to 64x64x3.

Revised: Model details For CIFAR10, we use pretrained StyleGAN available at the official website of

Revised: StyleGAN-Ada[31] 2 .

Revised: This pretrained model is a conditional StyleGAN that achieves best FID score.

Revised: We also experimented with the model with best Inception score 3 but did not observe significant difference in results.

Revised: For CIFAR100, since we did not find publicly available pretrained models, we trained one by ourself using the StyleGAN official code 4 .

Revised: For Atari, we train an unconditional StyleGAN using the official code.

Revised: It is difficult to train StyleGAN to converge on our Atari dataset with the default hyperparameters.

Revised: We found that increasing the batch size to 256 significantly help stabilize the training and accelerate convergence.

Revised: Linear classification

Revised: The quality of the pretrained representations is evaluated by training a supervised linear classifier on frozen representations h in the training set, and then testing it in the validation set.

Revised: We use batch size of 512, Adam optimizer with learning rate being 0 . 001 .

Revised: We finetune for 100 epochs in linear classification experiments.

Revised: Atari games Our evaluation metric for an agent on a game is human-normalized score (HNS), defined as agent _ score − random _ score human _ score − random _ score .

Revised: We calculate this per game by averaging scores overevaluation trajectories at the end of training, and across 10 random seeds for training.

Revised: We report both mean (Mn) and median (Mdn) HNS over the 26 Atari-100K games, as well as on how many games a method achieves super-human performance ( > H) and greater than random performance ( > 0).

Revised: D Experiment details on RL

Revised: We adhere closely to the parameter settings from URLB [41], which our implementation is based upon.

Revised: We list the hyper-parameters in Table

Revised: We employ clipped double Q-learning for the critic, where each Q -function is parametrized as a 3-layer MLP with ReLU activations after each layer except of the last.

Revised: The actor is also a 3-layer MLP with ReLU s that outputs mean and covariance for the diagonal Gaussian that represents the policy.

Revised: The hidden dimension is set to 1024 for both the critic and actor.

Revised: The actor and critic networks both have separate encoders, although we share the weights of the conv layers between them.

Revised: Furthermore, only the critic optimizer is allowed to update these weights (e.g. we stop the gradients from the actor before they propagate to the shared conv layers).

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: log( r + 1 . 0)

Revised: [Table]

Revised: (512 + |A| )

Revised: → 1024 → 512 ReLU MLP

Revised: [Table]

Revised: (2 × 512) → 1024 → |A| ReLU MLP

Revised: [Table]

Revised: # f: backbone # g: projection mlp # h: prediction mlp for x1, x2 in loader: # load tuples of consecutive observations h1, h2 = f(x1), f(x2) # representations, n-by-d z1, z2 = f(h1), f(h2)

Revised: # projections, n-by-d p1, p2 = h(z1), h(z2)

Revised: # predictions, n-by-d

Original: {+L = D(p1, z2)/2 + D(p2, z1)/
+}
Revised: {+L = D(p1, z2)/2 + D(p2, z1)/
+}

Revised: # loss

Revised: L.backward() # back-propagate update(f, h) # SGD update def D(p, z): # negative cosine similarity z = z.detach() # stop gradient p = normalize(p, dim=1)

Revised: # l2-normalize z = normalize(z, dim=1)

Revised: # l2-normalize return -(p*z).sum(dim=1).mean()

Revised: # P: policy # G: generator # alpha, beta: hyperparameters control the importance of action and state.

Revised: for t in range(episode_length): # z = random_normal

Revised: () # sample a random latent from Gaussian if t == 0:

Revised: # first step of an episode z_t =

Revised: z c =

Revised: random_label() # sample a random class label s_t

Revised: = G(z_t, c) # generate an observation a_t = P(s_t) # sample an action from policy else: zprime = alpha *

Revised: a_t + (1 - alpha) *

Revised: z z_t = beta * z_t + (1 - beta) zprime s_t

Revised: = G(z_t, c) # generate an observation a_t = P(s_t) # sample an action from policy