Original: Palm up: Playing in the Latent Manifold for Unsupervised Pretrainings
Revised: Palm up: Playing in the Latent Manifold for Unsupervised Pretraining

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence.
Revised: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence.

Original: Intelligent creatures, however, learn by interactingwith the environment, which changes the input sensory signals and the state of theenvironment.
Revised: Intelligent creatures, however, learn by interactingwith the environment, which changes the input sensory signals and the state of theenvironment.

Original: In this work, we aim to bring the best of both worlds and proposean algorithm that exhibits an exploratory behavior whilst it utilizes large diversedatasets.
Revised: In this work, we aim to bring the best of both worlds and proposean algorithm that exhibits an exploratory behavior whilst it utilizes large diversedatasets.

Original: Our key idea is to leverage deep generative models that are pretrained onstatic datasets and introduce a dynamic model in the latent space.
Revised: Our key idea is to leverage deep generative models that are pretrained onstatic datasets and introduce a dynamic model in the latent space.

Original: The transitiondynamics simply mixes an action and a random sampled latent.
Revised: The transitiondynamics simply mixes an action and a random sampled latent.

Original: It then appliesan exponential moving average for temporal persistency, the resulting latent isdecoded to image using pretrained generator.
Revised: It then appliesan exponential moving average for temporal persistency, the resulting latent isdecoded to image using pretrained generator.

Original: We then employ an unsupervisedreinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.
Revised: We then employ an unsupervisedreinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.

Original: We further leverage thetemporal information of this data to pair data points as a natural supervision for representation learning.
Revised: We further leverage thetemporal information of this data to pair data points as a natural supervision for representation learning.

Original: Our experiments suggest that the learned representations canbe successfully transferred to downstream tasks in both vision and reinforcementlearning domains.
Revised: Our experiments suggest that the learned representations canbe successfully transferred to downstream tasks in both vision and reinforcementlearning domains.

Original:  Introduction
Revised:  Introduction

Original: Large and diverse datasets have been the cornerstones of many impressive successes at the frontier ofartificial intelligence, such as protein folding [64, 28], image recognition [55, 13] and understandingnatural language [6, 12].
Revised: Large and diverse datasets have been the cornerstones of many impressive successes at the frontier ofartificial intelligence, such as protein folding [66, 28], image recognition [57, 13] and understandingnatural language [6, 12].

Original: Training machine learning models on diverse datasets that cover a breadthof human-written text and natural images often dramatically improves performance and enablesimpressive generalization capabilities [16, 6, 55, 12].
Revised: Training machine learning models on diverse datasets that cover a breadthof human-written text and natural images often dramatically improves performance and enablesimpressive generalization capabilities [16, 6, 57, 12].

Original: These models are learned from a fixed set ofimages, videos or languages, guided by supervision that comes from ground-truth labels [13, 55],self-supervised contrastive learning [10], or masked token prediction [6, 12] among other approaches.
Revised: These models are learned from a fixed set ofimages, videos or languages, guided by supervision that comes from ground-truth labels [13, 57],self-supervised contrastive learning [10], or masked token prediction [6, 12] among other approaches.

Original: In contrast, intelligent creatures learn to perform interactions that actively change the input sensorysignals and the state of the environment towards a desired configuration.
Revised: In contrast, intelligent creatures learn to perform interactions that actively change the input sensorysignals and the state of the environment towards a desired configuration.

Original: For example, in psychology,it has been shown that interaction with the environment is vital for developing flexible and generalizeintelligence [23, 74, 17].
Revised: For example, in psychology,it has been shown that interaction with the environment is vital for developing flexible and generalizeintelligence [23, 76, 17].

Original: During the first few months of interactions, infants develop meaningfulunderstandings about objects [67] and prefer to look at exemplars from a novel class (e.g., dogs) afterobserving exemplars from a different class (e.g., cats) [54].
Revised: During the first few months of interactions, infants develop meaningfulunderstandings about objects [69] and prefer to look at exemplars from a novel class (e.g., dogs) afterobserving exemplars from a different class (e.g., cats) [56].

Original: We hypothesis that by situating the agentin a high semantic complexity environment, the agent can develop interesting cognition abilities.
Revised: We hypothesis that by situating the agentin a high semantic complexity environment, the agent can develop interesting cognition abilities.

Original: Given the effectiveness of learning from large and diverse datasets and the significance of interactivelearning behaviors in intelligent creatures, it is therefore extremely important to connect them to buildbetter learning algorithms.
Revised: Given the effectiveness of learning from large and diverse datasets and the significance of interactivelearning behaviors in intelligent creatures, it is therefore extremely important to connect them to buildbetter learning algorithms.

Original: To bridge the gap, perhaps one straightforward solution is learning in the diverse real world [1, 53],however, teaching a robot to interact in the physical world is both time consuming and resourceintensive, and it is also difficult to scale up.
Revised: To bridge the gap, perhaps one straightforward solution is learning in the diverse real world [1, 55],however, teaching a robot to interact in the physical world is both time consuming and resourceintensive, and it is also difficult to scale up.

Original: An alternative is learning in simulated environmentswhich have got surged interest recently.
Revised: An alternative is learning in simulated environmentswhich have got surged interest recently.

Original: There simulators, such as Habitat [61, 70], RLBench [27],
Revised: There simulators, such as Habitat [63, 72], RLBench [27],

Original: House3D [77], and AI2THOR [36] enable the agent to interact with its environment.
Revised: House3D [79], and AI2THOR [36] enable the agent to interact with its environment.

Original: However, thevisual complexity of these simulated environments is far from matching the intricate real world.
Revised: However, thevisual complexity of these simulated environments is far from matching the intricate real world.

Original: Thekey limitation is that making hand-designed simulation that is close enough to what a camera in thereal-world would capture is both challenging and tedious.
Revised: Thekey limitation is that making hand-designed simulation that is close enough to what a camera in thereal-world would capture is both challenging and tedious.

Original: To remedy the issue, we propose a conceptually simple yet effective method that leverages existingdiverse datasets, builds an environment with high semantic complexity from them, and then performsinteractive learning in this environment.
Revised: To remedy the issue, we propose a conceptually simple yet effective method that leverages existingdiverse datasets, builds an environment with high semantic complexity from them, and then performsinteractive learning in this environment.

Original: We do so by leveraging deep generative models that aretrained in static datasets and introduce transition dynamics in the latent space of the generative model.
Revised: We do so by leveraging deep generative models that aretrained in static datasets and introduce transition dynamics in the latent space of the generative model.

Original: Specifically, at each time step, the transition dynamics simply mix action and a random sampledlatent.
Revised: Specifically, at each time step, the transition dynamics simply mix action and a random sampledlatent.

Original: It then applies an exponential moving average for temporal persistency, imitating the prevalenttemporal persistency in the real world.
Revised: It then applies an exponential moving average for temporal persistency, imitating the prevalenttemporal persistency in the real world.

Original: Finally, the resulting latent is decoded to an image using atrained generator.
Revised: Finally, the resulting latent is decoded to an image using atrained generator.

Original: The generator is a conditional generative model which is conditioned on a prompte.g.
Revised: The generator is a conditional generative model which is conditioned on a prompte.g.

Original: a class label sampled at the beginning of an episode for achieving further temporal persistency.
Revised: a class label sampled at the beginning of an episode for achieving further temporal persistency.

Original: For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity,although our method is not restricted to it, and can be also applied to other generative models such aslanguage conditioned model Dalle [56].
Revised: For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity,although our method is not restricted to it, and can be also applied to other generative models such aslanguage conditioned model Dalle [58].

Original: We employ unsupervised reinforcement learning (RL) [41] to explore this environment motivatedby imitating how intelligent creatures acquire perception and action skills by curiosity [23, 35,60].
Revised: We employ unsupervised reinforcement learning (RL) [42] to explore this environment motivatedby imitating how intelligent creatures acquire perception and action skills by curiosity [23, 35,62].

Original: Specifically, we use the nonparametric entropy maximization method named APT [44] whichencourages the agent to actively explore the environment to seek novel and unseen observations.
Revised: Specifically, we use the nonparametric entropy maximization method named APT [45] whichencourages the agent to actively explore the environment to seek novel and unseen observations.

Original: Similar to other pixel-based unsupervised RL methods, APT learns an abstract representation byusing off-the-shelf data augmentation and contrastive learning techniques from vision [37, 40, 39].
Revised: Similar to other pixel-based unsupervised RL methods, APT learns an abstract representation byusing off-the-shelf data augmentation and contrastive learning techniques from vision [37, 41, 40].

Original: While effective, designing these techniques requires domain knowledge.
Revised: While effective, designing these techniques requires domain knowledge.

Original: We show that by simplyleveraging the temporal nature, representation can be effectively learned.
Revised: We show that by simplyleveraging the temporal nature, representation can be effectively learned.

Original: We do so by maximizingthe similarity between representations of current and next observations based on siamese network [5]without needing to use domain knowledge or data augmentation.
Revised: We do so by maximizingthe similarity between representations of current and next observations based on siamese network [5]without needing to use domain knowledge or data augmentation.

Original: Our method is named as p l a ying inthe l atent m anifold for unsupervised pretraining (PALM).
Revised: Our method is named as p l a ying inthe l atent m anifold for unsupervised pretraining (PALM).

Original: We conduct experiments in CIFAR classification and out-of-distribution detection by transferringour unsupervised exploratory pretrained representations in StyleGAN-based environments.
Revised: We conduct experiments in CIFAR classification and out-of-distribution detection by transferringour unsupervised exploratory pretrained representations in StyleGAN-based environments.

Original: Ourexperiments show that the learned representations achieve competitive results with state-of-theart methods in image recognition and out-of-distribution detection despite being only trained insynthesized data without data augmentation.
Revised: Ourexperiments show that the learned representations achieve competitive results with state-of-theart methods in image recognition and out-of-distribution detection despite being only trained insynthesized data without data augmentation.

Original: We also train StyleGAN in observation data collectedfrom Atari and apply our method to it.
Revised: We also train StyleGAN in observation data collectedfrom Atari and apply our method to it.

Original: We found that the learned representation helped in maximizingmany Atari game rewards.
Revised: We found that the learned representation helped in maximizingmany Atari game rewards.

Original: Our major contributions are summarized below:
Revised: Our major contributions are summarized below:

Original: • We present a surprisingly simple yet effective approach to leverage generative models as an
Revised: • We present a surprisingly simple yet effective approach to leverage generative models as an

Original: interactive environment for unsupervised RL.
Revised: interactive environment for unsupervised RL.

Original: By doing so, we connect vision datasets with
Revised: By doing so, we connect vision datasets with

Original: RL, and enable learning representation by actively interacting with the environment.
Revised: RL, and enable learning representation by actively interacting with the environment.

Original: • We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent
Revised: • We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent

Original: to learn representations from a synthetic environment without data augmentations.
Revised: to learn representations from a synthetic environment without data augmentations.

Original: • We show that PALM matches SOTA self-supervised representation learning methods on
Revised: • We show that PALM matches SOTA self-supervised representation learning methods on

Original: CIFAR and out-of-distribution benchmarks.
Revised: CIFAR and out-of-distribution benchmarks.

Original: • We show that PALM outperforms strong model-free and model-based training from scratchRL.
Revised: • We show that PALM outperforms strong model-free and model-based training from scratchRL.

Original: It also achieves competitive scores as SOTA exploratory pre-training RL and offline-datapretraining RL methods.
Revised: It also achieves competitive scores as SOTA exploratory pre-training RL and offline-datapretraining RL methods.

Original:  Related work
Revised:  Related work

Original: Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on thetarget task has been explored in reinforcement learning to improve downstream task performance.
Revised: Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on thetarget task has been explored in reinforcement learning to improve downstream task performance.

Original: One common approach has been to allow the agent a period of fully-unsupervised interaction with theenvironment, during which the agent is trained to maximize a surrogate exploration-based task suchas the diversity of the states it encounters [44, 43, 78].
Revised: One common approach has been to allow the agent a period of fully-unsupervised interaction with theenvironment, during which the agent is trained to maximize a surrogate exploration-based task suchas the diversity of the states it encounters [45, 44, 80].

Original: Others have proposed to use self-supervisedobjectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of aninverse dynamics model [52, 7].
Revised: Others have proposed to use self-supervisedobjectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of aninverse dynamics model [54, 7].

Original: SGI [63] combines forward predictive representation learning [62]with inverse dynamics model [52] and demonstrate the power of representation pretraining for downstream RL tasks.
Revised: SGI [65] combines forward predictive representation learning [64]with inverse dynamics model [54] and demonstrate the power of representation pretraining for downstream RL tasks.

Original: Massive-scale unsupervised pretraining has shown strong results [8].
Revised: Massive-scale unsupervised pretraining has shown strong results [8].

Original: Laskin et al.[41] conducted a comparison of different unsupervised pretraining reinforcement learning algorithms.
Revised: Laskin et al.[42] conducted a comparison of different unsupervised pretraining reinforcement learning algorithms.

Original: Finally, Chaplot et al.
Revised: Finally, Chaplot et al.

Original: [9], Weihs et al.
Revised: [9], Weihs et al.

Original: [75] studied training RL agent in game simulators and transferring its representation to various vision tasks.
Revised: [77] studied training RL agent in game simulators and transferring its representation to various vision tasks.

Original: Their environments are equipped with carefully chosendomain-specific reward function to guide the learning of RL agent, and the architectures of their RLagents are fairly complicated.
Revised: Their environments are equipped with carefully chosendomain-specific reward function to guide the learning of RL agent, and the architectures of their RLagents are fairly complicated.

Original: Our work differs in that we do not rely on hand-crafted simulators and renderers which require a hugeamount of domain knowledge and effort to build, instead we leverage generative models as renders.
Revised: Our work differs in that we do not rely on hand-crafted simulators and renderers which require a hugeamount of domain knowledge and effort to build, instead we leverage generative models as renders.

Original: Unlike many prior work in unsupervised pretraining RL, our work does not focus on improvingtransfer performance to downstream RL tasks although it can be used for this purpose.
Revised: Unlike many prior work in unsupervised pretraining RL, our work does not focus on improvingtransfer performance to downstream RL tasks although it can be used for this purpose.

Original: Training with synthetic data Using deep generative models as a source of synthetic data forrepresentation learning has been studied in prior work [59, 26, 33].
Revised: Training with synthetic data Using deep generative models as a source of synthetic data forrepresentation learning has been studied in prior work [61, 26, 33].

Original: These generative models are fitto real image datasets and produce realistic-looking images as samples.
Revised: These generative models are fitto real image datasets and produce realistic-looking images as samples.

Original: Baradad et al.
Revised: Baradad et al.

Original: [2] studiedusing data sampled from random initialized generative models to train contrastive representations.
Revised: [2] studiedusing data sampled from random initialized generative models to train contrastive representations.

Original: Gowal et al.
Revised: Gowal et al.

Original: [20] studied combining data sampled from pretrained generative models with real datafor adversarial training and demonstrated improved results in robustness.
Revised: [20] studied combining data sampled from pretrained generative models with real datafor adversarial training and demonstrated improved results in robustness.

Original: The use of synthesized datahas been explored in reinforcement learning under the heading of domain randomization [72], where3D synthetic data is rendered under a variety of lighting conditions to transfer to real environmentswhere the lighting may be unknown.
Revised: The use of synthesized datahas been explored in reinforcement learning under the heading of domain randomization [74], where3D synthetic data is rendered under a variety of lighting conditions to transfer to real environmentswhere the lighting may be unknown.

Original: Our approach does away with the hand crafted simulationengine entirely by making the training data diverse through unsupervised exploration.
Revised: Our approach does away with the hand crafted simulationengine entirely by making the training data diverse through unsupervised exploration.

Original: Different from them, our work focus on leveraging a generative model as an interactive environmentand learn representation without using data augmentation.
Revised: Different from them, our work focus on leveraging a generative model as an interactive environmentand eliminating the need for data augmentation.

Original: Temporal persistent representation Using temporal persistent information for representationlearning has been proposed in the past with similar motivations as ours.
Revised: Temporal persistent representation Using temporal persistent information for representationlearning has been proposed in the past with similar motivations as ours.

Original: It has been used in learningrepresentation from videos [3, 76, 48, 19, 14, 51] by minimizing different metrics of representationdifference over a temporal segment.
Revised: It has been used in learningrepresentation from videos [3, 78, 49, 19, 14, 53] by minimizing different metrics of representationdifference over a temporal segment.

Original: Learning persistent representation has been explored in reinforcement learning, and has been demonstrated to improve data efficiency [50, 65, 62, 79] and improvedownstream task performance [68, 63].
Revised: Learning persistent representation has been explored in reinforcement learning, and has been demonstrated to improve data efficiency [52, 67, 64, 81] and improvedownstream task performance [70, 65].

Original: In relation to these prior efforts, our work studies visual representation learning from interactedexperiences based on real-world data.
Revised: In relation to these prior efforts, our work studies visual representation learning from interactedexperiences based on real-world data.

Original:  Preliminary
Revised:  Preliminary

Original: Unsupervised reinforcement learning Reinforcement learning considers the problem of findingan optimal policy for an agent that interacts with an uncertain environment and collects reward peraction [69].
Revised: Unsupervised reinforcement learning Reinforcement learning considers the problem of findingan optimal policy for an agent that interacts with an uncertain environment and collects reward peraction [71].

Original: The agent maximizes its cumulative reward by interacting with its environment.
Revised: The agent maximizes its cumulative reward by interacting with its environment.

Original: Formally, this problem can be viewed as a Markov decision process (MDP) defined by( S , A , T , ρ 0 , r, γ ) where S ⊆ R n s is a set of n s -dimensional states, A ⊆ R n a is a set of n adimensional actions, T : S ×
Revised: Formally, this problem can be viewed as a Markov decision process (MDP) defined by( S , A , T , ρ 0 , r, γ ) where S ⊆ R n s is a set of n s -dimensional states, A ⊆ R n a is a set of n adimensional actions, T : S ×

Original: A × S → [0 , 1] is the state transition probability distribution.ρ 0 : S → [0 , 1] is the distribution over initial states, r : S ×
Revised: A × S → [0 , 1] is the state transition probability distribution.ρ 0 : S → [0 , 1] is the distribution over initial states, r : S ×

Original: A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.
Revised: A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.

Original: At environment states s ∈ S , the agent take actions a ∈ A ,in the (unknown) environment dynamics defined by the transition probability T ( s ′ | s, a ) , and thereward function yields a reward immediately following the action a t performed in state s t .
Revised: At environment states s ∈ S , the agent take actions a ∈ A ,in the (unknown) environment dynamics defined by the transition probability T ( s ′ | s, a ) , and thereward function yields a reward immediately following the action a t performed in state s t .

Original: In valuebased reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a,state-action value function Q π ( s t , a t ) =
Revised: In valuebased reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a,state-action value function Q π ( s t , a t ) =

Original: E s t
Revised: E s t

Original: +1 ,a t +1 ,... (cid:2)(cid:80) ∞
Revised: +1 ,a t +1 ,... (cid:2)(cid:80) ∞

Original: l =

Revised: l =

Original: γ l r ( s t + l , a t + l ) (cid:3) .
Revised: γ l r ( s t + l , a t + l ) (cid:3) .

Original: A new policy can bederived from value function by acting ϵ -greedily with respect to the action values (discrete) or byusing policy gradient to maximize the value function (continuous).
Revised: A new policy can bederived from value function by acting ϵ -greedily with respect to the action values (discrete) or byusing policy gradient to maximize the value function (continuous).

Original: In unsupervised reinforcement learning, the reward function is defined as some form of intrinsicreward that is agnostic to standard task-specific reward function r := r intrinsic .
Revised: In unsupervised reinforcement learning, the reward function is defined as some form of intrinsicreward that is agnostic to standard task-specific reward function r := r intrinsic .

Original: The intrinsic rewardfunction is usually constructed for a better exploration and is computed using states and actionscollected by the agent.
Revised: The intrinsic rewardfunction is usually constructed for a better exploration and is computed using states and actionscollected by the agent.

Original: Generative adversarial model Generative Adversarial Networks (GANs) [18] consider the problem of generating photo realistic images.
Revised: Generative adversarial model Generative Adversarial Networks (GANs) [18] consider the problem of generating photo realistic images.

Original: The StyleGAN [30, 32, 31] architecture is one of thestate-of-the-art in high-resolution image generation for a multitude of different natural image categories such as faces, buildings, and animals.
Revised: The StyleGAN [30, 32, 31] architecture is one of thestate-of-the-art in high-resolution image generation for a multitude of different natural image categories such as faces, buildings, and animals.

Original: To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generatorarchitecture which consists of a mapping network and synthesis network.
Revised: To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generatorarchitecture which consists of a mapping network and synthesis network.

Original: The mapping networkconverts a latent vector z ∈ Z with Z ∈ R n into an intermediate latent space w ∈ W with W ∈ R n .
Revised: The mapping networkconverts a latent vector z ∈ Z with Z ∈ R n into an intermediate latent space w ∈ W with W ∈ R n .

Original: The mapping network is implemented using a multilayer perceptron that typically consists of 8 layers.
Revised: The mapping network is implemented using a multilayer perceptron that typically consists of 8 layers.

Original: The resulting vector w in that intermediate latent space is then transformed using learned affinetransformations and used as an input to a synthesis network.
Revised: The resulting vector w in that intermediate latent space is then transformed using learned affinetransformations and used as an input to a synthesis network.

Original: The synthesis network consists of multiple blocks that each takes three inputs.
Revised: The synthesis network consists of multiple blocks that each takes three inputs.

Original: First, they take afeature map that contains the current content information of the image that is to be generated.
Revised: First, they take afeature map that contains the current content information of the image that is to be generated.

Original: Second,each block takes a transformed representation of the vector w as an input to its style parts, followedby a normalization of the feature map.
Revised: Second,each block takes a transformed representation of the vector w as an input to its style parts, followedby a normalization of the feature map.

Original:  Method
Revised:  Method

Original: Our objective is to leverage pretrained deep generative models G :
Revised: Our objective is to leverage pretrained deep generative models G :

Original: Z × C
Revised: Z × C

Original: → S where Z denotelatents, C denote prompt or labels and S denote observations to build an interactive environment andtrain an unsupervised reinforcement learning agent in such environment for representation pretraining.
Revised: → S where Z denotelatents, C denote prompt or labels and S denote observations to build an interactive environment andtrain an unsupervised reinforcement learning agent in such environment for representation pretraining.

Original: 4.1 Latent environment dynamics
Revised: 4.1 Latent environment dynamics

Original: The transition dynamics are designed in the latent space of StyleGAN.
Revised: The transition dynamics are designed in the latent space of StyleGAN.

Original: At the beginning of an episode,a class label or prompt c is randomly sampled, at each time step, c and a latent z t that depends on theaction and previous latent z t =
Revised: At the beginning of an episode,a class label or prompt c is randomly sampled, at each time step, c and a latent z t that depends on theaction and previous latent z t =

Original: T ( a, z t − 1 ) are transformed by the synthesis network of StyleGANinto an image which serves as an observation s t =
Revised: T ( a, z t − 1 ) are transformed by the synthesis network of StyleGANinto an image which serves as an observation s t =

Original: G ( z t , c ) .
Revised: G ( z t , c ) .

Original: We note that while the environmentconditions on the label, the ground-truth information is not directly used by PALM.
Revised: We note that while the environmentconditions on the label, the ground-truth information is not directly used by PALM.

Original: The transition dynamics model consists of two steps.
Revised: The transition dynamics model consists of two steps.

Original: In the first step, it combines a randomly sampledlatent z with the agent’s action to obtain a new latent z ′ =
Revised: In the first step, it combines a randomly sampledlatent z with the agent’s action to obtain a new latent z ′ =

Original: T com ( z, a t ) .
Revised: T com ( z, a t ) .

Original: Although more advancedcombination methods can be used, we use simple weighted summation to combine latent and action
Revised: Although more advancedcombination methods can be used, we use simple weighted summation to combine latent and action

Original: T com ( z, a t ) =
Revised: T com ( z, a t ) =

Original: αa t
Revised: αa t

Original: +
Revised: +

Original: (1 − α ) z , where a is an action generated by the policy and α ∈ [0 , 1] is ahyperparameter controls the relative importance of action and latent.
Revised: (1 − α ) z , where a is an action generated by the policy and α ∈ [0 , 1] is ahyperparameter controls the relative importance of action and latent.

Original: The second step is a transformation T ema ( z ′ ) which is designed to ensure temporal persistency of theenvironment.
Revised: The second step is a transformation T ema ( z ′ ) which is designed to ensure temporal persistency of theenvironment.

Original: We use a simple exponential moving average (EMA) to implement T ema .
Revised: We use a simple exponential moving average (EMA) to implement T ema .

Original: Specifically,this module generate new latent by following
Revised: Specifically,this module generate new latent by following

Original: [Equation]
Revised: [Equation]

Original: where z ′ is the combined latent and action, z t is the exponential moving average at timestep t, andβ ∈
Revised: where z ′ is the combined latent and action, z t is the exponential moving average at timestep t, andβ ∈

Original: [0 , 1] is a hyperparameter that controls how fast the environment changes in respond to agent.
Revised: [0 , 1] is a hyperparameter that controls how fast the environment changes in respond to agent.

Original: We used β = 0 .
Revised: We used β = 0 .

Original: 95 in most of our experiments per our initial experiments.
Revised: 95 in most of our experiments per our initial experiments.

Original: The high β is motivated by observations in unsupervised RL and linguistics.
Revised: The high β is motivated by observations in unsupervised RL and linguistics.

Original: It has been shown that ahighly random environment hinders the learning of unsupervised RL agents which is also known asthe "Noisy-TV" problem [7].
Revised: It has been shown that ahighly random environment hinders the learning of unsupervised RL agents which is also known asthe "Noisy-TV" problem [7].

Original: It also has an interesting connection to the uniform information densityhypothesis in linguistics which states that—subject to the constraints of the grammar—humans prefersentences that distribute information equally across the linguistic signal, i.e. , evenly distributedsurprise [42].
Revised: It also has an interesting connection to the uniform information densityhypothesis in linguistics which states that—subject to the constraints of the grammar—humans prefersentences that distribute information equally across the linguistic signal, i.e. , evenly distributedsurprise [43].

Original: In addition to the latent z t , we also condition the generator G ( z, c ) on a random sample class labelc sampled at the beginning of an episode, resulting in further persistent consistency.
Revised: In addition to the latent z t , we also condition the generator G ( z, c ) on a random sample class labelc sampled at the beginning of an episode, resulting in further persistent consistency.

Original: Finally, theenvironment dynamics can be written as T :
Revised: Finally, theenvironment dynamics can be written as T :

Original: = G ( T ema ( T com ( z, a )) , c ) .
Revised: = G ( T ema ( T com ( z, a )) , c ) .

Original: 4.2 Temporal persistent representation
Revised: 4.2 Temporal persistent representation

Original: Our goal is to pretrain a representation f : S → H which is a mapping that maps observations s tolower dimensional representations h , using the data (experience replay) collected by the agent.
Revised: Our goal is to pretrain a representation f : S → H which is a mapping that maps observations s tolower dimensional representations h , using the data (experience replay) collected by the agent.

Original: Dataaugmentations are key ingredients of unsupervised representation learning methods [see e.g. 10, 11].
Revised: Dataaugmentations are key ingredients of unsupervised representation learning methods [see e.g. 10, 11].

Original: However, designing augmentations requires domain knowledge.
Revised: However, designing augmentations requires domain knowledge.

Original: Our representation learning is instead based on temporal information.
Revised: Our representation learning is instead based on temporal information.

Original: Our key idea is that since eachtrajectory is generated by interacting with a deep generative model conditions on the same label andtemporal persistent latent, it provides natural augmentations of visual content under various changingfactors, such as occlusion and illumination.
Revised: Our key idea is that since eachtrajectory is generated by interacting with a deep generative model conditions on the same label andtemporal persistent latent, it provides natural augmentations of visual content under various changingfactors, such as occlusion and illumination.

Original: Some rollout trajectories during training are presentedin Figure 2.
Revised: Some rollout trajectories during training are presentedin Figure 2.

Original: In addition, the exploratory learning behavior of the agent increases the diversity of thetrajectory.
Revised: In addition, the exploratory learning behavior of the agent increases the diversity of thetrajectory.

Original: Specifically, the idea is to train the encoder f ( s ) to produce embeddings that are persistent in time inthe same trajectory, i.e. , two presentations h t = f
Revised: Specifically, the idea is to train the encoder f ( s ) to produce embeddings that are persistent in time inthe same trajectory, i.e. , two presentations h t = f

Original: ( s t ) and h t +1 = f
Revised: ( s t ) and h t +1 = f

Original: ( s t +1 ) should be similar.
Revised: ( s t +1 ) should be similar.

Original: To doso, we use the Siamese network [5] and its modern training techniques from SimSiam [11].
Revised: To doso, we use the Siamese network [5] and its modern training techniques from SimSiam [11].

Original: Specially,we maximize the similarity between q ( g ( f ( s ))) and stop_gradient ( g ( f ( s ′ ))) where q and gare multilayer perceptrons.
Revised: Specially,we maximize the similarity between q ( g ( f ( s ))) and stop_gradient ( g ( f ( s ′ ))) where q and gare multilayer perceptrons.

Original: More details can be found in appendix.
Revised: More details can be found in appendix.

Original: 4.3 Unsupervised exploration
Revised: 4.3 Unsupervised exploration

Original: The objective of unsupervised exploration is to maximize the diversity of the data seen by the agent,imitating how intelligent creatures develop recognition.
Revised: The objective of unsupervised exploration is to maximize the diversity of the data seen by the agent,imitating how intelligent creatures develop recognition.

Original: An observation s t is first encoded using the encoder f to get representation h t = f
Revised: An observation s t is first encoded using the encoder f to get representation h t = f

Original: ( s t ) .
Revised: ( s t ) .

Original: The policytakes representation h t as input and generates action a t of the same dimension.
Revised: The policytakes representation h t as input and generates action a t of the same dimension.

Original: The policy distributionis a tanh Gaussian following prior work in unsupervised reinforcement learning [78, 41].
Revised: The policy distributionis a tanh Gaussian following prior work in unsupervised reinforcement learning [80, 42].

Original: We resort to maximizing the nonparametric entropy of the representation space which has been widelyused in unsupervised reinforcement learning [66, 4, 44, 78].
Revised: We resort to maximizing the nonparametric entropy of the representation space which has been widelyused in unsupervised reinforcement learning [68, 4, 45, 80].

Original: By explicitly encouraging the agents tovisit diverse states, it effectively encourages the agents to explore environments, and thus learn morediverse behaviors.
Revised: By explicitly encouraging the agents tovisit diverse states, it effectively encourages the agents to explore environments, and thus learn morediverse behaviors.

Original: The estimator computes the distance between each particle h i =
Revised: The estimator computes the distance between each particle h i =

Original: f ( s i ) and its k -thnearest neighbor h ⋆i .
Revised: f ( s i ) and its k -thnearest neighbor h ⋆i .

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ 2 is ℓ 2 norm.
Revised: where ∥ · ∥ 2 is ℓ 2 norm.

Original: We associate each transition from experience replay { ( s, a, s ′ ) } with anintrinsic reward given by
Revised: We associate each transition from experience replay { ( s, a, s ′ ) } with anintrinsic reward given by

Original: [Equation]
Revised: [Equation]

Original: Putting everything together, the diagram of our method is shown in Figure 1.
Revised: Putting everything together, the diagram of our method is shown in Figure 1.

Original:  Results
Revised:  Results

Original: It is important to stress that this work focuses on studying how combined exploratory pretraining andstatic datasets contribute to representation learning in artificial agents and not on developing a new,state-of-the-art, methodology for representation learning.
Revised: It is important to stress that this work focuses on studying how combined exploratory pretraining andstatic datasets contribute to representation learning in artificial agents and not on developing a new,state-of-the-art, methodology for representation learning.

Original: Nevertheless, to better situate our results inthe context of existing work, we provide strong baselines in our experiments.
Revised: Nevertheless, to better situate our results inthe context of existing work, we provide strong baselines in our experiments.

Original: Image classification
Revised: Image classification

Original: We test PALM on image classification tasks using a ResNet-18 as encoderf .
Revised: We test PALM on image classification tasks using a ResNet-18 as encoderf .

Original: The RL agent interacts with the environment for 50M steps, and trains representation using theprocedure described in the method section.
Revised: The RL agent interacts with the environment for 50M steps, and trains representation using theprocedure described in the method section.

Original: The hyperparameters of the representation learning followthe default hyperparameters of SimSiam [11] and more details can be found in the supplementalmaterial.
Revised: The hyperparameters of the representation learning followthe default hyperparameters of SimSiam [11] and more details can be found in the supplementalmaterial.

Original: We compare our method with a series of strong methods in static datasets, including state-of-the-artcontrastive representation learning methods SimCLR [10] and SimSiam, supervised trained model,and CLIP [55] which is pretrained on a large-scale paired image-text datasets.
Revised: We compare our method with a series of strong methods in static datasets, including state-of-the-artcontrastive representation learning methods SimCLR [10] and SimSiam, supervised trained model,and CLIP [57] which is pretrained on a large-scale paired image-text datasets.

Original: [Figure]
Revised: [Figure]

Original:  a n
Revised: R a n

Original: [Figure]
Revised: [Figure]

Original: o m
Revised: o m

Original: P A L M
Revised: P A L M

Original: P A L M + d a t a a u g
Revised: P A L M + d a t a a u g

Original: S i m S
Revised: S i m S

Original: i a m
Revised: i a m

Original: S i m C L
Revised: S i m C L

Original: R
Revised: R

Original: C L I P
Revised: C L I P

Original: S u p e
Revised: S u p e

Original: r
Revised: r

Original: v i s e d
Revised: v i s e d

Original: [Figure]
Revised: [Figure]

Revised: P

Original: [Figure]
Revised: [Figure]

Revised: M

Original: [Figure]
Revised: [Figure]

Original: R a n d o m P A L M P A L M + d a t a a u g
Revised: a t a a u g

Original: S i m S
Revised: S i m S

Original: i am
Revised: i a

Revised: [Figure]

Original: S i m C L
Revised: S i m C L

Original: R
Revised: R

Original: C L I P
Revised: C L I P

Original: S u p e
Revised: [Figure]

Original: r v i s e d
Revised: r v i s e d

Original: Figure 3 shows the linear classification performance of each method, PALM is surprisingly effectivegiven the lack of data augmentation and real data.
Revised: Figure 3 shows the linear classification performance of each method, PALM is surprisingly effectivegiven the lack of data augmentation and real data.

Original: Note that the baselines do not have an option notto use data augmentation.
Revised: Note that the baselines do not have an option notto use data augmentation.

Original: Additionally, adding the same data augmentation techniques used in SimSiam to our method furtherimproves the results, matching or outperforming current SOTA static representation learning algorithms.
Revised: Additionally, adding the same data augmentation techniques used in SimSiam to our method furtherimproves the results, matching or outperforming current SOTA static representation learning algorithms.

Original: Compared with the SOTA method SimSiam, PALM + data augmentation actively explore anenvironment instead of using a static dataset, and the representation learning and data augmentationare applied to synthesized temporal pair instead of two views of the same image.
Revised: Compared with the SOTA method SimSiam, PALM + data augmentation actively explore anenvironment instead of using a static dataset, and the representation learning and data augmentationare applied to synthesized temporal pair instead of two views of the same image.

Original: [Table]
Revised: [Table]

Original: Out-of-distribution detection While some prior work demonstrated that self-supervised learningapproaches significantly improve OOD detection performance [25, 15], their self-supervised pre-trainheavily relies on domain-specific data augmentations, we want to study how PALM performs on
Revised: Out-of-distribution detection While some prior work demonstrated that self-supervised learningapproaches significantly improve OOD detection performance [25, 15], their self-supervised pre-trainheavily relies on domain-specific data augmentations, we want to study how PALM performs on

Original: OOD benchmarks.
Revised: OOD benchmarks.

Original: We compare PALM with standard supervised model [22, 24] and contrastive representation learningbased methods including SupCLR [34] and CSI [71].
Revised: We compare PALM with standard supervised model [22, 24] and contrastive representation learningbased methods including SupCLR [34] and CSI [73].

Original: CSI demonstrates state-of-the-art results bycombining contrastive representation learning and supervised learning.
Revised: CSI demonstrates state-of-the-art results bycombining contrastive representation learning and supervised learning.

Original: Table 1 shows the comparisonon multiple datasets.
Revised: Table 1 shows the comparisonon multiple datasets.

Original: PALM outperforms CSI in 4 out of 5 out-of-the-distribution datasets, despiteusing only synthesized data and no data augmentation.
Revised: PALM outperforms CSI in 4 out of 5 out-of-the-distribution datasets, despiteusing only synthesized data and no data augmentation.

Original: We attribute the effectiveness of PALM on
Revised: We attribute the effectiveness of PALM on

Revised: OOD detection to active exploration stimulates the model to learn the invariance of data.

Revised: In doing so,the representation is optimized for distinguishing different observations.

Original: Data efficient representation in RL In this experiment, we are interested in testing PALM pretrained model for downstream reinforcement learning tasks.
Revised: Data efficient representation in RL In this experiment, we are interested in testing PALM pretrained model for downstream reinforcement learning tasks.

Original: To do so, we focus our experimentationon the Atari 100k benchmark introduced by Kaiser et al.
Revised: To do so, we focus our experimentation on the Atari 100k benchmark introduced by Kaiser et al.

Original: [29], in which agents are allowed only 100ksteps of interaction with their environment.
Revised: [29], in which agents are allowed only 100ksteps of interaction with their environment.

Original: There has been extensive usage of Atari for representationlearning and exploratory pretraining [8, 44, 63, 21].
Revised: There has been extensive usage of Atari for representationlearning and exploratory pretraining [8, 45, 65, 21].

Original: Finetuning after exploratory pretraining is highlyeffective [e.g., 63, 8], providing strong baselines to compare to.
Revised: Finetuning after exploratory pretraining is highlyeffective [e.g., 65, 8], providing strong baselines to compare to.

Original: Unlike experiments in CIFAR datasets, here we train an unconditional StyleGAN model in a datasetcollected by running a double DQN [47, 73] agent.
Revised: Unlike experiments in CIFAR datasets, here we train an unconditional StyleGAN model in a datasetcollected by running a double DQN [48, 75] agent.

Original: The dataset consists of 50M frames for eachgame.
Revised: The dataset consists of 50M frames for eachgame.

Original: We use 6M steps for exploratory pretraining in the synthesized environment and we utilize
Revised: We use 6M steps for exploratory pretraining in the synthesized environment and we utilize

Original: DrQ [37] for online finetuning.[Table]
Revised: DrQ [37] for online finetuning.

Original: Synthesized

Original: [Table]
Revised: [Table]

Original: 50M

Original: [Table]

Original: Table 2 shows the comparison of
Revised: Table 2 shows the comparison of

Original: PALM with baselines.
Revised: PALM with baselines.

Original: Comparingwith exploratory pretraining baselines(APT [44], ATC [68], and SGI [63])that are pretrained in Atari games directly, PALM achieves surprisinglygood results considering that it istrained in a simple synthesized environment.
Revised: Comparingwith exploratory pretraining baselines(APT [45], ATC [70], and SGI [65])that are pretrained in Atari games directly, PALM achieves surprisinglygood results considering that it istrained in a simple synthesized environment.

Original: PALM achieves highermedium and mean score than DrQin Atari 100k despite the online finetuning of PALM is the same as
Revised: PALM achieves highermedium and mean score than DrQin Atari 100k despite the online finetuning of PALM is the same as

Original: DrQ, demonstrating the benefit ofpretrained representation.
Revised: DrQ, demonstrating the benefit ofpretrained representation.

Original: Comparedwith offline-data pretraining baselines(ATC and SGI), there is still a gapbetween PALM and the state-of-thearts, which is probably due to PALM’senvironment being trained on observations only while ATC and SGI aretrained on both observations and actions in a model-based style.
Revised: Comparedwith offline-data pretraining baselines(ATC and SGI), there is still a gapbetween PALM and the state-of-thearts, which is probably due to PALM’senvironment being trained on observations only while ATC and SGI aretrained on both observations and actions in a model-based style.

Original: Worth mentioning that while PALM utilizes 50M offline action-freeexperience for pretraining, the amount is less than the 250M online action labeled experience used in
Revised: Worth mentioning that while PALM utilizes 50M offline action-freeexperience for pretraining, the amount is less than the 250M online state-action experience used in

Original: APT.
Revised: APT.

Original: We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform
Revised: We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform

Original: DrQ without data augmentation given the same amount of online experience.
Revised: DrQ without data augmentation given the same amount of online experience.

Original:  Analysis
Revised:  Analysis

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Persistent environment and active interactionare important. Next, we perform an ablationstudy, to study the importance of a persistentenvironment.
Revised: Persistent environment and active interactionare important Next, we perform an ablationstudy, to study the importance of a persistentenvironment.

Original: We aim to answer the followingquestion.
Revised: We aim to answer the followingquestion.

Original: How important it is to have action(controlled by α ) and state (controlled by β )in the transition dynamics?
Revised: How important it is to have action(controlled by α ) and state (controlled by β ) inthe transition dynamics?

Original: When β =0, thereis effectively no state, i.e. , its a bandit.
Revised: When β =0, there iseffectively no state, i.e. , its a bandit.

Original: Asshown in Figure 4, when decreasing the importance of states which is denoted as PALM( α = 0 .
Revised: As shownin Figure 4, when decreasing the importance ofstates which is denoted as PALM ( α = 0 .

Original: 5 β = 0 . 5 ), the performance decreases.
Revised: 5 β =0 . 5 ), the performance decreases.

Original: The extreme case β = 0 shows a significantdrop in accuracy.
Revised: The extremecase β = 0 shows a significant drop in accuracy.

Original: These results indicate that itis important to have temporal persistent transition dynamics.
Revised: These results indicate that it is important to havetemporal persistent transition dynamics.

Original: When α = 0 , action is ignored intransition dynamics, this ablative baseline performs significantly worse than its default α = 0 .
Revised: When α = 0 , action is ignored in transition dynamics, thisablative baseline performs significantly worse than its default α = 0 .

Original: When replacing the agent’s action with random noise which is denoted as PALM + random action,we also see a significant performance drop.
Revised: When replacing the agent’saction with random noise which is denoted as PALM + random action, we also see a significantperformance drop.

Original: These results show having active interaction is importantto the results of PALM.
Revised: These results show having active interaction is important to the results of PALM.

Original: We present the results of more combinations of α and β in Figure 5, theresults show that as α decreases from 0.95 to 0, because of the decreasing importance of action,the accuracy continues to drop. α = 0 . 95 in general performs better than α = 1 . 0 , indicatesmixing a small amount of random noise with action helps exploration and improves representationlearning. When α or β is zero, the results are significantly worse.
Revised: [Figure]

Original: We also see that β ∈ [0 . 5 , 0 . 75]performs significantly better than other values, showing that persistence of dynamics is crucial. [Figure] [Figure]
Revised: PALM passive PALM (random) passive PALM (ordered) static PALM

Original: Exploratory Learning as Curriculum Learning.
Revised: Exploratory Learning as Curriculum Learning

Original: As we have seen, representations learned from exploratory learning in a synthesized environment canbe surprisingly powerful.
Revised: As we have seen, representations learned fromexploratory learning in a synthesized environment can be surprisingly powerful.

Original: We are interested in havinga perhaps preliminary study of the reasons.
Revised: We are interested inhaving a perhaps preliminary study of the reasons.

Original: To do so, we compare PALM with its passive version(passive PALM), static version (staticPALM), andmethods proposed in Jahanian et al. In therandom variant, the positives pairs are sampled from G ( z, c = c ′ )
Revised: To do so, we compare PALM with its passive version (passive PALM) and static version (static

Original: (i.e. fix the class, take two randomdraws from z). We denote this baseline as Random Jahanian et al. In Gaussian Jahanian et al, the positivepairs are given by z and z+w, where w is Gaussian.
Revised: PALM).

Original: Static PALM is not trained in the interactive environment, instead, it is trained with real images.
Revised: Static PALM is not trained in the interactive environment, instead, it is trained with realimages.

Original: Specifically, we randomly sample two images of the same label from a dataset and use them as atemporal pair to train the representation learning of PALM.We note that static PALM is related to
Revised: Specifically, we randomly sample two images of the same label from a dataset and use themas a temporal pair to train the representation learning of PALM.

Original: SupCon [34], but it differs in that static PALM does not use any data augmentation while SupConleverages a combination of data augmentations.

Original: SupCon also uses a large number of negativeexamples from different categories.

Original: Passive PALM is trained offline on the experience of PALM.
Revised: Passive PALM is trained offline on the experience of PALM.

Original: We first store all the online experience(50M in total) of PALM, then we use the saved experience to train passive PALM without having tointeract with the environment.
Revised: We first store all the online experience(50M in total) of PALM, then we use the saved experience to train passive PALM without having tointeract with the environment.

Original: We consider two options of passive PALM.
Revised: We consider two options of passive PALM.

Original: The first one is randompassive PALM which randomly samples consecutive observations from the stored offline experience.
Revised: The first one is randompassive PALM which randomly samples consecutive observations from the stored offline experience.

Original: The second one is ordered passive PALM which is trained on the same order of observations as inonline experience.
Revised: The second one is ordered passive PALM which is trained on the same order of observations as inonline experience.

Revised: Figure 5 shows the comparison results.

Original: With the number of data increasing, PALM keeps improving with the amount of training data andsignificantly outperforms static PALM even though static PALM is trained on real datasets.
Revised: With the number of data increasing, PALM keeps improvingwith the amount of training data and significantly outperforms static PALM even though static PALMis trained on real datasets.

Original: Webelieve the reason is that the diverse synthesized data collected by PALM can greatly help to learnrepresentation.
Revised: We believe the reason is that the diverse synthesized data collected by PALM can greatly help to learn representation.

Original: In the low data regime, carefully designed non-active learning based methods [26]outperforms PALM, possibly due to RL training being sample inefficient. With the increasing numberof synthesized samples, PALM continues to improve and significantly outperform all baselines.
Revised: PALM nearly always outperforms random passive

Original: The results also show that PALM scales better than non active learning based methods. PALMnearly always outperforms random passive PALM, demonstrating the effectiveness of exploratorylearning.
Revised: PALM, demonstrating the effectiveness of exploratory learning.

Original: Interestingly but not surprisingly, ordered passive PALM performs nearly the same as PALM,indicating the exploratory learning behavior of PALM might play an important role in curriculumlearning.
Revised: Interestingly but not surprisingly,ordered passive PALM performs nearly the same as PALM, indicating the exploratory learningbehavior of PALM might play an important role in curriculum learning.

Original: The agent is optimized to search for novel samples in terms of their representation, by doingso, the searched samples should become more difficult to distinguish.
Revised: The agent is optimized tosearch for novel samples in terms of their representation, by doing so, the searched samples shouldbecome more difficult to distinguish.

Original: We believe a comprehensiveunderstanding of the effectiveness of PALM is out of the scope of this work, we tend to leave it asfuture work.[Table] Diversity analysis.
Revised: We believe a comprehensive understanding of the effectivenessof PALM is out of the scope of this work, we tend to leave it as future work.

Original: In order to measure the diversity of the

Original: [Table]

Original: sampled synthetic data, we randomly sample 10K images

Original: [Table]

Original: from the CIFAR-10 train set and generate 10K images

Original: [Table]

Original: from StyleGAN by random sample latents.

Original: We also gen-

Original: [Table]

Original: erate 10K images with PALM in latent space.

Original: For eachsample in each set, we find its closest neighbor in Inception feature space (obtained after the pooling layer).

Original: Thecoverage is defined as the proportion of nearest neighborsthat are unique in the train set.

Original: A better active exploration method would produce samples that areequally likely to be close to any image in the train set.

Original: PALM achieves a better coverage score thanrandom sampling in latent space.

Original: Although the Euclidean distance in Inception feature space isinaccurate, the better coverage achieved by PALM indicates that by exploring the latent manifold itgets more diverse data.

Original:  Conclusion and discussion
Revised:  Conclusion and discussion

Original: Our work presents a method to connect learning from large, diverse static datasets and learning byinteraction, and demonstrates promising results of doing so.
Revised: Our work presents a method to connect learning from large, diverse static datasets and learning byinteraction, and demonstrates promising results of doing so.

Original: We believe it opens up an interestingfuture direction for advancing both reinforcement learning and self-supervised visual representationlearning.
Revised: We believe it opens up an interestingfuture direction for advancing both reinforcement learning and self-supervised visual representationlearning.

Original: For limitations, since it is computational expensive to draw samples from large generative model,interacting with such environment is significantly slower than well optimized physical simulationsused in RL, e.g. , Mujoco.
Revised: For limitations, since it is computational expensive to draw samples from large generative model,interacting with such environment is significantly slower than well optimized physical simulationsused in RL, e.g. , Mujoco.

Original: Addressing such limitation would make this work more accessible andfacilitate future research.
Revised: Addressing such limitation would make this work more accessible andfacilitate future research.

Original: For future work, it would be interesting to design or learn better transition dynamics.
Revised: For future work, it would be interesting to design or learn better transition dynamics.

Original: It would be alsointeresting to evaluate our method in RL settings where a lot of action-free data are available andwon’t have to be collected (Minecraft for example).

Original: It would be also interesting to use a languageconditioned generative model as an alternative e g. Dalle [57] and Dalle v2 [58], so the environmenthas more complexity by prompting it with natural language
Revised: It would be alsointeresting to use a language conditioned generative model as an alternative e.g. Dalle [59] and Dallev2 [60], so the environment has more complexity by prompting it with natural language.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes] The instructionswill be provided in the supplemental material and we plan to release the code andpre-trained models to facilitate further research.(b)
Revised: [Yes] The instructions will be provided in the supplemental material and we plan to release the code and pre-trained models to facilitate further research.

Original:  Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] The full details will be provided in supplementary material.
Revised: [Yes] The full details will be provided in supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] Our experiments on classification and OOD detection
Revised: [Yes] Our experiments on classification and OOD detection

Original: experiments are not sensitive to random seeds.
Revised: experiments are not sensitive to random seeds.

Original: For our RL experiments, we run the
Revised: For our RL experiments, we run the

Original: evaluate multiple times and include the details in experiment section and supplementarymaterial.
Revised: evaluate multiple times and include the details in experiment section and supplementary material.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] The full details will be provided in
Revised: [Yes] The full details will be provided in

Original: supplementary material.
Revised: supplementary material.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The datasets we use are public and do not
Revised: The datasets we use are public and do not

Original: contain personally identifiable information.
Revised: contain personally identifiable information.

Original: If you used crowdsourcing or conducted research with human subjects..
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original:  Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N
Revised: [N/A]

Original: [Table]
Revised: [Figure]

Original:  you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Figure]

Original: Board 
Revised: Board (

Original:  approvals, applicable
Revised: ) approvals, if applicable?

Original: N [Table] you
Revised: [N/A] [Figure]

Original:  the estimated hourly  paid  participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [Table]
Revised: [N/A]

Original: spent

Original: participant compensation

Original: N [Table]
Revised: [Figure]