Original: Palm up: Playing in the Latent Manifold for Unsupervised Pretraining
Revised: Palm up: Playing in the Latent Manifold for Unsupervised Pretraining

Original: Abstract
Revised: Abstract

Original: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence.
Revised: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence.

Original: Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment.
Revised: Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment.

Original: In this work, we aim to bring the best of both worlds and propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets.
Revised: In this work, we aim to bring the best of both worlds and propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets.

Original: Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space.
Revised: Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space.

Original: The transition dynamics simply mixes an action and a random sampled latent.
Revised: The transition dynamics simply mixes an action and a random sampled latent.

Original: It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator.
Revised: It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator.

Original: We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.
Revised: We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.

Original: We further leverage the temporal information of this data to pair data points as a natural supervision for representation learning.
Revised: We further leverage the temporal information of this data to pair data points as a natural supervision for representation learning.

Original: Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains.
Revised: Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains.

Original:  Introduction
Revised:  Introduction

Original: Large and diverse datasets have been the cornerstones of many impressive successes at the frontier of artificial intelligence, such as protein folding [64, 28], image recognition [55, 13] and understanding natural language [6, 12].
Revised: Large and diverse datasets have been the cornerstones of many impressive successes at the frontier of artificial intelligence, such as protein folding [64, 28], image recognition [55, 13] and understanding natural language [6, 12].

Original: Training machine learning models on diverse datasets that cover a breadth of human-written text and natural images often dramatically improves performance and enables impressive generalization capabilities [16, 6, 55, 12].
Revised: Training machine learning models on diverse datasets that cover a breadth of human-written text and natural images often dramatically improves performance and enables impressive generalization capabilities [16, 6, 55, 12].

Original: These models are learned from a fixed set of images, videos, or languages, guided by supervision that comes from ground-truth labels [13, 55], self-supervised contrastive learning [10] or masked token prediction [6, 12] among other approaches.
Revised: These models are learned from a fixed set of images, videos, or languages, guided by supervision that comes from ground-truth labels [13, 55], self-supervised contrastive learning [10] or masked token prediction [6, 12] among other approaches.

Original: In contrast, intelligent creatures learn to perform interactions that actively change the input sensory signals and the state of the environment towards a desired configuration.
Revised: In contrast, intelligent creatures learn to perform interactions that actively change the input sensory signals and the state of the environment towards a desired configuration.

Original: For example, in psychology, it has been shown that interaction with the environment is vital for developing flexible and generalize intelligence [23, 74, 17].
Revised: For example, in psychology, it has been shown that interaction with the environment is vital for developing flexible and generalize intelligence [23, 74, 17].

Original: During the first few months of interactions, infants develop meaningful understandings about objects [67] and prefer to look at exemplars from a novel class (e.g., dogs) after observing exemplars from a different class (e.g., cats) [54].
Revised: During the first few months of interactions, infants develop meaningful understandings about objects [67] and prefer to look at exemplars from a novel class (e.g., dogs) after observing exemplars from a different class (e.g., cats) [54].

Original: We hypothesis that by situating the agent in a high semantic complexity environment, the agent can develop interesting cognition abilities.
Revised: We hypothesis that by situating the agent in a high semantic complexity environment, the agent can develop interesting cognition abilities.

Original: Given the effectiveness of learning from large and diverse datasets and the significance of interactive learning behaviors in intelligent creatures, it is therefore extremely important to connect them to build better learning algorithms.
Revised: Given the effectiveness of learning from large and diverse datasets and the significance of interactive learning behaviors in intelligent creatures, it is therefore extremely important to connect them to build better learning algorithms.

Original: To bridge the gap, perhaps one straightforward solution is learning in the diverse real world [1, 53], however, teaching a robot to interact in the physical world is both time consuming and resource intensive, and it is also difficult to scale up.
Revised: To bridge the gap, perhaps one straightforward solution is learning in the diverse real world [1, 53], however, teaching a robot to interact in the physical world is both time consuming and resource intensive, and it is also difficult to scale up.

Original: An alternative is learning in simulated environments which have got surged interest recently.
Revised: An alternative is learning in simulated environments which have got surged interest recently.

Original: There simulators, such as Habitat [61, 70], RLBench [27],
Revised: There simulators, such as Habitat [61, 70], RLBench [27],

Original: House3D [77], and AI2THOR [36] enable the agent to interact with its environment.
Revised: House3D [77], and AI2THOR [36] enable the agent to interact with its environment.

Original: However, the visual complexity of these simulated environments is far from matching the intricate real world.
Revised: However, the visual complexity of these simulated environments is far from matching the intricate real world.

Original: The key limitation is that making hand-designed simulation that is close enough to what a camera in the real-world would capture is both challenging and tedious.
Revised: The key limitation is that making hand-designed simulation that is close enough to what a camera in the real-world would capture is both challenging and tedious.

Original: To remedy the issue, we propose a conceptually simple yet effective method that leverages existing diverse datasets, builds an environment with high semantic complexity from them, and then performs interactive learning in this environment.
Revised: To remedy the issue, we propose a conceptually simple yet effective method that leverages existing diverse datasets, builds an environment with high semantic complexity from them, and then performs interactive learning in this environment.

Original: We do so by leveraging deep generative models that are trained in static datasets and introduce transition dynamics in the latent space of the generative model.
Revised: We do so by leveraging deep generative models that are trained in static datasets and introduce transition dynamics in the latent space of the generative model.

Original: Specifically, at each time step, the transition dynamics simply mix action and a random sampled latent.
Revised: Specifically, at each time step, the transition dynamics simply mix action and a random sampled latent.

Original: It then applies an exponential moving average for temporal persistency, imitating the prevalent temporal persistency in the real world.
Revised: It then applies an exponential moving average for temporal persistency, imitating the prevalent temporal persistency in the real world.

Original: Finally, the resulting latent is decoded to an image using a trained generator.
Revised: Finally, the resulting latent is decoded to an image using a trained generator.

Original: The generator is a conditional generative model which is conditioned on a prompt e.g. a class label sampled at the beginning of an episode for achieving further temporal persistency.
Revised: The generator is a conditional generative model which is conditioned on a prompt e.g. a class label sampled at the beginning of an episode for achieving further temporal persistency.

Original: For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity, although our method is not restricted to it, and can be also applied to other generative models such as language conditioned model Dalle [56].
Revised: For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity, although our method is not restricted to it, and can be also applied to other generative models such as language conditioned model Dalle [56].

Original: We employ unsupervised reinforcement learning (RL) [41] to explore this environment motivated by imitating how intelligent creatures acquire perception and action skills by curiosity [23, 35, 60].
Revised: We employ unsupervised reinforcement learning (RL) [41] to explore this environment motivated by imitating how intelligent creatures acquire perception and action skills by curiosity [23, 35, 60].

Original: Specifically, we use the nonparametric entropy maximization method named APT [44] which encourages the agent to actively explore the environment to seek novel and unseen observations.
Revised: Specifically, we use the nonparametric entropy maximization method named APT [44] which encourages the agent to actively explore the environment to seek novel and unseen observations.

Original: Similar to other pixel-based unsupervised RL methods, APT learns an abstract representation by using off-the-shelf data augmentation and contrastive learning techniques from vision [37, 40, 39].
Revised: Similar to other pixel-based unsupervised RL methods, APT learns an abstract representation by using off-the-shelf data augmentation and contrastive learning techniques from vision [37, 40, 39].

Original: While effective, designing these techniques requires domain knowledge.
Revised: While effective, designing these techniques requires domain knowledge.

Original: We show that by simply leveraging the temporal nature, representation can be effectively learned.
Revised: We show that by simply leveraging the temporal nature, representation can be effectively learned.

Original: We do so by maximizing the similarity between representations of current and next observations based on siamese network [5] without needing to use domain knowledge or data augmentation.
Revised: We do so by maximizing the similarity between representations of current and next observations based on siamese network [5] without needing to use domain knowledge or data augmentation.

Original: Our method is named as p l a ying in the l atent m anifold for unsupervised pretraining (PALM).
Revised: Our method is named as p l a ying in the l atent m anifold for unsupervised pretraining (PALM).

Original: We conduct experiments in CIFAR classification and out-of-distribution detection by transferring our unsupervised exploratory pretrained representations in StyleGAN-based environments.
Revised: We conduct experiments in CIFAR classification and out-of-distribution detection by transferring our unsupervised exploratory pretrained representations in StyleGAN-based environments.

Original: Our experiments show that the learned representations achieve competitive results with state-of-theart methods in image recognition and out-of-distribution detection despite being only trained in synthesized data without data augmentation.
Revised: Our experiments show that the learned representations achieve competitive results with state-of-theart methods in image recognition and out-of-distribution detection despite being only trained in synthesized data without data augmentation.

Original: We also train StyleGAN in observation data collected from Atari and apply our method to it.
Revised: We also train StyleGAN in observation data collected from Atari and apply our method to it.

Original: We found that the learned representation helped in maximizing many Atari game rewards.
Revised: We found that the learned representation helped in maximizing many Atari game rewards.

Original: Our major contributions are summarized below:
Revised: Our major contributions are summarized below:

Original: • We present a surprisingly simple yet effective approach to leverage generative models as an interactive environment for unsupervised RL.
Revised: • We present a surprisingly simple yet effective approach to leverage generative models as an interactive environment for unsupervised RL.

Original: By doing so, we connect vision datasets with RL, and enable learning representation by actively interacting with the environment.
Revised: By doing so, we connect vision datasets with RL, and enable learning representation by actively interacting with the environment.

Original: • We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent to learn representations from a synthetic environment without data augmentations.
Revised: • We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent to learn representations from a synthetic environment without data augmentations.

Original: • We show that PALM matches SOTA self-supervised representation learning methods on CIFAR and out-of-distribution benchmarks.
Revised: • We show that PALM matches SOTA self-supervised representation learning methods on CIFAR and out-of-distribution benchmarks.

Original: • We show that PALM outperforms strong model-free and model-based training from scratch RL.
Revised: • We show that PALM outperforms strong model-free and model-based training from scratch RL.

Original: It also achieves competitive scores as SOTA exploratory pre-training RL and offline-data pretraining RL methods.
Revised: It also achieves competitive scores as SOTA exploratory pre-training RL and offline-data pretraining RL methods.

Original:  Related work
Revised:  Related work

Original: Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on the target task has been explored in reinforcement learning to improve downstream task performance.
Revised: Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on the target task has been explored in reinforcement learning to improve downstream task performance.

Original: One common approach has been to allow the agent a period of fully-unsupervised interaction with the environment, during which the agent is trained to maximize a surrogate exploration-based task such as the diversity of the states it encounters [44, 43, 78].
Revised: One common approach has been to allow the agent a period of fully-unsupervised interaction with the environment, during which the agent is trained to maximize a surrogate exploration-based task such as the diversity of the states it encounters [44, 43, 78].

Original: Others have proposed to use self-supervised objectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of an inverse dynamics model [52, 7].
Revised: Others have proposed to use self-supervised objectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of an inverse dynamics model [52, 7].

Original: SGI [63] combines forward predictive representation learning [62] with inverse dynamics model [52] and demonstrate the power of representation pretraining for downstream RL tasks.
Revised: SGI [63] combines forward predictive representation learning [62] with inverse dynamics model [52] and demonstrate the power of representation pretraining for downstream RL tasks.

Original: Massive-scale unsupervised pretraining has shown strong results [8].
Revised: Massive-scale unsupervised pretraining has shown strong results [8].

Original: Laskin et al.
Revised: Laskin et al.

Original: [41] conducted a comparison of different unsupervised pretraining reinforcement learning algorithms.
Revised: [41] conducted a comparison of different unsupervised pretraining reinforcement learning algorithms.

Original: Finally, Chaplot et al.
Revised: Finally, Chaplot et al.

Original: [9], Weihs et al.
Revised: [9], Weihs et al.

Original: [75] studied training RL agent in game simulators and transferring its representation to various vision tasks.
Revised: [75] studied training RL agent in game simulators and transferring its representation to various vision tasks.

Original: Their environments are equipped with carefully chosen domain-specific reward function to guide the learning of RL agent, and the architectures of their RL agents are fairly complicated.
Revised: Their environments are equipped with carefully chosen domain-specific reward function to guide the learning of RL agent, and the architectures of their RL agents are fairly complicated.

Original: Our work differs in that we do not rely on hand-crafted simulators and renderers which require a huge amount of domain knowledge and effort to build, instead we leverage generative models as renders.
Revised: Our work differs in that we do not rely on hand-crafted simulators and renderers which require a huge amount of domain knowledge and effort to build, instead we leverage generative models as renders.

Original: Unlike many prior work in unsupervised pretraining RL, our work does not focus on improving transfer performance to downstream RL tasks although it can be used for this purpose.
Revised: Unlike many prior work in unsupervised pretraining RL, our work does not focus on improving transfer performance to downstream RL tasks although it can be used for this purpose.

Original: Training with synthetic data Using deep generative models as a source of synthetic data for representation learning has been studied in prior work [59, 26, 33].
Revised: Training with synthetic data Using deep generative models as a source of synthetic data for representation learning has been studied in prior work [59, 26, 33].

Original: These generative models are fit to real image datasets and produce realistic-looking images as samples.
Revised: These generative models are fit to real image datasets and produce realistic-looking images as samples.

Original: Baradad et al.
Revised: Baradad et al.

Original: [2] studied using data sampled from random initialized generative models to train contrastive representations.
Revised: [2] studied using data sampled from random initialized generative models to train contrastive representations.

Original: Gowal et al.
Revised: Gowal et al.

Original: [20] studied combining data sampled from pretrained generative models with real data for adversarial training and demonstrated improved results in robustness.
Revised: [20] studied combining data sampled from pretrained generative models with real data for adversarial training and demonstrated improved results in robustness.

Original: The use of synthesized data has been explored in reinforcement learning under the heading of domain randomization [72], where 3D synthetic data is rendered under a variety of lighting conditions to transfer to real environments where the lighting may be unknown.
Revised: The use of synthesized data has been explored in reinforcement learning under the heading of domain randomization [72], where 3D synthetic data is rendered under a variety of lighting conditions to transfer to real environments where the lighting may be unknown.

Original: Our approach does away with the hand crafted simulation engine entirely by making the training data diverse through unsupervised exploration.
Revised: Our approach does away with the hand crafted simulation engine entirely by making the training data diverse through unsupervised exploration.

Original: Different from them, our work focus on leveraging a generative model as an interactive environment and learn representation without using data augmentation.
Revised: Different from them, our work focus on leveraging a generative model as an interactive environment and learn representation without using data augmentation.

Original: Temporal persistent representation Using temporal persistent information for representation learning has been proposed in the past with similar motivations as ours.
Revised: Temporal persistent representation Using temporal persistent information for representation learning has been proposed in the past with similar motivations as ours.

Original: It has been used in learning representation from videos [3, 76, 48, 19, 14, 51] by minimizing different metrics of representation difference over a temporal segment.
Revised: It has been used in learning representation from videos [3, 76, 48, 19, 14, 51] by minimizing different metrics of representation difference over a temporal segment.

Original: Learning persistent representation has been explored in reinforcement learning, and has been demonstrated to improve data efficiency [50, 65, 62, 79] and improve downstream task performance [68, 63].
Revised: Learning persistent representation has been explored in reinforcement learning, and has been demonstrated to improve data efficiency [50, 65, 62, 79] and improve downstream task performance [68, 63].

Original: In relation to these prior efforts, our work studies visual representation learning from interacted experiences based on real-world data.
Revised: In relation to these prior efforts, our work studies visual representation learning from interacted experiences based on real-world data.

Original:  Preliminary
Revised:  Preliminary

Original: Unsupervised reinforcement learning Reinforcement learning considers the problem of finding an optimal policy for an agent that interacts with an uncertain environment and collects reward per action [69].
Revised: Unsupervised reinforcement learning Reinforcement learning considers the problem of finding an optimal policy for an agent that interacts with an uncertain environment and collects reward per action [69].

Original: The agent maximizes its cumulative reward by interacting with its environment.
Revised: The agent maximizes its cumulative reward by interacting with its environment.

Original: Formally, this problem can be viewed as a Markov decision process (MDP) defined by ( S , A , T , ρ 0 , r, γ ) where S ⊆ R n s is a set of n s -dimensional states, A ⊆ R n a is a set of n adimensional actions, T : S ×
Revised: Formally, this problem can be viewed as a Markov decision process (MDP) defined by ( S , A , T , ρ 0 , r, γ ) where S ⊆ R n s is a set of n s -dimensional states, A ⊆ R n a is a set of n adimensional actions, T : S ×

Original: A × S → [0 , 1] is the state transition probability distribution.
Revised: A × S → [0 , 1] is the state transition probability distribution.

Original: ρ 0 : S → [0 , 1] is the distribution over initial states, r : S ×
Revised: ρ 0 : S → [0 , 1] is the distribution over initial states, r : S ×

Original: A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.
Revised: A → R is the reward function, and γ ∈ [0 , 1) is the discount factor.

Original: At environment states s ∈ S , the agent take actions a ∈ A , in the (unknown) environment dynamics defined by the transition probability T ( s ′ | s, a ) , and the reward function yields a reward immediately following the action a t performed in state s t .
Revised: At environment states s ∈ S , the agent take actions a ∈ A , in the (unknown) environment dynamics defined by the transition probability T ( s ′ | s, a ) , and the reward function yields a reward immediately following the action a t performed in state s t .

Original: In valuebased reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a, state-action value function Q π ( s t , a t ) =
Revised: In valuebased reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a, state-action value function Q π ( s t , a t ) =

Original: E s t
Revised: E s t

Original: +1 ,a t +1 ,... (cid:2)(cid:80) ∞
Revised: +1 ,a t +1 ,... (cid:2)(cid:80) ∞

Original: l =

Revised: l =

Original: γ l r ( s t + l , a t + l ) (cid:3) .
Revised: γ l r ( s t + l , a t + l ) (cid:3) .

Original: A new policy can be derived from value function by acting ϵ -greedily with respect to the action values (discrete) or by using policy gradient to maximize the value function (continuous).
Revised: A new policy can be derived from value function by acting ϵ -greedily with respect to the action values (discrete) or by using policy gradient to maximize the value function (continuous).

Original: In unsupervised reinforcement learning, the reward function is defined as some form of intrinsic reward that is agnostic to standard task-specific reward function r := r intrinsic .
Revised: In unsupervised reinforcement learning, the reward function is defined as some form of intrinsic reward that is agnostic to standard task-specific reward function r := r intrinsic .

Original: The intrinsic reward function is usually constructed for a better exploration and is computed using states and actions collected by the agent.
Revised: The intrinsic reward function is usually constructed for a better exploration and is computed using states and actions collected by the agent.

Original: Generative adversarial model Generative Adversarial Networks (GANs) [18] consider the problem of generating photo realistic images.
Revised: Generative adversarial model Generative Adversarial Networks (GANs) [18] consider the problem of generating photo realistic images.

Original: The StyleGAN [30, 32, 31] architecture is one of the state-of-the-art in high-resolution image generation for a multitude of different natural image categories such as faces, buildings, and animals.
Revised: The StyleGAN [30, 32, 31] architecture is one of the state-of-the-art in high-resolution image generation for a multitude of different natural image categories such as faces, buildings, and animals.

Original: To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generator architecture which consists of a mapping network and synthesis network.
Revised: To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generator architecture which consists of a mapping network and synthesis network.

Original: The mapping network converts a latent vector z ∈ Z with Z ∈ R n into an intermediate latent space w ∈ W with W ∈ R n .
Revised: The mapping network converts a latent vector z ∈ Z with Z ∈ R n into an intermediate latent space w ∈ W with W ∈ R n .

Original: The mapping network is implemented using a multilayer perceptron that typically consists of 8 layers.
Revised: The mapping network is implemented using a multilayer perceptron that typically consists of 8 layers.

Original: The resulting vector w in that intermediate latent space is then transformed using learned affine transformations and used as an input to a synthesis network.
Revised: The resulting vector w in that intermediate latent space is then transformed using learned affine transformations and used as an input to a synthesis network.

Original: The synthesis network consists of multiple blocks that each takes three inputs.
Revised: The synthesis network consists of multiple blocks that each takes three inputs.

Original: First, they take a feature map that contains the current content information of the image that is to be generated.
Revised: First, they take a feature map that contains the current content information of the image that is to be generated.

Original: Second, each block takes a transformed representation of the vector w as an input to its style parts, followed by a normalization of the feature map.
Revised: Second, each block takes a transformed representation of the vector w as an input to its style parts, followed by a normalization of the feature map.

Original:  Method
Revised:  Method

Original: Our objective is to leverage pretrained deep generative models G :
Revised: Our objective is to leverage pretrained deep generative models G :

Original: Z × C
Revised: Z × C

Original: → S where Z denote latents, C denote prompt or labels and S denote observations to build an interactive environment and train an unsupervised reinforcement learning agent in such environment for representation pretraining.
Revised: → S where Z denote latents, C denote prompt or labels and S denote observations to build an interactive environment and train an unsupervised reinforcement learning agent in such environment for representation pretraining.

Original: 4.1 Latent environment dynamics
Revised: 4.1 Latent environment dynamics

Original: The transition dynamics are designed in the latent space of StyleGAN.
Revised: The transition dynamics are designed in the latent space of StyleGAN.

Original: At the beginning of an episode, a class label or prompt c is randomly sampled, at each time step, c and a latent z t that depends on the action and previous latent z t =
Revised: At the beginning of an episode, a class label or prompt c is randomly sampled, at each time step, c and a latent z t that depends on the action and previous latent z t =

Original: T ( a, z t − 1 ) are transformed by the synthesis network of StyleGAN into an image which serves as an observation s t =
Revised: T ( a, z t − 1 ) are transformed by the synthesis network of StyleGAN into an image which serves as an observation s t =

Original: G ( z t , c ) .
Revised: G ( z t , c ) .

Original: We note that while the environment conditions on the label, the ground-truth information is not directly used by PALM.
Revised: We note that while the environment conditions on the label, the ground-truth information is not directly used by PALM.

Original: The transition dynamics model consists of two steps.
Revised: The transition dynamics model consists of two steps.

Original: In the first step, it combines a randomly sampled latent z with the agent’s action to obtain a new latent z ′ =
Revised: In the first step, it combines a randomly sampled latent z with the agent’s action to obtain a new latent z ′ =

Original: T com ( z, a t ) .
Revised: T com ( z, a t ) .

Original: Although more advanced combination methods can be used, we use simple weighted summation to combine latent and action
Revised: Although more advanced combination methods can be used, we use simple weighted summation to combine latent and action

Original: T com ( z, a t ) =
Revised: T com ( z, a t ) =

Original: αa t
Revised: αa t

Original: +
Revised: +

Original: (1 − α ) z , where a is an action generated by the policy and α ∈ [0 , 1] is a hyperparameter controls the relative importance of action and latent.
Revised: (1 − α ) z , where a is an action generated by the policy and α ∈ [0 , 1] is a hyperparameter controls the relative importance of action and latent.

Original: The second step is a transformation T ema ( z ′ ) which is designed to ensure temporal persistency of the environment.
Revised: The second step is a transformation T ema ( z ′ ) which is designed to ensure temporal persistency of the environment.

Original: We use a simple exponential moving average (EMA) to implement T ema .
Revised: We use a simple exponential moving average (EMA) to implement T ema .

Original: Specifically, this module generate new latent by following
Revised: Specifically, this module generate new latent by following

Original: [Equation]
Revised: [Equation]

Original: where z ′ is the combined latent and action, z t is the exponential moving average at timestep t, and β ∈ [0 , 1] is a hyperparameter that controls how fast the environment changes in respond to agent.
Revised: where z ′ is the combined latent and action, z t is the exponential moving average at timestep t, and β ∈ [0 , 1] is a hyperparameter that controls how fast the environment changes in respond to agent.

Original: We used β = 0 .
Revised: We used β = 0 .

Original: 95 in most of our experiments per our initial experiments.
Revised: 95 in most of our experiments per our initial experiments.

Original: The high β is motivated by observations in unsupervised RL and linguistics.
Revised: The high β is motivated by observations in unsupervised RL and linguistics.

Original: It has been shown that a highly random environment hinders the learning of unsupervised RL agents which is also known as the "Noisy-TV" problem [7].
Revised: It has been shown that a highly random environment hinders the learning of unsupervised RL agents which is also known as the "Noisy-TV" problem [7].

Original: It also has an interesting connection to the uniform information density hypothesis in linguistics which states that—subject to the constraints of the grammar—humans prefer sentences that distribute information equally across the linguistic signal, i.e. , evenly distributed surprise [42].
Revised: It also has an interesting connection to the uniform information density hypothesis in linguistics which states that—subject to the constraints of the grammar—humans prefer sentences that distribute information equally across the linguistic signal, i.e. , evenly distributed surprise [42].

Original: In addition to the latent z t , we also condition the generator G ( z, c ) on a random sample class label c sampled at the beginning of an episode, resulting in further persistent consistency.
Revised: In addition to the latent z t , we also condition the generator G ( z, c ) on a random sample class label c sampled at the beginning of an episode, resulting in further persistent consistency.

Original: Finally, the environment dynamics can be written as T :
Revised: Finally, the environment dynamics can be written as T :

Original: = G ( T ema ( T com ( z, a )) , c ) .
Revised: = G ( T ema ( T com ( z, a )) , c ) .

Original: 4.2 Temporal persistent representation
Revised: 4.2 Temporal persistent representation

Original: Our goal is to pretrain a representation f : S → H which is a mapping that maps observations s to lower dimensional representations h , using the data (experience replay) collected by the agent.
Revised: Our goal is to pretrain a representation f : S → H which is a mapping that maps observations s to lower dimensional representations h , using the data (experience replay) collected by the agent.

Original: Data augmentations are key ingredients of unsupervised representation learning methods [see e.g. 10, 11].
Revised: Data augmentations are key ingredients of unsupervised representation learning methods [see e.g. 10, 11].

Original: However, designing augmentations requires domain knowledge.
Revised: However, designing augmentations requires domain knowledge.

Original: Our representation learning is instead based on temporal information.
Revised: Our representation learning is instead based on temporal information.

Original: Our key idea is that since each trajectory is generated by interacting with a deep generative model conditions on the same label and temporal persistent latent, it provides natural augmentations of visual content under various changing factors, such as occlusion and illumination.
Revised: Our key idea is that since each trajectory is generated by interacting with a deep generative model conditions on the same label and temporal persistent latent, it provides natural augmentations of visual content under various changing factors, such as occlusion and illumination.

Original: Some rollout trajectories during training are presented in Figure 2.
Revised: Some rollout trajectories during training are presented in Figure 2.

Original: In addition, the exploratory learning behavior of the agent increases the diversity of the trajectory.
Revised: In addition, the exploratory learning behavior of the agent increases the diversity of the trajectory.

Original: Specifically, the idea is to train the encoder f ( s ) to produce embeddings that are persistent in time in the same trajectory, i.e. , two presentations h t = f
Revised: Specifically, the idea is to train the encoder f ( s ) to produce embeddings that are persistent in time in the same trajectory, i.e. , two presentations h t = f

Original: ( s t ) and h t +1 = f
Revised: ( s t ) and h t +1 = f

Original: ( s t +1 ) should be similar.
Revised: ( s t +1 ) should be similar.

Original: To do so, we use the Siamese network [5] and its modern training techniques from SimSiam [11].
Revised: To do so, we use the Siamese network [5] and its modern training techniques from SimSiam [11].

Original: Specially, we maximize the similarity between q ( g ( f ( s ))) and stop_gradient ( g ( f ( s ′ ))) where q and g are multilayer perceptrons.
Revised: Specially, we maximize the similarity between q ( g ( f ( s ))) and stop_gradient ( g ( f ( s ′ ))) where q and g are multilayer perceptrons.

Original: More details can be found in appendix.
Revised: More details can be found in appendix.

Original: 4.3 Unsupervised exploration
Revised: 4.3 Unsupervised exploration

Original: The objective of unsupervised exploration is to maximize the diversity of the data seen by the agent, imitating how intelligent creatures develop recognition.
Revised: The objective of unsupervised exploration is to maximize the diversity of the data seen by the agent, imitating how intelligent creatures develop recognition.

Original: An observation s t is first encoded using the encoder f to get representation h t = f
Revised: An observation s t is first encoded using the encoder f to get representation h t = f

Original: ( s t ) .
Revised: ( s t ) .

Original: The policy takes representation h t as input and generates action a t of the same dimension.
Revised: The policy takes representation h t as input and generates action a t of the same dimension.

Original: The policy distribution is a tanh Gaussian following prior work in unsupervised reinforcement learning [78, 41].
Revised: The policy distribution is a tanh Gaussian following prior work in unsupervised reinforcement learning [78, 41].

Original: We resort to maximizing the nonparametric entropy of the representation space which has been widely used in unsupervised reinforcement learning [66, 4, 44, 78].
Revised: We resort to maximizing the nonparametric entropy of the representation space which has been widely used in unsupervised reinforcement learning [66, 4, 44, 78].

Original: By explicitly encouraging the agents to visit diverse states, it effectively encourages the agents to explore environments, and thus learn more diverse behaviors.
Revised: By explicitly encouraging the agents to visit diverse states, it effectively encourages the agents to explore environments, and thus learn more diverse behaviors.

Original: The estimator computes the distance between each particle h i =
Revised: The estimator computes the distance between each particle h i =

Original: f ( s i ) and its k -th nearest neighbor h ⋆i .
Revised: f ( s i ) and its k -th nearest neighbor h ⋆i .

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ 2 is ℓ 2 norm.
Revised: where ∥ · ∥ 2 is ℓ 2 norm.

Original: We associate each transition from experience replay { ( s, a, s ′ ) } with an intrinsic reward given by
Revised: We associate each transition from experience replay { ( s, a, s ′ ) } with an intrinsic reward given by

Original: [Equation]
Revised: [Equation]

Original: Putting everything together, the diagram of our method is shown in Figure 1.
Revised: Putting everything together, the diagram of our method is shown in Figure 1.

Original:  Results
Revised:  Results

Original: It is important to stress that this work focuses on studying how combined exploratory pretraining and static datasets contribute to representation learning in artificial agents and not on developing a new, state-of-the-art, methodology for representation learning.
Revised: It is important to stress that this work focuses on studying how combined exploratory pretraining and static datasets contribute to representation learning in artificial agents and not on developing a new, state-of-the-art, methodology for representation learning.

Original: Nevertheless, to better situate our results in the context of existing work
Revised: Nevertheless, to better situate our results in the context of existing work

Original: [Figure]
Revised: [Figure]

Original: we provide strong baselines in our experiments
Revised: we provide strong baselines in our experiments

Original: [Figure]
Revised: [Figure]

Original: Image classification We test PALM on image classification tasks using a ResNet-18 as encoder f .
Revised: Image classification We test PALM on image classification tasks using a ResNet-18 as encoder f .

Original: The RL agent interacts with the environment for 50M steps, and trains representation using the procedure described in the method section.
Revised: The RL agent interacts with the environment for 50M steps, and trains representation using the procedure described in the method section.

Original: The hyperparameters of the representation learning follow the default hyperparameters of SimSiam [11] and more details can be found in the supplemental material.
Revised: The hyperparameters of the representation learning follow the default hyperparameters of SimSiam [11] and more details can be found in the supplemental material.

Original: We compare our method with a series of strong methods in static datasets, including state-of-the-art contrastive representation learning methods SimCLR [10] and SimSiam, supervised trained model, and CLIP [55] which is pretrained on a large-scale paired image-text datasets.
Revised: We compare our method with a series of strong methods in static datasets, including state-of-the-art contrastive representation learning methods SimCLR [10] and SimSiam, supervised trained model, and CLIP [55] which is pretrained on a large-scale paired image-text datasets.

Original: Figure 3 shows the linear classification performance of each method, PALM is surprisingly effective given the lack of data augmentation and real data.
Revised: Figure 3 shows the linear classification performance of each method, PALM is surprisingly effective given the lack of data augmentation and real data.

Original: Note that the baselines do not have an option not to use data augmentation.
Revised: Note that the baselines do not have an option not to use data augmentation.

Original: Additionally, adding the same data augmentation techniques used in SimSiam to our method further improves the results, matching or outperforming current SOTA static representation learning algorithms.
Revised: Additionally, adding the same data augmentation techniques used in SimSiam to our method further improves the results, matching or outperforming current SOTA static representation learning algorithms.

Original: Compared with the SOTA method SimSiam, PALM + data augmentation actively explore an environment instead of using a static dataset
Revised: Compared with the SOTA method SimSiam, PALM + data augmentation actively explore an environment instead of using a static dataset

Original: [Table]
Revised: [Table]

Original: and the representation learning and data augmentation are applied to synthesized temporal pair instead of two views of the same image
Revised: and the representation learning and data augmentation are applied to synthesized temporal pair instead of two views of the same image

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: of-distribution detection
Revised: of-distribution detection

Original: While some prior work demonstrated that self-supervised learning approaches significantly improve OOD detection performance [25, 15], their self-supervised pre-train heavily relies on domain-specific data augmentations, we want to study how PALM performs on
Revised: While some prior work demonstrated that self-supervised learning approaches significantly improve OOD detection performance [25, 15], their self-supervised pre-train heavily relies on domain-specific data augmentations, we want to study how PALM performs on

Original: OOD benchmarks.
Revised: OOD benchmarks.

Original: We compare PALM with standard supervised model [22, 24] and contrastive representation learning based methods including SupCLR [34] and CSI [71].
Revised: We compare PALM with standard supervised model [22, 24] and contrastive representation learning based methods including SupCLR [34] and CSI [71].

Original: CSI demonstrates state-of-the-art results by combining contrastive representation learning and supervised learning.
Revised: CSI demonstrates state-of-the-art results by combining contrastive representation learning and supervised learning.

Original: Table 1 shows the comparison on multiple datasets.
Revised: Table 1 shows the comparison on multiple datasets.

Original: PALM outperforms CSI in 4 out of 5 out-of-the-distribution datasets, despite using only synthesized data and no data augmentation.
Revised: PALM outperforms CSI in 4 out of 5 out-of-the-distribution datasets, despite using only synthesized data and no data augmentation.

Original: We attribute the effectiveness of PALM on
Revised: We attribute the effectiveness of PALM on

Original: OOD detection to active exploration stimulates the model to learn the invariance of data.
Revised: OOD detection to active exploration stimulates the model to learn the invariance of data.

Original: In doing so, the representation is optimized for distinguishing different observations.
Revised: In doing so, the representation is optimized for distinguishing different observations.

Original: Data efficient representation in RL In this experiment, we are interested in testing
Revised: Data efficient representation in RL In this experiment, we are interested in testing

Original: PALM pre-trained model for downstream reinforcement learning tasks.
Revised: PALM pre-trained model for downstream reinforcement learning tasks.

Original: To do so, we focus our experimentation on the Atari 100k benchmark introduced by Kaiser et al.
Revised: To do so, we focus our experimentation on the Atari 100k benchmark introduced by Kaiser et al.

Original: [29], in which agents are allowed only 100k steps of interaction with their environment.
Revised: [29], in which agents are allowed only 100k steps of interaction with their environment.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Exploratory
Revised: Exploratory

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: 250M
Revised: 250M

Original: ATC
Revised: ATC

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3M
Revised: 3M

Original: SGI
Revised: SGI

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 6M
Revised: 6M

Original: Offline-data Pretraining + Finetuning
Revised: Offline-data Pretraining + Finetuning

Original: ATC 0.
Revised: ATC 0.

Original: 3M
Revised: 3M

Original: SGI 0.
Revised: SGI 0.

Original: 6M
Revised: 6M

Original: Synthesized Exploratory Pretraining + Finetuning
Revised: Synthesized Exploratory Pretraining + Finetuning

Original: PALM (ours) 0.
Revised: PALM (ours) 0.

Original: 50M + 6M
Revised: 50M + 6M

Original: There has been extensive usage of
Revised: There has been extensive usage of

Original: Atari for representation learning and exploratory pretraining [8, 44, 63, 21].
Revised: Atari for representation learning and exploratory pretraining [8, 44, 63, 21].

Original: Finetuning after exploratory pretraining is highly effective [e.g., 63, 8], providing strong baselines to compare to.
Revised: Finetuning after exploratory pretraining is highly effective [e.g., 63, 8], providing strong baselines to compare to.

Original: Unlike experiments in
Revised: Unlike experiments in

Original: CIFAR datasets, here we train an unconditional StyleGAN model in a dataset collected by running a double DQN [47, 73] agent.
Revised: CIFAR datasets, here we train an unconditional StyleGAN model in a dataset collected by running a double DQN [47, 73] agent.

Original: The dataset consists of 50M frames for each game.
Revised: The dataset consists of 50M frames for each game.

Original: We use 6M steps for exploratory pretraining in the synthesized environment and we utilize DrQ [37] for online finetuning.
Revised: We use 6M steps for exploratory pretraining in the synthesized environment and we utilize DrQ [37] for online finetuning.

Original: Table 2 shows the comparison of
Revised: Table 2 shows the comparison of

Original: PALM with baselines.
Revised: PALM with baselines.

Original: Comparing with exploratory pretraining baselines (APT [44], ATC [68], and SGI [63]) that are pretrained in Atari games directly, PALM achieves surprisingly good results considering that it is trained in a simple synthesized environment.
Revised: Comparing with exploratory pretraining baselines (APT [44], ATC [68], and SGI [63]) that are pretrained in Atari games directly, PALM achieves surprisingly good results considering that it is trained in a simple synthesized environment.

Original: PALM achieves higher medium and mean score than DrQ in Atari 100k despite the online finetuning of PALM is the same as DrQ, demonstrating the benefit of pretrained representation.
Revised: PALM achieves higher medium and mean score than DrQ in Atari 100k despite the online finetuning of PALM is the same as DrQ, demonstrating the benefit of pretrained representation.

Original: Compared with offline-data pretraining baselines (ATC and SGI), there is still a gap between PALM and the state-of-the-arts, which is probably due to PALM’s environment being trained on observations only while ATC and SGI are trained on both observations and actions in a model-based style.
Revised: Compared with offline-data pretraining baselines (ATC and SGI), there is still a gap between PALM and the state-of-the-arts, which is probably due to PALM’s environment being trained on observations only while ATC and SGI are trained on both observations and actions in a model-based style.

Original: Worth mentioning that while PALM utilizes 50M offline action-free experience for pretraining, the amount is less than the 250M online action labeled experience used in APT.
Revised: Worth mentioning that while PALM utilizes 50M offline action-free experience for pretraining, the amount is less than the 250M online action labeled experience used in APT.

Original: We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform DrQ without data augmentation given the same amount of online experience.
Revised: We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform DrQ without data augmentation given the same amount of online experience.

Original:  Analysis
Revised:  Analysis

Original: [Figure]
Revised: [Figure]

Original: Persistent environment and active interaction
Revised: Persistent environment and active interaction

Original: are important.
Revised: are important.

Original: Next, we perform an ablation study, to study the importance of a persistent environment.
Revised: Next, we perform an ablation study, to study the importance of a persistent environment.

Original: We aim to answer the following question.
Revised: We aim to answer the following question.

Original: How important it is to have action (controlled by α ) and state (controlled by β ) in the transition dynamics?
Revised: How important it is to have action (controlled by α ) and state (controlled by β ) in the transition dynamics?

Original: When β =0, there is effectively no state, i.e. , its a bandit.
Revised: When β =0, there is effectively no state, i.e. , its a bandit.

Original: As shown in Figure 4, when decreasing the importance of states which is denoted as PALM ( α = 0 .
Revised: As shown in Figure 4, when decreasing the importance of states which is denoted as PALM ( α = 0 .

Original: 5 β = 0 . 5 ), the performance decreases.
Revised: 5 β = 0 . 5 ), the performance decreases.

Original: The extreme case β = 0 shows a significant drop in accuracy.
Revised: The extreme case β = 0 shows a significant drop in accuracy.

Original: These results indicate that it is important to have temporal persistent transition dynamics.
Revised: These results indicate that it is important to have temporal persistent transition dynamics.

Original: When α = 0 , action is ignored in transition dynamics, this ablative baseline performs significantly worse than its default α = 0 .
Revised: When α = 0 , action is ignored in transition dynamics, this ablative baseline performs significantly worse than its default α = 0 .

Original: When replacing the agent’s action with random noise which is denoted as PALM + random action, we also see a significant performance drop.
Revised: When replacing the agent’s action with random noise which is denoted as PALM + random action, we also see a significant performance drop.

Original: These results show having active interaction is important to the results of PALM.
Revised: These results show having active interaction is important to the results of PALM.

Original: We present the results of more combinations of α and β in Figure 5, the results show that as α decreases from 0.95 to 0, because of the decreasing importance of action, the accuracy continues to drop.
Revised: We present the results of more combinations of α and β in Figure 5, the results show that as α decreases from 0.95 to 0, because of the decreasing importance of action, the accuracy continues to drop.

Original: α = 0 .
Revised: α = 0 .

Original: 95 in general performs better than α = 1 . 0 , indicates mixing a small amount of random noise with action helps exploration and improves representation learning.
Revised: 95 in general performs better than α = 1 . 0 , indicates mixing a small amount of random noise with action helps exploration and improves representation learning.

Original: When α or β is zero, the results are significantly worse.
Revised: When α or β is zero, the results are significantly worse.

Original: We also see that β ∈ [0 . 5 , 0 .
Revised: We also see that β ∈ [0 . 5 , 0 .

Original: 75] performs significantly better than other values, showing that persistence of dynamics is crucial.
Revised: 75] performs significantly better than other values, showing that persistence of dynamics is crucial.

Original: [Figure]
Revised: [Figure]

Original: Exploratory Learning as Curriculum Learning.
Revised: Exploratory Learning as Curriculum Learning.

Original: As we have seen, representations learned from exploratory learning in a synthesized environment can be surprisingly powerful.
Revised: As we have seen, representations learned from exploratory learning in a synthesized environment can be surprisingly powerful.

Original: We are interested in having a perhaps preliminary study of the reasons.
Revised: We are interested in having a perhaps preliminary study of the reasons.

Original: To do so, we compare PALM with its passive version (passive PALM), static version (static PALM), and methods proposed in Jahanian et al.
Revised: To do so, we compare PALM with its passive version (passive PALM), static version (static PALM), and methods proposed in Jahanian et al.

Original: In the random variant, the positives pairs are sampled from
Revised: In the random variant, the positives pairs are sampled from

Original: G ( z, c = c ′ )
Revised: G ( z, c = c ′ )

Original: (i.e. fix the class, take two random draws from z).
Revised: (i.e. fix the class, take two random draws from z).

Original: We denote this baseline as Random
Revised: We denote this baseline as Random

Original: Jahanian et al.
Revised: Jahanian et al.

Original: In Gaussian Jahanian et al, the positive pairs are given by z and z+w, where w is Gaussian.
Revised: In Gaussian Jahanian et al, the positive pairs are given by z and z+w, where w is Gaussian.

Original: Static PALM is not trained in the interactive environment, instead, it is trained with real images.
Revised: Static PALM is not trained in the interactive environment, instead, it is trained with real images.

Original: Specifically, we randomly sample two images of the same label from a dataset and use them as a temporal pair to train the representation learning of PALM.
Revised: Specifically, we randomly sample two images of the same label from a dataset and use them as a temporal pair to train the representation learning of PALM.

Original: We note that static PALM is related to SupCon [34], but it differs in that static PALM does not use any data augmentation while SupCon leverages a combination of data augmentations.
Revised: We note that static PALM is related to SupCon [34], but it differs in that static PALM does not use any data augmentation while SupCon leverages a combination of data augmentations.

Original: SupCon also uses a large number of negative examples from different categories.
Revised: SupCon also uses a large number of negative examples from different categories.

Original: Passive PALM is trained offline on the experience of PALM.
Revised: Passive PALM is trained offline on the experience of PALM.

Original: We first store all the online experience (50M in total) of PALM, then we use the saved experience to train passive PALM without having to interact with the environment.
Revised: We first store all the online experience (50M in total) of PALM, then we use the saved experience to train passive PALM without having to interact with the environment.

Original: We consider two options of passive PALM.
Revised: We consider two options of passive PALM.

Original: The first one is random passive PALM which randomly samples consecutive observations from the stored offline experience.
Revised: The first one is random passive PALM which randomly samples consecutive observations from the stored offline experience.

Original: The second one is ordered passive PALM which is trained on the same order of observations as in online experience.
Revised: The second one is ordered passive PALM which is trained on the same order of observations as in online experience.

Original: With the number of data increasing, PALM keeps improving with the amount of training data and significantly outperforms static PALM even though static PALM is trained on real datasets.
Revised: With the number of data increasing, PALM keeps improving with the amount of training data and significantly outperforms static PALM even though static PALM is trained on real datasets.

Original: We believe the reason is that the diverse synthesized data collected by PALM can greatly help to learn representation.
Revised: We believe the reason is that the diverse synthesized data collected by PALM can greatly help to learn representation.

Original: In the low data regime, carefully designed non-active learning based methods [26] outperforms PALM, possibly due to RL training being sample inefficient.
Revised: In the low data regime, carefully designed non-active learning based methods [26] outperforms PALM, possibly due to RL training being sample inefficient.

Original: With the increasing number of synthesized samples, PALM continues to improve and significantly outperform all baselines.
Revised: With the increasing number of synthesized samples, PALM continues to improve and significantly outperform all baselines.

Original: The results also show that PALM scales better than non active learning based methods.
Revised: The results also show that PALM scales better than non active learning based methods.

Original: PALM nearly always outperforms random passive PALM, demonstrating the effectiveness of exploratory learning.
Revised: PALM nearly always outperforms random passive PALM, demonstrating the effectiveness of exploratory learning.

Original: Interestingly but not surprisingly, ordered passive PALM performs nearly the same as PALM, indicating the exploratory learning behavior of PALM might play an important role in curriculum learning.
Revised: Interestingly but not surprisingly, ordered passive PALM performs nearly the same as PALM, indicating the exploratory learning behavior of PALM might play an important role in curriculum learning.

Original: The agent is optimized to search for novel samples in terms of their representation, by doing so, the searched samples should become more difficult to distinguish.
Revised: The agent is optimized to search for novel samples in terms of their representation, by doing so, the searched samples should become more difficult to distinguish.

Original: We believe a comprehensive understanding of the effectiveness of PALM is out of the scope of this work, we tend to leave it as future work.
Revised: We believe a comprehensive understanding of the effectiveness of PALM is out of the scope of this work, we tend to leave it as future work.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Diversity analysis.
Revised: Diversity analysis.

Original: In order to measure the diversity of the sampled synthetic data, we randomly sample 10K images from the CIFAR-10 train set and generate 10K images from StyleGAN by random sample latents.
Revised: In order to measure the diversity of the sampled synthetic data, we randomly sample 10K images from the CIFAR-10 train set and generate 10K images from StyleGAN by random sample latents.

Original: We also generate 10K images with PALM in latent space.
Revised: We also generate 10K images with PALM in latent space.

Original: For each sample in each set, we find its closest neighbor in Inception feature space (obtained after the pooling layer).
Revised: For each sample in each set, we find its closest neighbor in Inception feature space (obtained after the pooling layer).

Original: The coverage is defined as the proportion of nearest neighbors that are unique in the train set.
Revised: The coverage is defined as the proportion of nearest neighbors that are unique in the train set.

Original: A better active exploration method would produce samples that are equally likely to be close to any image in the train set.
Revised: A better active exploration method would produce samples that are equally likely to be close to any image in the train set.

Original: PALM achieves a better coverage score than random sampling in latent space.
Revised: PALM achieves a better coverage score than random sampling in latent space.

Original: Although the Euclidean distance in Inception feature space is inaccurate, the better coverage achieved by PALM indicates that by exploring the latent manifold it gets more diverse data.
Revised: Although the Euclidean distance in Inception feature space is inaccurate, the better coverage achieved by PALM indicates that by exploring the latent manifold it gets more diverse data.

Original:  Conclusion and discussion
Revised:  Conclusion and discussion

Original: Our work presents a method to connect learning from large, diverse static datasets and learning by interaction, and demonstrates promising results of doing so.
Revised: Our work presents a method to connect learning from large, diverse static datasets and learning by interaction, and demonstrates promising results of doing so.

Original: We believe it opens up an interesting future direction for advancing both reinforcement learning and self-supervised visual representation learning.
Revised: We believe it opens up an interesting future direction for advancing both reinforcement learning and self-supervised visual representation learning.

Original: For limitations, since it is computational expensive to draw samples from large generative model, interacting with such environment is significantly slower than well optimized physical simulations used in RL, e.g. , Mujoco.
Revised: For limitations, since it is computational expensive to draw samples from large generative model, interacting with such environment is significantly slower than well optimized physical simulations used in RL, e.g. , Mujoco.

Original: Addressing such limitation would make this work more accessible and facilitate future research.
Revised: Addressing such limitation would make this work more accessible and facilitate future research.

Original: For future work it would be interesting to design or learn better transition dynamics.
Revised: For future work it would be interesting to design or learn better transition dynamics.

Original: It would be also interesting to evaluate our method in RL settings where a lot of action-free data are available and won’t have to be collected (Minecraft for example).
Revised: It would be also interesting to evaluate our method in RL settings where a lot of action-free data are available and won’t have to be collected (Minecraft for example).

Original: It would be also interesting to use a language conditioned generative model as an alternative e.g. Dalle [57, 58], so the environment has more complexity by prompting it with natural language.
Revised: It would be also interesting to use a language conditioned generative model as an alternative e.g. Dalle [57, 58], so the environment has more complexity by prompting it with natural language.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank Guillaume Desjardins for insightful discussion and giving constructive comments.
Revised: We would like to thank Guillaume Desjardins for insightful discussion and giving constructive comments.

Original: We would also like to thank anonymous reviewers for their helpful feedbackChecklist
Revised: We would also like to thank anonymous reviewers for their helpful feedback

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The instructions will be provided in the supplemental material and we plan to release the code and pre-trained models to facilitate further research.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] The full details will be provided in supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Our experiments on classification and OOD detection experiments are not sensitive to random seeds.

Original: For our RL experiments, we run the evaluate multiple times and include the details in experiment section and supplementary material.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] The full details will be provided in supplementary material.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]

Original: The datasets we use are public and do not contain personally identifiable information.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Experiment Details on Representation Pretraining

Original: We follow the same optimization step of SimSiam [11].

Original: The input image size is 32 × 32.

Original: We use SGD with base lr =0 .

Original: 03 and a cosine decay [46] schedule for 800 epochs, weight decay =0 .

Original: 0005 , momentum =0 . 9 , and batch size =512 .

Original: The encoder f is a

Original: ResNet-18 [22] with the last fc layer removed.

Original: We use the same projection and prediction networks in SimSiam.

Original: Specifically, the projection MLP g has 3 layers, and has BN applied to each fully-connected (fc) layer, including its output fc.

Original: There is no ReLU for the last fc layer in g and the hidden size is 2048.

Original: The prediction MLP ( h ) has 2 layers, and has BN applied to its hidden fc layers.

Original: Its output fc does not have BN or ReLU.

Original: The dimension of h ’s input and output is d =2048 , and h ’s hidden layer’s dimension is 512 , making h a bottleneck structure.

Original: After pretraining, the representation encoder f is finetuned for downstream tasks.

Original: B Representation Learning Loss and Pseudocode

Original: We use the symmetrized cosine similarity loss from SimSiam.

Original: Specifically, for two consecutive observations s 1 and s 2 , denoting the two output vectors as p 1 = h

Original: ( g ( f ( s 1 ))) and z 2 = g

Original: ( f ( s 2 )) , we minimize their negative cosine similarity:

Original: [Equation]

Original: where ∥·∥ 2 is ℓ 2 -norm.

Original: And the loss function is defined as

Original: [Equation]

Original: Algorithm 1 shows the pseudocode of representation learning.

Original: C Experiment and dataset details

Original: Datasets CIFAR-10 [38] and CIFAR-100 [38] consist of 50,000 training and 10,000 test images with 10 and 20 (super-class) image classes, respectively.

Original: For CIFAR-10, out-of-distribution (OOD) samples are as follows: SVHN [49] consists of 26,032 test images with 10 digits, resized LSUN [80] consists of 10,000 test images of 10 different scenes, Interp.

Original: consists of 10,000 test images of linear interpolation of CIFAR-10 test images.

Original: CelebA [45], a labeled dataset consisting of over 200 ,face images and each with 40 attribute annotation.

Original: The Atari observation dataset is collected using double DQN [47, 73] with n-step return (n=3) and 3 frames stacking.

Original: In total we collected 50M images and each observation is resized to 64x64x3.

Original: Model details For CIFAR10, we use pretrained StyleGAN available at the official website of

Original: StyleGAN-Ada[31] 2 .

Original: This pretrained model is a conditional StyleGAN that achieves best FID score.

Original: We also experimented with the model with best Inception score 3 but did not observe significant difference in results.

Original: For CIFAR100, since we did not find publicly available pretrained models, we trained one by ourself using the StyleGAN official code 4 .

Original: For Atari, we train an unconditional StyleGAN using the official code.

Original: It is difficult to train StyleGAN to converge on our Atari dataset with the default hyperparameters.

Original: We found that increasing the batch size to 256 significantly help stabilize the training and accelerate convergence.

Original: Linear classification

Original: The quality of the pretrained representations is evaluated by training a supervised linear classifier on frozen representations h in the training set, and then testing it in the validation set.

Original: We use batch size of 512, Adam optimizer with learning rate being 0 . 001 .

Original: We finetune for 100 epochs in linear classification experiments.

Original: Atari games Our evaluation metric for an agent on a game is human-normalized score (HNS), defined as agent _ score − random _ score human _ score − random _ score .

Original: We calculate this per game by averaging scores overevaluation trajectories at the end of training, and across 10 random seeds for training.

Original: We report both mean (Mn) and median (Mdn) HNS over the 26 Atari-100K games, as well as on how many games a method achieves super-human performance ( > H) and greater than random performance ( > 0).

Original: D Experiment details on RL

Original: We adhere closely to the parameter settings from URLB [41], which our implementation is based upon.

Original: We list the hyper-parameters in Table

Original: We employ clipped double Q-learning for the critic, where each Q -function is parametrized as a 3-layer MLP with ReLU activations after each layer except of the last.

Original: The actor is also a 3-layer MLP with ReLU s that outputs mean and covariance for the diagonal Gaussian that represents the policy.

Original: The hidden dimension is set to 1024 for both the critic and actor.

Original: The actor and critic networks both have separate encoders, although we share the weights of the conv layers between them.

Original: Furthermore, only the critic optimizer is allowed to update these weights (e.g. we stop the gradients from the actor before they propagate to the shared conv layers).

Original: [Table]

Original: [Table]

Original: [Table]

Original: log( r + 1 . 0)

Original: [Table]

Original: (512 + |A| )

Original: → 1024 → 512 ReLU MLP

Original: [Table]

Original: (2 × 512) → 1024 → |A| ReLU MLP

Original: [Table]

Original: # f: backbone # g: projection mlp # h: prediction mlp for x1, x2 in loader: # load tuples of consecutive observations h1, h2 = f(x1), f(x2) # representations, n-by-d z1, z2 = f(h1), f(h2)

Original: # projections, n-by-d p1, p2 = h(z1), h(z2)

Original: # predictions, n-by-d

Original: [-L = D(p1, z2)/2 + D(p2, z1)/
-]
Revised: [-L = D(p1, z2)/2 + D(p2, z1)/
-]

Original: # loss

Original: L.backward() # back-propagate update(f, h) # SGD update def D(p, z): # negative cosine similarity z = z.detach() # stop gradient p = normalize(p, dim=1)

Original: # l2-normalize z = normalize(z, dim=1)

Original: # l2-normalize return -(p*z).sum(dim=1).mean()

Original: # P: policy # G: generator # alpha, beta: hyperparameters control the importance of action and state.

Original: for t in range(episode_length): # z = random_normal

Original: () # sample a random latent from Gaussian if t == 0:

Original: # first step of an episode z_t =

Original: z c =

Original: random_label() # sample a random class label s_t

Original: = G(z_t, c) # generate an observation a_t = P(s_t) # sample an action from policy else: zprime = alpha *

Original: a_t + (1 - alpha) *

Original: z z_t = beta * z_t + (1 - beta) zprime s_t

Original: = G(z_t, c) # generate an observation a_t = P(s_t) # sample an action from policy