Original: C URIOSITY -D RIVEN U NSUPERVISED D ATA C OLLECTION FOR O FFLINE R EINFORCEMENT L EARNING
Revised: C URIOSITY -D RIVEN U NSUPERVISED D ATA C OLLECTION FOR O FFLINE R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In offline reinforcement learning (RL), while the majority of efforts are focusing on engineering sophisticated learning algorithms given a fixed dataset, very few works have been carried out to improve the dataset quality itself.
Revised: In offline reinforcement learning (RL), while the majority of efforts are focusing on engineering sophisticated learning algorithms given a fixed dataset, very few works have been carried out to improve the dataset quality itself.

Original: More importantly, it is even challenging to collect a task-agnostic dataset such that the offline RL agent can learn multiple skills from it.
Revised: More importantly, it is even challenging to collect a task-agnostic dataset such that the offline RL agent can learn multiple skills from it.

Original: In this paper, we propose a C uriosity-driven U nsupervised D ata C ollection (CUDC) method to improve the data collection process.
Revised: In this paper, we propose a C uriositydriven U nsupervised D ata C ollection (CUDC) method to improve the data collection process.

Original: Specifically, we quantify the agent’s internal belief to estimate the probability of the k -step future states being reachable from the current states.
Revised: Specifically, we quantify the agent’s internal belief to estimate the probability of the k -step future states being reachable from the current states.

Original: Different from existing approaches that implicitly assume limited feature space with  fixed environment steps, CUDC is capable of adapting the number of environment steps to explore.
Revised: Different from existing approaches that implicitly assume limited feature space with a fixed temporal distance between current and next states, CUDC is capable of adapting how many steps into the future that the dynamics model should predict.

Original: Thus, the feature representation can besubstantially diversified with the dynamics information.
Revised: Thus, the feature representation can be diversified with the dynamics information.

Original: With this adaptive reachability mechanism in place, the agent can navigate itself to collect higher-quality data with curiosity.
Revised: With this adaptive reachability mechanism in place, the agent can navigate itself to collect higher-quality data with curiosity.

Original: Empirically, CUDC surpasses existing unsupervised methods in sample efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.
Revised: Empirically, CUDC surpasses existing unsupervised methods in sample efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning has demonstrated remarkable breakthroughs in games, robotics, and navigation in complex environments (Kiran et al., 2021; Singh et al., 2022; Sun et al., 2022a).
Revised: Deep reinforcement learning has demonstrated remarkable breakthroughs in games, robotics, and navigation in complex environments (Kiran et al., 2021; Singh et al., 2022; Sun et al., 2022a).

Original: For online RL, agents constantly update the policy to acquire different skills through active interactions with the environments.
Revised: For online RL, agents constantly update the policy to acquire different skills through active interactions with the environments.

Original: However, online RL is impractical in many real-world environments as direct interactions with the environments might be expensive or dangerous (Kiran et al., 2021; Singh et al., 2022).
Revised: However, online RL is impractical in many real-world environments as direct interactions with the environments might be expensive or dangerous (Kiran et al., 2021; Singh et al., 2022).

Original: In recent years, offline RL has become a promising research area to cope with limited interactions, where agents learn a policy exclusively from previously-collected experiences stored in a fixed dataset (Levine et al., 2020; Kostrikov et al., 2021; Fujimoto & Gu, 2021).
Revised: In recent years, offline RL has become a promising research area to cope with limited interactions, where agents learn a policy exclusively from previously-collected experiences stored in a fixed dataset (Levine et al., 2020; Kostrikov et al., 2021; Fujimoto & Gu, 2021).

Original: In view of the growing popularity of offline RL, the majority of current research focuses on modelcentric practices by successively developing new algorithms (Kumar et al., 2020; Janner et al., 2021;
Revised: In view of the growing popularity of offline RL, the majority of current research focuses on modelcentric practices by successively developing new algorithms (Kumar et al., 2020; Janner et al., 2021;

Original: Matsushima et al., 2021; Emmons et al., 2022; Kumar et al., 2022).
Revised: Matsushima et al., 2021; Emmons et al., 2022; Kumar et al., 2022).

Original: Despite the rapid progress in these algorithmic advances, their performances are inevitably limited by the quality of the precollected dataset itself.
Revised: Despite the rapid progress in these algorithmic advances, their performances are inevitably limited by the quality of the precollected dataset itself.

Original: Recently, the concept of data-centric approaches has become critical in the machine learning community, emphasizing the importance of improving the training data quality over algorithmic advances (Ng, 2021;
Revised: Recently, the concept of data-centric approaches has become critical in the machine learning community, emphasizing the importance of improving the training data quality over algorithmic advances (Ng, 2021;

Original: Motamedi et al., 2021; Patel et al., 2022).
Revised: Motamedi et al., 2021; Patel et al., 2022).

Original: Motivated by this, the offline RL research community attempts to eye on ways of engineering the training data (Prudencio et al., 2022).
Revised: Motivated by this, the offline RL research community attempts to eye on ways of engineering the training data (Prudencio et al., 2022).

Original: To focus on more useful data, one solution is to exploit the sample importance with sampling (Zhang et al., 2020) or re-weighting (Wu et al., 2021).
Revised: To focus on more useful data, one solution is to exploit the sample importance with sampling (Zhang et al., 2020) or re-weighting (Wu et al., 2021).

Original: Different from this approach, we aim to collect a higher-quality dataset that can be directly used for offline RL agents.
Revised: Different from this approach, we aim to collect a higher-quality dataset that can be directly used for offline RL agents.

Original: More importantly, it is even more desirable yet challenging to collect a task-agnostic dataset such that offline RL agents are able to extract effective policies for multiple downstream tasks.
Revised: More importantly, it is even more desirable yet challenging to collect a task-agnostic dataset such that offline RL agents are able to extract effective policies for multiple downstream tasks.

Original: To analyze and understand these challenges, ExORL (Yarats et al., 2022) empirically shows that unsupervised
Revised: To analyze and understand these challenges, ExORL (Yarats et al., 2022) empirically shows that unsupervised

Original: RL methods are superior to supervised methods to collect the exploratory data that allows even vanilla off-policy RL algorithms to effectively learn offline and acquire different skills.
Revised: RL methods are superior to supervised methods to collect the exploratory data that allows even vanilla off-policy RL algorithms to effectively learn offline and acquire different skills.

Original: Nevertheless, the existing methods learn the feature representation by pre-defining a fixed number of environment steps between current states and future states, which  limits the diversity in  feature representation.
Revised: Nevertheless, these existing methods pre-define a fixed temporal distance between current states and future states to train the models, which implicitly limits the diversity in the learned feature representation.

Original: As we observe that fixing the environment step limits the feature space and results in low-quality dataset, it is desired to enhance feature representation by exploiting the reachability from current state to more distant future states.
Revised: As we observe that fixing the temporal distance between current and future states may limit the feature space and result in low-quality dataset, it is desired to enhance feature representation by exploiting the reachability from current state to more distant future states.

Original: Although existing works of reachability analysis have been introduced in RL (Savinov et al., 2019; P´er´e et al., 2018; Ivanovic et al., 2019; Yu et al., 2022), these approaches are however not directly applicable.
Revised: Although existing works of reachability analysis have been introduced in RL (Savinov et al., 2019; P´er´e et al., 2018; Ivanovic et al., 2019; Yu et al., 2022), these approaches are however not directly applicable.

Original: For example, Savinov et al.
Revised: For example, Savinov et al.

Original: (2019) only considers the reachability in a binary case, and it requires extensive comparisons to the stored embeddings in memory.
Revised: (2019) only considers the reachability in a binary case, and it requires extensive comparisons to the stored embeddings in memory.

Original: Moreover, the reachability in goal space exploration requires kernel density estimation, increasing the computational cost substantially.
Revised: Moreover, the reachability in goal space exploration requires kernel density estimation, increasing the computational cost substantially.

Original: Different from these approaches, we propose a C uriosity-driven U nsupervised D ata C ollection (CUDC) method with a novel reachability module.
Revised: Different from these approaches, we propose a C uriosity-driven U nsupervised D ata C ollection (CUDC) method with a novel reachability module.

Original: Inspired by the fact that human curiosity can enhance the learning process by motivating human beings on the novel knowledge that is beyond ones’ perception (Rossing & Long, 1981; Markey & Loewenstein, 2014; Sun et al., 2022b), CUDC facilitates the agent to collect a dataset curiously without any task-specific reward.
Revised: Inspired by the fact that human curiosity can enhance the learning process by motivating human beings on the novel knowledge that is beyond ones’ perception (Rossing & Long, 1981; Markey & Loewenstein, 2014; Sun et al., 2022b), CUDC facilitates the agent to collect a dataset curiously without any task-specific reward.

Original: In particular, we define the reachability module to characterize the probability of a k -step future state being reachable from the current state, with no episodic memory  required.
Revised: In particular, we define the reachability module to characterize the probability of a k -step future state being reachable from the current state, with no episodic memory or feature space density modeling required.

Original: This module allows the agent to automatically determine the number of environment steps to explore without directly modeling the feature space density, where the  feature representation can be incorporated with the information of dynamics.
Revised: This module allows the agent to automatically determine how many steps into the future that the dynamics model should predict, where the learned feature representation can be incorporated with the information of dynamics.

Original: Compared with the existing unsupervised methods, it avoids relying on the fixed feature space by gradually expanding to more distant future states.
Revised: Compared with the existing unsupervised methods, it avoids relying on the fixed feature space by gradually expanding to more distant future states.

Original: With the enhanced representation learning, a mixed intrinsic reward encourages curious exploration towards more meaningful state-action space as well as the under-learned states.
Revised: With the enhanced representation learning, a mixed intrinsic reward encourages curious exploration towards more meaningful state-action space as well as the under-learned states.

Original: As a result, the collected dataset can lead to improved sample efficiency and better performances in downstreabelongs to noveltym offline RL tasks.
Revised: As a result, the collected dataset can lead to improved sample efficiency and better performances in downstream offline RL tasks.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: We are the first to introduce reachability for improving data collection in offline RL, which is defined in a more efficient way and can enable the agent to navigate curiosity-driven learning coherently.
Revised: We are the first to introduce reachability for improving data collection in offline RL, which is defined in a more efficient way and can enable the agent to navigate curiosity-driven learning coherently.

Original: We empirically show that adapting the number ofenvironment steps  to perform increasingly challengingfuture state prediction can enhance feature representation with information of the dynamics, thereby improving the collected dataset quality. 3) With the learned state and action representations, CUDC additionally incentivizes the agent to explore diverse state-action space as well as the under-learned states with high prediction errors through a mixed intrinsic reward and regularization.
Revised: We empirically show that adapting the number of steps between current and future states to perform increasingly challenging prediction can enhance feature representation with information of the dynamics, thereby improving the collected dataset quality. 3) With the learned state and action representations, CUDC additionally incentivizes the agent to explore diverse state-action space as well as the under-learned states with high prediction errors through a mixed intrinsic reward and regularization.

Original: Under the ExORL (Yarats et al., 2022) setting, CUDC outperforms the other unsupervised methods to collect the dataset that can be learned offline in multiple downstream tasks of the DeepMind control suite (Tassa et al., 2018).
Revised: Under the ExORL (Yarats et al., 2022) setting, CUDC outperforms the other unsupervised methods to collect the dataset that can be learned offline in multiple downstream tasks of the DeepMind control suite (Tassa et al., 2018).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Reachability in RL Savinov et al.
Revised: Reachability in RL Savinov et al.

Original: (2019) constructed a reachability network to estimate how many environment steps to take for reaching a particular state.
Revised: (2019) constructed a reachability network to estimate how many environment steps to take for reaching a particular state.

Original: It intrinsically rewards the agent to explore the state that is unreachable i.e., takes more than a fixed threshold step, from other states in memory.
Revised: It intrinsically rewards the agent to explore the state that is unreachable i.e., takes more than a fixed threshold step, from other states in memory.

Original: However, it only takes the binary case of reachability into consideration and is quite inefficient when comparing the similarity with all stored states in memory.
Revised: However, it only takes the binary case of reachability into consideration and is quite inefficient when comparing the similarity with all stored states in memory.

Original: In the goal exploration tasks, P´er´e et al.
Revised: In the goal exploration tasks, P´er´e et al.

Original: (2018) defined the reachability of a goal with an estimated density and proposed to sample increasingly difficult goals to reach during exploration.
Revised: (2018) defined the reachability of a goal with an estimated density and proposed to sample increasingly difficult goals to reach during exploration.

Original: Although the goal space can be learned in an unsupervised manner other than in a specifically engineered way, its sampling process requires a kernel density estimator, increasing the computational cost substantially.
Revised: Although the goal space can be learned in an unsupervised manner other than in a specifically engineered way, its sampling process requires a kernel density estimator, increasing the computational cost substantially.

Original: Following the similar idea, BARC (Ivanovic et al., 2019) adapts the initial state distribution gradually from easyto-reach to challenging-to-reach goals.
Revised: Following the similar idea, BARC (Ivanovic et al., 2019) adapts the initial state distribution gradually from easyto-reach to challenging-to-reach goals.

Original: As a result, agents can perform well even in a hard robotic control task.
Revised: As a result, agents can perform well even in a hard robotic control task.

Original: Recently, RCRL (Yu et al., 2022) has shown that leveraging reachability analysis (Hsu et al., 2021) can help learn an optimal safe policy by expanding the limited conservative feasible set to the largest feasible set of the state space.
Revised: Recently, RCRL (Yu et al., 2022) has shown that leveraging reachability analysis (Hsu et al., 2021) can help learn an optimal safe policy by expanding the limited conservative feasible set to the largest feasible set of the state space.

Original: Curiosity-Driven RL Curiosity-driven RL intrinsically encourages agents to explore the task environment in a human-like way, which is of vital importance when the task-specific rewards are sparse or absent (Aubret et al., 2019; Sun et al., 2022b).
Revised: Curiosity-Driven RL Curiosity-driven RL intrinsically encourages agents to explore the task environment in a human-like way, which is of vital importance when the task-specific rewards are sparse or absent (Aubret et al., 2019; Sun et al., 2022b).

Original: The main type of curiosity-driven RL is to incorporate an intrinsic reward that self-motivates agents to explore based on various aspects of the state, such as novelty (Bellemare et al., 2016), entropy (Seo et al., 2021; Liu & Abbeel, 2021b), reachability (Savinov et al., 2019), prediction errors (Pathak et al., 2017; Berseth et al., 2020), complexity (Campero et al., 2020), and uncertainty (Pathak et al., 2019; Sekar et al., 2020; Li et al.,
Revised: The main type of curiosity-driven RL is to incorporate an intrinsic reward that self-motivates agents to explore based on various aspects of the state, such as novelty (Bellemare et al., 2016), entropy (Seo et al., 2021; Liu & Abbeel, 2021b), reachability (Savinov et al., 2019), prediction errors (Pathak et al., 2017; Berseth et al., 2020), complexity (Campero et al., 2020), and uncertainty (Pathak et al., 2019; Sekar et al., 2020; Li et al.,

Original: Another type of curiosity-driven RL is to prioritize the experience replay towards underexplored states (Schaul et al., 2016; Zhao & Tresp, 2019; Brittain et al., 2019; Jiang et al., 2021).
Revised: Another type of curiosity-driven RL is to prioritize the experience replay towards underexplored states (Schaul et al., 2016; Zhao & Tresp, 2019; Brittain et al., 2019; Jiang et al., 2021).

Original: However, the curiosity mechanism can be further exploited and introduced into other components of RL for exploration, as shown in CCLF (Sun et al., 2022a).
Revised: However, the curiosity mechanism can be further exploited and introduced into other components of RL for exploration, as shown in CCLF (Sun et al., 2022a).

Original: Therefore, we propose to curiously adapt the environment step to explore more distant future states, enhancing the learned representation space.
Revised: Therefore, we propose to curiously adapt the temporal distance to explore more distant future states, enhancing the learned representation space with dynamics information.

Original: Meanwhile, our method regularizes the Q-learning by assigning the importance weights through a curiosity weight to focus more on under-learned tuples.
Revised: Meanwhile, our method regularizes the Q-learning by assigning the importance weights through a curiosity weight to focus more on under-learned tuples.

Original: Unsupervised Data Collection ExORL (Yarats et al., 2022) evaluates 9 unsupervised data collection algorithms, demonstrating a superior capability over supervised methods for multi-task offline learning.
Revised: Unsupervised Data Collection ExORL (Yarats et al., 2022) evaluates 9 unsupervised data collection algorithms, demonstrating a superior capability over supervised methods for multi-task offline learning.

Original: In particular, knowledge-driven models of ICM (Pathak et al., 2017),
Revised: In particular, knowledge-driven models of ICM (Pathak et al., 2017),

Original: Disagreement (Pathak et al., 2019), and RND (Burda et al., 2019) encourage agents to explore by maximizing the prediction errors of the states.
Revised: Disagreement (Pathak et al., 2019), and RND (Burda et al., 2019) encourage agents to explore by maximizing the prediction errors of the states.

Original: Data-driven models of APT (Liu & Abbeel, 2021b) and ProtoRL (Yarats et al., 2021) incentivize to uniformly explore the entire state space.
Revised: Data-driven models of APT (Liu & Abbeel, 2021b) and ProtoRL (Yarats et al., 2021) incentivize to uniformly explore the entire state space.

Original: By leveraging some prior information, competence-based models of DIAYN (Eysenbach et al., 2019), SMM (Lee et al., 2019) and APS (Liu & Abbeel, 2021a) encourage agents to learn diverse skills.
Revised: By leveraging some prior information, competence-based models of DIAYN (Eysenbach et al., 2019), SMM (Lee et al., 2019) and APS (Liu & Abbeel, 2021a) encourage agents to learn diverse skills.

Original: However, all these methods are not tailored for data collection; instead, they were originally proposed for online pretraining and fine-turning at task learning process as evaluated in the URLB benchmark (Laskin et al., 2021).
Revised: However, all these methods are not tailored for data collection; instead, they were originally proposed for online pretraining and fine-turning at task learning process as evaluated in the URLB benchmark (Laskin et al., 2021).

Original: Moreover, their feature space is limited by relying on a fixed environment step k between the current state and future state.
Revised: Moreover, their feature space is limited by relying on a fixed temporal distance k between the current state and future state for model training.

Original: Concurrently, Explore2Offline (Lambert et al., 2022) leverages the intrinsic model predictive control to simulate trajectories and their resulting predicted intrinsic rewards, but it does not consider the sample importance while updating the policy.
Revised: Concurrently, Explore2Offline (Lambert et al., 2022) leverages the intrinsic model predictive control to simulate trajectories and their resulting predicted intrinsic rewards, but it does not consider the sample importance while updating the policy.

Original:  C URIOSITY -D RIVEN U NSUPERVISED D ATA C OLLECTION (CUDC) 3. P ROBLEM S ETTING
Revised:  C URIOSITY -D RIVEN U NSUPERVISED D ATA C OLLECTION (CUDC) 3. P ROBLEM S ETTING

Original: We consider the problem of multi-task offline learning with three main steps of data collection, reward relabeling, and downstream offline learning as described in both ExORL(Yarats et al., 2022) and Explore2Offline (Lambert et al., 2022).
Revised: We consider the problem of multi-task offline learning with three main steps of data collection, reward relabeling, and downstream offline learning as described in both ExORL(Yarats et al., 2022) and Explore2Offline (Lambert et al., 2022).

Original: In the data collection phase, the exploratory agent (data collector) has the access to a Markov Decision Process (MDP) environment with a state of the environment s ∈ S , an action a ∈ A based on a policy π ( s ) , a transition probability p ( s ′ | s, a ) mapping from the current state s and action a to the next state s ′ , a reward r , and a discount factor γ ∈ [0 , 1) weighting future rewards.
Revised: In the data collection phase, the exploratory agent (data collector) has the access to a Markov Decision Process (MDP) environment with a state of the environment s ∈ S , an action a ∈ A based on a policy π ( s ) , a transition probability p ( s ′ | s, a ) mapping from the current state s and action a to the next state s ′ , a reward r , and a discount factor γ ∈ [0 , 1) weighting future rewards.

Original: In particular, the exploratory agent online interacts with the environment and stores the unlabeled tuples ( s, a, s ′ ) in the dataset D .
Revised: In particular, the exploratory agent online interacts with the environment and stores the unlabeled tuples ( s, a, s ′ ) in the dataset D .

Original: The second phase is to relabel the collected dataset D using the given reward function about the downstream task r ( s, a ) for each tuple.
Revised: The second phase is to relabel the collected dataset D using the given reward function about the downstream task r ( s, a ) for each tuple.

Original: It transfers information from task-agnostic exploration to downstream tasks.
Revised: It transfers information from task-agnostic exploration to downstream tasks.

Original: The last step is to perform multiple downstream tasks with an offline RL agent on the labeled dataset.
Revised: The last step is to perform multiple downstream tasks with an offline RL agent on the labeled dataset.

Original: In this paper, we focus on the most challenging part of this problem setting, which is the task-agnostic data collection and we evaluate the quality of the collected dataset in multiple downstream tasks.
Revised: In this paper, we focus on the most challenging part of this problem setting, which is the task-agnostic data collection and we evaluate the quality of the collected dataset in multiple downstream tasks.

Original: [Figure]
Revised: [Figure]

Original: 3. F RAMEWORK O VERVIEW
Revised: 3. F RAMEWORK O VERVIEW

Original: In this section, we propose a C uriosity-driven U nsupervised D ata C ollection (CUDC) method, which employs DDPG (Lillicrap et al., 2015) as the base RL algorithm for the exploratory agent.
Revised: In this section, we propose a C uriosity-driven U nsupervised D ata C ollection (CUDC) method, which employs DDPG (Lillicrap et al., 2015) as the base RL algorithm for the exploratory agent.

Original: As shown in Figure 1 (right), a novel reachability module is constructed to measure the probability of a k -step future state being reachable from the current state.
Revised: As shown in Figure 1 (right), a novel reachability module is constructed to measure the probability of a k -step future state being reachable from the current state.

Original: With this module in place, the exploratory agent is encouraged to diversely explore by a mixed intrinsic reward, and meanwhile regularize the critic-actor update to focus more on under-learned tuples.
Revised: With this module in place, the exploratory agent is encouraged to diversely explore by a mixed intrinsic reward, and meanwhile regularize the critic-actor update to focus more on under-learned tuples.

Original: Most importantly, the number of environment steps is adaptively increased to incorporate the dynamics information in  feature representation.
Revised: Most importantly, the temporal distance of k -step between current and future states is adaptively increased to incorporate the dynamics information in the learned feature representation.

Original: As a result, the exploration becomes more diverse.
Revised: As a result, the exploration and data collection become more diverse.

Original: More details are listed in Algorithm 1.
Revised: More details are listed in Algorithm 1.

Original: 3. T HE R EACHABILITY M ODULE
Revised: 3. T HE R EACHABILITY M ODULE

Original: In ExORL(Yarats et al., 2022), the existing unsupervised methods assume the limited feature space by implicitly fixing the environment step k between current and future states.
Revised: In ExORL(Yarats et al., 2022), the existing unsupervised methods assume the limited feature space by implicitly fixing the temporal distance k between current and future states to train the model.

Original: To diversity the feature representation, a natural choice is to employ the reachability analysis that can adjustthe environment step k during exploration.
Revised: To diversity the feature representation, a natural choice is to employ the reachability analysis that can adjust k during exploration.

Original: However, the existing implementations of reachability are not desired due to the limited capability by binary classification of reachable states (Savinov et al., 2019), or costly density estimation of goal space (P ´ er ´ e et al., 2018).
Revised: However, the existing implementations of reachability are not desired due to the limited capability by binary classification of reachable states (Savinov et al., 2019), or costly density estimation of goal space (P ´ er ´ e et al., 2018).

Original: As these approaches are not practical in the offline RL setting, we define the reachability in CUDC by the self-supervised estimation on the probability of a k -step future state s t i + k being reachable from the current state, with no expensive density estimation or manual labelling required.s t i .
Revised: As these approaches are not practical in the offline RL setting, we define the reachability in CUDC by the self-supervised estimation on the probability of a k -step future state s t i + k being reachable from the current state s t i , with no expensive density estimation or manual labelling required.

Original: As a result, the proposed method can enhance feature representation when expanding the feature space through an adaptive environment step.
Revised: As a result, the proposed method can enhance feature representation when expanding the feature space through an adaptive k -step.

Original: In fact, this motivation has been shown effective in the other works of reachability, such as constrained
Revised: In fact, this motivation has been shown effective in the other works of reachability, such as constrained

Original: RL (Yu et al., 2022) and robotics (Ivanovic et al., 2019).
Revised: RL (Yu et al., 2022) and robotics (Ivanovic et al., 2019).

Original: Given a batch of unlabeled tuples ( s t i , a t i , s t i + k , k )
Revised: Given a batch of unlabeled tuples ( s t i , a t i , s t i + k , k )

Original: ni
Revised: ni

Original: =1 , we first encode the state features z s ti = ϕ
Revised: =1 , we first encode the state features z s ti = ϕ

Original: s ( s t i ) , z s ti + k
Revised: s ( s t i ) , z s ti + k

Original: = ϕ s ( s t i + k ) and action feature z a ti =
Revised: = ϕ s ( s t i + k ) and action feature z a ti =

Original: ϕ a ( a t i ) with a state encoder ϕ s ( · ) and an action encoder ϕ a ( · ) , and perform the one-hot encoding for the environment step k .
Revised: ϕ a ( a t i ) with a state encoder ϕ s ( · ) and an action encoder ϕ a ( · ) , and perform the one-hot encoding for the temporal distance k .

Original: To facilitate the reachability analysis, a forward dynamic network ˆ z s ti + k
Revised: To facilitate the reachability analysis, a forward dynamic network ˆ z s ti + k

Original: = f s ( z s ti , z a ti , k ; θ s ) is constructed, which takes the inputs of z s ti , z a ti , and the encoded k to predict the future state feature ˆ z s ti + k , fully exploiting the information of dynamics.
Revised: = f s ( z s ti , z a ti , k ; θ s ) is constructed, which takes the inputs of z s ti , z a ti , and the encoded k to predict the future state feature ˆ z s ti + k , fully exploiting the information of dynamics.

Original: It can be updated by an l 2 norm loss of ||
Revised: It can be updated by an l 2 norm loss of ||

Original: z s ti + k −
Revised: z s ti + k −

Original: ˆ z s ti
Revised: ˆ z s ti

Original: + k
Revised: + k

Original: || 2 .
Revised: || 2 .

Original: In addition, we want to enforce ˆ z s ti + k to match with its own z s ti + k as much as possible, while keeping apart from the other future states within the same batch.
Revised: In addition, we want to enforce ˆ z s ti + k to match with its own z s ti + k as much as possible, while keeping apart from the other future states within the same batch.

Original: The intuition is that each future state should be most reachable from its own current state.
Revised: This contrastive intuition is that each future state should be most reachable from its own current state, which can quantify the reachability in a simple and efficient way.

Revised: In fact, the self-supervised contrastive learning has been recently introduced in RL and is shown to be capable of learning rich representation with more semantic latents (Srinivas et al., 2020; Liu & Abbeel, 2021b).

Original: Therefore, the reachability module can estimate the probability of s t i + k being reachable from s t i by
Revised: Following this intuition, we can estimate the probability of s t i + k being reachable from s t i by

Original: [Equation]
Revised: [Equation]

Original: (1) where n is the batch size, h ( · ) is a deterministic projection function, W is a hidden weight to compute the similarity between the two projections, and ¯ h ( · ) as well as ¯ z ( · ) are respectively the momentum-based moving average of the projection and state feature to ensure consistency and stability (He et al., 2020).
Revised: where n is the batch size, h ( · ) is a deterministic projection function, W is a hidden weight to compute the similarity between the two projections, and ¯ h ( · ) as well as ¯ z ( · ) are respectively the momentum-based moving average of the projection and state feature to ensure consistency and stability (He et al., 2020).

Original: In this way, this reachability network can be updated by the  loss of L reach = − (cid:80) ni
Revised: In this way, this reachability network can be updated by the contrastive loss of L reach = − (cid:80) ni

Original: =1 log l i in a self-supervised way without maunal labeling.
Revised: =1 log l i in a self-supervised way without maunal labeling.

Original: To further improve the representation learning, the reachability module also includes two inverse models for predicting action feature ˆ z a ti and environment step ˆ k .
Revised: To further improve the representation learning, the reachability module also includes two inverse models for predicting action feature ˆ z a ti and temporal distance ˆ k .

Original: Similar to ICM
Revised: Similar to ICM

Original: (Pathak et al., 2017) and Disagreement (Pathak et al., 2019), we define ˆ z a ti =
Revised: (Pathak et al., 2017) and Disagreement (Pathak et al., 2019), we define ˆ z a ti =

Original: f a ( z s ti ,
Revised: f a ( z s ti ,

Original: z s ti + k , k ; θ a ) with a backward loss of ||
Revised: z s ti + k , k ; θ a ) with a backward loss of ||

Original: z a ti − ˆ z
Revised: z a ti − ˆ z

Original: a ti
Revised: a ti

Original: || 2 , which ensures the encoded features are robust to environment variations that are uncontrollable by the agent.
Revised: || 2 , which ensures the encoded features are robust to environment variations that are uncontrollable by the agent.

Original: For the inverse model of the environment step, ˆ k = f k
Revised: For the inverse model of the k -step, ˆ k = f k

Original: ( z s ti ,
Revised: ( z s ti ,

Original: z s ti + k ; θ k ) characterizes the prediction with a distribution P ( k ) .
Revised: z s ti + k ; θ k ) characterizes the prediction with a distribution P ( k ) .

Original: It is updated through a cross-entropy loss, which enables the encoders to capture the dynamics information in the encoded features.
Revised: It is updated through a cross-entropy loss, which enables the encoders to capture the dynamics information in the encoded features.

Original: The agent can therefore update its internal belief in a self-supervised manner without any expensive labeling process compared to supervised learning.
Revised: The agent can therefore update its internal belief in a self-supervised manner without any expensive labeling process compared to supervised learning.

Original: More importantly, we can allow the environment step to adapt during learning, rather than relying on a fixed hyperparameter in many existing unsupervised methods.
Revised: More importantly, we can allow the temporal distance of k -step to adapt during learning, rather than relying on a fixed hyperparameter in many existing unsupervised methods.

Original: It is of vital importance as the feature representations of both states and actions become more informative and robust when adjusting the environment step.
Revised: It is of vital importance as the feature representations of both states and actions become more informative and robust when adjusting the temporal distance of k -step.

Original: Meanwhile, this module is also capable of computing a curiosity weight w i for each tuple i as
Revised: Meanwhile, this module is also capable of computing a curiosity weight w i for each tuple i as

Original: [Equation]
Revised: [Equation]

Original: Intuitively, a large value of w i means that the agent mistakenly believes the true future state is unreachable from the current state, which induces high curiosity due to the conflict with agent’s current internal belief.
Revised: Intuitively, a large value of w i means that the agent mistakenly believes the true future state is unreachable from the current state, which induces high curiosity due to the conflict with agent’s current internal belief.

Original: It further indicates that this under-learned transition tuple contains novel information, and the encoders are not capable of extracting meaningful features yet.
Revised: It further indicates that this under-learned transition tuple contains novel information, and the encoders are not capable of extracting meaningful features yet.

Original: With this reachability module in place, we can enable the agent to perform the task-agnostic dataset collection in a curious manner.
Revised: With this reachability module in place, we can enable the agent to perform the task-agnostic dataset collection in a curious manner.

Original: Detailed curiosity-driven learning is illustrated in the next subsection.
Revised: Detailed curiosity-driven learning is illustrated in the next subsection.

Original: 3. C URIOSITY -D RIVEN L EARNING
Revised: 3. C URIOSITY -D RIVEN L EARNING

Original: It should be noted that reachability in previous works is only incorporated as an intrinsic reward to encourage diverse exploration.
Revised: It should be noted that reachability in previous works is only incorporated as an intrinsic reward to encourage diverse exploration.

Original: In contrast, our proposed CUDC extends the reachability to more stages of learning.
Revised: In contrast, our proposed CUDC extends the reachability to more stages of learning.

Original: Firstly, it adapts the number of environment steps to explore, enhancing feature representation with the dynamics information of more distant future states.
Revised: Firstly, it adapts the temporal distance, i.e. k -step to explore, enhancing feature representation with the dynamics information of more distant future states.

Original: Secondly, it incorporates a mixed intrinsic reward to encourage effective exploration with the expanded feature space.
Revised: Secondly, it incorporates a mixed intrinsic reward to encourage effective exploration with the expanded feature space.

Original: Lastly, it regularizes the critic-actor update for the DDPG algorithm by utilizing the curiosity weights to focus more on under-learned tuples.
Revised: Lastly, it regularizes the critic-actor update for the DDPG algorithm by utilizing the curiosity weights to focus more on under-learned tuples.

Original: Different from the 8 existing methods evaluated in ExORL that only utilize intrinsic rewards as curiosity, our CUDC extends curiosity-driven learning to different
Revised: Different from the 8 existing methods evaluated in ExORL that only utilize intrinsic rewards as curiosity, our CUDC extends curiosity-driven learning to different

Original: RL components, improving the data collection process coherently.
Revised: RL components, improving the data collection process coherently.

Original: 3.4. E NHANCE F EATURE R EPRESENTATION WITH A DAPTIVE E NVIRONMENT S TEPS
Revised: 3.4. E NHANCE F EATURE R EPRESENTATION WITH A DAPTIVE k -S TEP

Original: It is worth noting that all 8 methods discussed in ExORL limit the autonomy of feature space by expecting the agent to reach the future states in exactly 3 steps.
Revised: It is worth noting that all 8 methods discussed in ExORL limit the autonomy of feature space by expecting the agent to reach the future states in exactly 3 steps.

Original: Recent works on online pre-training
Revised: Recent works on online pre-training

Original: RL such as SPR (Schwarzer et al., 2020) and SGI (Schwarzer et al., 2021) predict agent’s own latent state representations multiple steps into the future, addressing the challenge of sample efficiency.
Revised: RL such as SPR (Schwarzer et al., 2020) and SGI (Schwarzer et al., 2021) predict agent’s own latent state representations multiple steps into the future, addressing the challenge of sample efficiency.

Original: However, this approach requires iterative predictions by calling the forward dynamic network for k times.
Revised: However, this approach requires iterative predictions by calling the forward dynamic network for k times.

Original: In contrast, CUDC can automatically adjust the environment step k to directly perform the k -step future state estimation in the proposed reachability module, without increasing the computational complexity substantially.
Revised: In contrast, CUDC can automatically adjust the temporal distance k to directly perform the k -step future state estimation in the proposed reachability module, without increasing the computational complexity substantially.

Original: To expand the feature space by leveraging the dynamics information, our main idea is to keep the reachability estimation increasingly challenging with an adaptive environment step.
Revised: To expand the feature space by leveraging the dynamics information, our main idea is to keep the reachability estimation increasingly challenging with an adaptive k -step.

Original: In particular, we adaptively adjust thenumber of enviroment steps k by encouraging curiosity, i.e., k is increased by 1 if the agent is not curious enough for the current reachability analysis.
Revised: In particular, we adaptively adjust the k by encouraging curiosity, i.e., k is increased by 1 if the agent is not curious enough for the current reachability analysis.

Original: For this purpose, we set a threshold C w for low curiosity and a threshold C k for the proportion of tuples with low curiosity.
Revised: For this purpose, we set a threshold C w for low curiosity and a threshold C k for the proportion of tuples with low curiosity.

Original: Thus, the agent will adapt k when (cid:80) ni 1 [w i < C w ] n > C k .
Revised: Thus, the agent will adapt k when (cid:80) ni 1 [w i < C w ] n > C k .

Original: The intuition is that the agent should be encouraged to explore further based on the reachability, when they are no longer curious about the majority of tuples within the same batch.
Revised: The intuition is that the agent should be encouraged to explore further based on the reachability, when they are no longer curious about the majority of tuples within the same batch.

Original: As a result, the feature representation can be enhanced by learning the dynamics of more distant future states.
Revised: As a result, the feature representation can be enhanced by learning the dynamics of more distant future states.

Original: With the expanded feature space, the data collection is performed towards more informative and diverse features.
Revised: With the expanded feature space, the data collection is performed towards more informative and diverse features.

Original: It should be noted that there are many possible ways to vary the environment step such as  sampling from a probabilistic distribution.
Revised: It should be noted that there are many possible ways to vary the k -step such as by sampling from a probabilistic distribution.

Original: For this reason, an ablation study is also conducted to validate the effectiveness of the proposed curiosity-driven method, compared to other sampling-based methods.
Revised: For this reason, an ablation study is also conducted to validate the effectiveness of the proposed curiositydriven method, compared to other sampling-based methods.

Original: 3.4. I NCORPORATE A M IXED I NTRINSIC R EWARD
Revised: 3.4. I NCORPORATE A M IXED I NTRINSIC R EWARD

Original: CUDC employs a mixed intrinsic reward of state-action entropy and prediction error of future states.
Revised: CUDC employs a mixed intrinsic reward of state-action entropy and prediction error of future states.

Original: On one hand, the existing methods of APT (Liu & Abbeel, 2021b) and RE3 (Seo et al., 2021) have shown that agents can be encouraged to explore the state space more uniformly, by utilizing the particle-based k-nearest neighbors state entropy as an intrinsic reward.
Revised: On one hand, the existing methods of APT (Liu & Abbeel, 2021b) and RE3 (Seo et al., 2021) have shown that agents can be encouraged to explore the state space more uniformly, by utilizing the particle-based k-nearest neighbors state entropy as an intrinsic reward.

Original: This is consistent with the entropy maximization principle (Beirlant et al., 1997; Singh et al., 2003).
Revised: This is consistent with the entropy maximization principle (Beirlant et al., 1997; Singh et al., 2003).

Original: However, we believe that the agent should diversely explore not only the state space, but also the action space.
Revised: However, we believe that the agent should diversely explore not only the state space, but also the action space.

Original: Therefore,
Revised: Therefore,

Original: CUDC extends state embedding to state-action embedding and we prove that the k-nearest neighbor entropy estimation in the state-action representation space can be used for entropy maximization.

Original: Algorithm 1 Implementation of the proposed CUDC
Revised: Algorithm 1 Implementation of the proposed CUDC

Original: Initialize parameters of encoders ϕ s and ϕ a , forward dynamic network f s , inverse models f a and f k , projection h , critic Q , policy π , hidden weight W , and the environment step k
Revised: Initialize parameters of encoders ϕ s and ϕ a , forward dynamic network f s , inverse models f a and f k , projection h , critic Q , policy π , hidden weight W , and the temporal distance k

Original: Initialize the task-agnostic dataset D = ∅ for each timestep t do // COLLECT TRANSITIONS
Revised: Initialize the task-agnostic dataset D = ∅ for each time step t do // COLLECT TRANSITIONS

Original: Interact with the environment using the policy a t ∼ π ( s t ) and observe s t +
Revised: Interact with the environment using the policy a t ∼ π ( s t ) and observe s t +

Original: D ∪ ( s t , a t , s t +1 )
Revised: D ∪ ( s t , a t , s t +1 )

Original: → D // UPDATE INTERNAL BELIEF OF REACHABILITY
Revised: → D // UPDATE INTERNAL BELIEF OF REACHABILITY

Original: Sample a minibatch { ( s t i , a t i , s t
Revised: Sample a minibatch { ( s t i , a t i , s t

Original: i + k , k ) } ni =1 ∼ D for each tuple i in the minibatch do
Revised: i + k , k ) } ni =1 ∼ D for each tuple i in the minibatch do

Original: Encode the state and action, and predict the t i + k ’s future state feature ˆ z s ti + k
Revised: Encode the state and action, and predict the t i + k ’s future state feature ˆ z s ti + k

Original: Evaluate the curiosity weight w i according to Equation (2)
Revised: Evaluate the curiosity weight w i according to Equation (2)

Original: Compute the intrinsic reward r i using w i by Equation (3) end for
Revised: Compute the intrinsic reward r i using w i by Equation (3) end for

Original: Update the internal belief of reachability module //ADAPT
Revised: Update the internal belief of reachability module //ADAPT

Original: THE ENVIRONMENT STEP TO PREDICT if 1 n (cid:80) n i 1 [w i < C w ] > C k then
Revised: THE K-STEP TO PREDICT if 1 n (cid:80) n i 1 [w i < C w ] > C k then

Original: Increase the environment step k = k +end if //REGULARIZE CRITIC-ACTOR UPDATE
Revised: Increase the temporal distance by k = k +end if //REGULARIZE CRITIC-ACTOR UPDATE

Original: Update the critic Q with regularization by Equation (4)
Revised: Update the critic Q with regularization by Equation (4)

Original: Update the actor π with regularization
Revised: Update the actor π with regularization

Original: Perform the momentum update for ¯ h and ¯ z end for
Revised: Perform the momentum update for ¯ h and ¯ z end for

Revised: CUDC extends state embedding to state-action embedding and we prove that the k-nearest neighbor entropy estimation in the state-action representation space can be used for entropy maximization.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Let u
Revised: Let u

Original: = ( z s , z a ) represent the state-action representation.
Revised: = ( z s , z a ) represent the state-action representation.

Original: The particle-based entropy
Revised: The particle-based entropy

Original: H ( u ) is proportional to a K-nearest neighbor (K-NN) distance,
Revised: H ( u ) is proportional to a K-nearest neighbor (K-NN) distance,

Original: H ( u ) ∝ n (cid:88)
Revised: H ( u ) ∝ n (cid:88)

Original: i = log ||
Revised: i = log ||

Original: u
Revised: u

Original: i − u K-NN
Revised: i − u K-NN

Original: i || 2 .
Revised: i || 2 .

Original: Following the same idea to treat each tuple as a particle (Liu & Abbeel, 2021b; Seo et al., 2021), we formulate an intrinsic reward to estimate the particle-based entropy, r H ( s t
Revised: Following the same idea to treat each tuple as a particle (Liu & Abbeel, 2021b; Seo et al., 2021), we formulate an intrinsic reward to estimate the particle-based entropy, r H ( s t

Original: i , a t i ) = log( 1 N K (cid:80) ||
Revised: i , a t i ) = log( 1 N K (cid:80) ||

Original: u i − u K-NN
Revised: u i − u K-NN

Original: i || 2 + 1) where u i =
Revised: i || 2 + 1) where u i =

Original: ( ϕ s ( s t i ) , ( ϕ a ( a t i )) and N K is the number of K-NN.
Revised: ( ϕ s ( s t i ) , ( ϕ a ( a t i )) and N K is the number of K-NN.

Original: As the encoded features are constantly updated to capture the dynamics of more distant future states in the reachability module, the proposed r H can lead to a more diverse state-action space exploration, where the state-of-the-art off-policy RL algorithm SAC (Haarnoja et al., 2018) has also shown that exploration through a discounted maximum-policy (actor)-entropy term is more effective.
Revised: As the encoded features are constantly updated to capture the dynamics of more distant future states in the reachability module, the proposed r H can lead to a more diverse state-action space exploration, where the state-of-the-art off-policy RL algorithm SAC (Haarnoja et al., 2018) has also shown that exploration through a discounted maximum-policy (actor)-entropy term is more effective.

Original: On the other hand, the prediction errors of future states are often employed as the intrinsic reward to incentivize the agent to explore surprising states that are beyond their exceptions (Pathak et al., 2017; Burda et al., 2018).
Revised: On the other hand, the prediction errors of future states are often employed as the intrinsic reward to incentivize the agent to explore surprising states that are beyond their exceptions (Pathak et al., 2017; Burda et al., 2018).

Original: CUDC additionally integrates the prediction error r E ( s t i , a t i ) =
Revised: CUDC additionally integrates the prediction error r E ( s t i , a t i ) =

Original: ||
Revised: ||

Original: z s ti +
Revised: z s ti +

Original: k
Revised: k

Original: −
Revised: −

Original: ˆ z s ti + k
Revised: ˆ z s ti + k

Original: || 2 in the intrinsic reward as well, where the reachability module is conveniently re-used without additional networks.
Revised: || 2 in the intrinsic reward as well, where the reachability module is conveniently re-used without additional networks.

Original: Finally, the mixed intrinsic reward in CUDC is given by
Revised: Finally, the mixed intrinsic reward in CUDC is given by

Original: [Equation]
Revised: [Equation]

Original: where α controls the prioritization of under-learned state exploration and β is a constant for numerical stability.
Revised: where α controls the prioritization of under-learned state exploration and β is a constant for numerical stability.

Original: 3.4. R EGULARIZE THE CRITIC - ACTOR UPDATE
Revised: 3.4. R EGULARIZE THE CRITIC - ACTOR UPDATE

Original: Furthermore, CUDC adaptively regularizes the backbone DDPG algorithm to focus more on underlearned tuples, by utilizing the curiosity weight w i .
Revised: Furthermore, CUDC adaptively regularizes the backbone DDPG algorithm to focus more on underlearned tuples, by utilizing the curiosity weight w i .

Original: In particular, w = ( w 1 , w 2 , · · · , w n ) quantitatively describes the curiosity weight on each transition tuple, which can be seamlessly used to characterize the sample importance and regularize both critic and actor updates.
Revised: In particular, w = ( w 1 , w 2 , · · · , w n ) quantitatively describes the curiosity weight on each transition tuple, which can be seamlessly used to characterize the sample importance and regularize both critic and actor updates.

Original: Therefore, we can perform the Q-learning in DDPG by minimizing the following objective,
Revised: Therefore, we can perform the Q-learning in DDPG by minimizing the following objective,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the policy can be updated by maximizing E ( s,a,s ′ )
Revised: Meanwhile, the policy can be updated by maximizing E ( s,a,s ′ )

Original: ∼D [ wQ ( s, π ( s ))] .
Revised: ∼D [ wQ ( s, π ( s ))] .

Original: In this way, CUDC facilitates the agent to adapt the learning process in a self-supervised manner in the sense that the regularization is controlled by the conceptualized curiosity to exploit sample importance.
Revised: In this way, CUDC facilitates the agent to adapt the learning process in a self-supervised manner in the sense that the regularization is controlled by the conceptualized curiosity to exploit sample importance.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Environments We evaluate the proposed CUDC on continuous control tasks with state observations in the DeepMind control suite (Tassa et al., 2018), which consists of 12 downstream tasks across 3 main challenging domains: Walker, Quadruped, and Jaco Arm.
Revised: Environments We evaluate the proposed CUDC on continuous control tasks with state observations in the DeepMind control suite (Tassa et al., 2018), which consists of 12 downstream tasks across 3 main challenging domains: Walker, Quadruped, and Jaco Arm.

Original: Walker is a controllable entity with locomotion-related balancing controls, where it can learn walking, running, flipping and standing.
Revised: Walker is a controllable entity with locomotion-related balancing controls, where it can learn walking, running, flipping and standing.

Original: Quadruped is a passively stable body in a more challenging 3D environment to learn various locomotion skills of walking, running, standing, and jumping.
Revised: Quadruped is a passively stable body in a more challenging 3D environment to learn various locomotion skills of walking, running, standing, and jumping.

Original: Jaco Arm is 6 degree-of-freedom robotic arm with a three-finger gripper for object manipulation, where the downstream tasks require it to reach different positions.
Revised: Jaco Arm is 6 degree-of-freedom robotic arm with a three-finger gripper for object manipulation, where the downstream tasks require it to reach different positions.

Original: Note that PointMass Maze is not included since most baseline methods in ExORL have already demonstrated excellent performances.
Revised: Note that PointMass Maze is not included since most baseline methods in ExORL have already demonstrated excellent performances.

Original: Baseline Models We compare CUDC against state-of-the-art unsupervised methods across all three categories as benchmarked in ExORL, i.e., a knowledge-driven baseline of ICM (Pathak et al., 2017), data-driven baselines of APT (Liu & Abbeel, 2021b) and ProtoRL (Yarats et al., 2021), and a competence-driven baseline of APS (Liu & Abbeel, 2021a).
Revised: Baseline Models We compare CUDC against state-of-the-art unsupervised methods across all three categories as benchmarked in ExORL, i.e., a knowledge-driven baseline of ICM (Pathak et al., 2017), data-driven baselines of APT (Liu & Abbeel, 2021b) and ProtoRL (Yarats et al., 2021), and a competence-driven baseline of APS (Liu & Abbeel, 2021a).

Original: Meanwhile, a random data collector is also included, which collects the data by performing randomly sampled actions.
Revised: Meanwhile, a random data collector is also included, which collects the data by performing randomly sampled actions.

Original: The other four methods discussed in ExORL are excluded since their performance are less competitive.
Revised: The other four methods discussed in ExORL are excluded since their performance are less competitive.

Original: We set the same hyperparameters and model architecture as reported in ExORL.
Revised: We set the same hyperparameters and model architecture as reported in ExORL.

Original: To demonstrate that all proposed components play important roles in the performance, we also compare four versions of
Revised: To demonstrate that all proposed components play important roles in the performance, we also compare four versions of

Original: CUDC as follows.
Revised: CUDC as follows.

Original: CUDC ICMvary and CUDC APTvary : adapting the environment step k  by the intrinsic rewards based on the original ICM and APT methods.
Revised: CUDC ICMvary and CUDC APTvary : adapting the temporal distance of k -step by the intrinsic rewards based on the original ICM and APT methods.

Original: CUDC reward : extending to state-action entropy with a mixed intrinsic reward based on CUDC APTvary .
Revised: CUDC reward : extending to state-action entropy with a mixed intrinsic reward based on CUDC APTvary .

Original: CUDC reach : adding the full reachability module without regularization based on CUDC reward .
Revised: CUDC reach : adding the full reachability module without regularization based on CUDC reward .

Original: The detailed implementation and differences from the full model are summarized in Appendix A.
Revised: The detailed implementation and differences from the full model are summarized in Appendix A.

Original: Model Training and Evaluation To impose model stability during learning, we restrict the environment step k to be increased from 3 to 6 and set the upper and lower bounds for the regularization weights to ensure stability.
Revised: Model Training and Evaluation To impose model stability during learning, we restrict the kstep to be increased from 3 to 6 and set the upper and lower bounds for the regularization weights to ensure stability.

Original: For further details regarding the network implementation and hyperparameter setting of the proposed CUDC, readers can refer to Appendix A. During data collection, all methods are trained using a DDPG (Lillicrap et al., 2015) agent as the backbone for fairness.
Revised: For further details regarding the network implementation and hyperparameter setting of the proposed CUDC, readers can refer to Appendix A. During data collection, all methods are trained using a DDPG (Lillicrap et al., 2015) agent as the backbone for fairness.

Original: They interact with 3 domain environments in the absence of extrinsic rewards for 1M steps.
Revised: They interact with 3 domain environments in the absence of extrinsic rewards for 1M steps.

Original: For the main results, a total of 90 datasets (6 algorithms × 3 main tasks × 5 seeds) are collected.
Revised: For the main results, a total of 90 datasets (6 algorithms × 3 main tasks × 5 seeds) are collected.

Original: After that, relabeling is performed for each downstream task on the collected dataset.
Revised: After that, relabeling is performed for each downstream task on the collected dataset.

Original: During the evaluation, a TD (Fujimoto et al., 2018) agent learns offline from each relabeled dataset for 500K steps.
Revised: During the evaluation, a TD (Fujimoto et al., 2018) agent learns offline from each relabeled dataset for 500K steps.

Original: We report the performance score at 100K step for sample efficiency and at 500K step for learning performance.
Revised: We report the performance score at 100K step for sample efficiency and at 500K step for learning performance.

Original: Main Results on 12 Downstream Tasks As shown in Figure 2, no single baseline method can greatly improve the dataset quality for all domains.
Revised: Main Results on 12 Downstream Tasks As shown in Figure 2, no single baseline method can greatly improve the dataset quality for all domains.

Original: In contrast, the dataset collected by CUDC is capable of boosting the offline agent’s learning performances at 500K step in all 12 downstream tasks across 3 challenging domains, as highlighted in Table 1.
Revised: In contrast, the dataset collected by CUDC is capable of boosting the offline agent’s learning performances at 500K step in all 12 downstream tasks across 3 challenging domains, as highlighted in Table 1.

Original: Specifically, CUDC outperforms the competence-based method (APS) in the Walker domain by 6%, outperforms the data-based method (APT) in the Quadruped domain by 51%, and outperforms the knowledge-based method (ICM) in the Jaco Arm domain by 10%.
Revised: Specifically, CUDC outperforms the competence-based method (APS) in the Walker domain by 6%, outperforms the data-based method (APT) in the Quadruped domain by 51%, and outperforms the knowledge-based method (ICM) in the Jaco Arm domain by 10%.

Original: As for the sample efficiency, it can be observed in Figure 2 that CUDC leads to the best sample efficiency with a significant margin on 3 downstream tasks of Quadruped.
Revised: As for the sample efficiency, it can be observed in Figure 2 that CUDC leads to the best sample efficiency with a significant margin on 3 downstream tasks of Quadruped.

Original: In the easy domain of Walker, CUDC can help the offline agent to converge faster in 3 downstream tasks.
Revised: In the easy domain of Walker, CUDC can help the offline agent to converge faster in 3 downstream tasks.

Original: However, the sample efficiency in the hardest domain of Jaco Arm is not desirable.
Revised: However, the sample efficiency in the hardest domain of Jaco Arm is not desirable.

Original: It might be caused by too much complexity in this most challenging environment, increasing the difficulty of reachablility analysis.
Revised: It might be caused by too much complexity in this most challenging environment, increasing the difficulty of reachablility analysis.

Original: Effects of Adapting the k -Step We empirically show that adapting the environment steps to explore more distant future states can certainly help to enhance the feature representation, and thereby improve the data collection process. By comparing the results  in Figure 3, CUDC ICM vary has outper-
Revised: More results are included in Appendix C.1 and evaluation using another offline RL algorithm of CQL (Kumar et al., 2020) is conducted in Appendix C.2.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Walker, Flip
Revised: Walker, Flip

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: , Run
Revised: Walker, Run

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: Walker, Stand

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Walk
Revised: Walk

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: A v e r age E v a l ua t i on S c
Revised: A v e r age E v a l ua t i on S c

Original: o r e s
Revised: o r e s

Original: [Figure]
Revised: [Figure]

Revised: Effects of Adapting the k -Step We empirically show that adapting the temporal distance to explore more distant future states can certainly help to enhance the feature representation

Original: formed ICM significantly, with on average 1.25 × sample efficiency at 100K step and 1.16 × offline learning performance at 500K step.
Revised: [Figure] and thereby improve the data collection process. By comparing the results in Figure 3, CUDC ICM vary has outperformed ICM significantly, with on average 1.25 × sample efficiency at 100K step and 1.16 × offline learning performance at 500K step.

Original: Similarly, CUDC APTvary obtains respectively 1.12 × and 1.04 × scores at 100K and 500K steps across 4 downstream tasks, compared with APT.
Revised: Similarly, CUDC APTvary obtains respectively 1.12 × and 1.04 × scores at 100K and 500K steps across 4 downstream tasks, compared with APT.

Original: We note that the standard deviation increases a bit, which might be caused by the introduced complexity of more distant future state when improving the learned representation.
Revised: We note that the standard deviation increases a bit, which might be caused by the introduced complexity of more distant future state when improving the learned representation.

Original: Thus, it is desirable to find an adaptive way to smooth this process, e.g., by incorporating the other proposed components inherently.
Revised: Thus, it is desirable to find an adaptive way to smooth this process, e.g., by incorporating the other proposed components inherently.

Original:  Effectiveness of the Other Proposed Components We additionally incorporate the mixed intrinsic reward to CUDC APTvary as CUDC reward , and it further improves both sample-efficiency and learning capability of the offline RL agent, shown in Table 4 and Figure 3.
Revised: [Figure] Effectiveness of the Other Proposed Components We additionally incorporate the mixed intrinsic reward to CUDC APTvary as CUDC reward , and it further improves both sample-efficiency and learning capability of the offline RL agent, shown in Table 4 and Figure 3.

Original: As the new intrinsic reward en-
Revised: As the new intrinsic reward encourages a more uniform exploration on the state-action space and focuses more on under learned states, the performance at 100K is inevitably unstable, with a 67% increase of standard deviation

Original: [Figure]
Revised: [Figure]

Original: courages a more uniform exploration on the state-action space and focuses more on under learned states, the performance at 100K is inevitably unstable, with a 67% increase of standard deviation.
Revised: Therefore [Figure]

Original: Therefore, it is necessary to leverage the proposed reachability module as agent’s internal belief and facilitate the data collection.
Revised: it is necessary to leverage the proposed reachability module as agent’s internal belief and facilitate the data collection.

Original: By comparing CUDC reach with CUDC reward , the dataset collected by
Revised: By comparing CUDC reach with CUDC reward , the dataset collected by

Original: CUDC reach stabilizes the offline learning process by reducing 48% and 25% of standard deviation at 100K and 500K steps.
Revised: CUDC reach stabilizes the offline learning process by reducing 48% and 25% of standard deviation at 100K and 500K steps.

Original: However, its performance scores slightly decrease in 2 tasks.
Revised: However, its performance scores slightly decrease in 2 tasks.

Original: To address this problem, the full model further regularizes the critic-actor update with a curiosity weight, compared with CUDC reach .
Revised: To address this problem, the full model further regularizes the critic-actor update with a curiosity weight, compared with CUDC reach .

Original: It achieves the best performances with minimum standard deviation at 500K step.
Revised: It achieves the best performances with minimum standard deviation at 500K step.

Revised: More experiments are conducted in Appendix C.4 by removing each proposed components.

Original: It can be concluded that all components play important roles to collect a useful dataset for improving the downstream task learning and resolving instability on offline RL agents
Revised: It can be concluded that all components play important roles to collect a useful dataset for improving the downstream task learning and resolving instability on offline RL agents

Original: [Figure]
Revised: [Figure]

Original: Adjusting the Environment Step in Different Ways One may wonder how the feature representation improves with different ways of adjusting the environment step k in the reachability module.
Revised: Adjusting the k -Step in Different Ways One may wonder how the feature representation improves with different ways of adjusting the temporal distance k in the reachability module.

Original: Thus, we explicitly conduct an ablation study on the Walker domain by sampling k uniformly (Uniform) from 3 to 6 and normally (Normal) with an increasing mean.
Revised: Thus, we explicitly conduct an ablation study on the Walker domain by sampling k uniformly (Uniform) from 3 to 6 and normally (Normal) with an increasing mean.

Original: It can be observed in Figurethat Uniform performs the worst in all 4 tasks as it cannot adapt the environment step to enhance the representation learning.
Revised: It can be observed in Figure 4 that Uniform performs the worst in all 4 tasks as it cannot adapt the temporal distance in a curious way to enhance the representation learning.

Original: At 500K step, it only achieves 85% overall learning capability with 300% increase in the standard deviation, compared to CUDC.
Revised: At 500K step, it only achieves 85% overall learning capability with 300% increase in the standard deviation, compared to CUDC.

Original: Normal to some extent adapts the environment step through an increasing mean, and it even outperforms CUDC in the Flip task.
Revised: Normal to some extent adapts k through an increasing mean, and it even outperforms CUDC in the Flip task.

Original: However, its overall performance is still 4.5% weaker than CUDC and its standard deviation is 128% higher than
Revised: However, its overall performance is still 4.5% weaker than CUDC and its standard deviation is 128% higher than

Original: CUDC, indicating the instability issue.
Revised: CUDC, indicating the instability issue.

Original: Overall, the curious way proposed in CUDC is the best and it is interesting to investigate more adaptive ways in the future.
Revised: Overall, the curious adaptation method proposed in CUDC is the best and it is interesting to investigate more adaptive ways in the future.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose CUDC, a curiosity-driven unsupervised data collection method to improve the dataset quality for offline RL agents in the multi-task setting.
Revised: We propose CUDC, a curiosity-driven unsupervised data collection method to improve the dataset quality for offline RL agents in the multi-task setting.

Original: A reachability module is introduced, quantifying agent’s internal belief of estimating the probability of a k -step future state being reachable form the current state.
Revised: A reachability module is introduced, quantifying agent’s internal belief of estimating the probability of a k -step future state being reachable form the current state.

Original: Substantially, CUDC can enhance feature representation by adaptively allowing the agent to explore more distant future states in the reachability module.
Revised: Substantially, CUDC can enhance feature representation by adaptively allowing the agent to explore more distant future states in the reachability module.

Original: Empirically, CUDC demonstrates superior data collection capability with improved sample efficiency and better performances in downstream multi-task offline learning.
Revised: Empirically, CUDC demonstrates superior data collection capability with improved sample efficiency and better performances in downstream multi-task offline learning.

Original: We also present this work with insightful empirical evidence of effective data collection methods for future research.
Revised: We also present this work with insightful empirical evidence of effective data collection methods for future research.

Original: A CUDC I MPLEMENTATION AND H YPERPARAMETER S ETTING A. I MPLEMENTATION D ETAILS A.1. CUDC
Revised: A CUDC I MPLEMENTATION AND H YPERPARAMETER S ETTING A. I MPLEMENTATION D ETAILS A.1. CUDC

Original: In the reachability module of CUDC shown in Figure 1, the state encoder ϕ
Revised: In the reachability module of CUDC shown in Figure 1, the state encoder ϕ

Original: s is a 1-layer MLP with the ReLU activation.
Revised: s is a 1-layer MLP with the ReLU activation.

Original: Subsequently, the output is passed to a single normalized fully-connected layer by LayerNorm (Lei Ba et al., 2016) with the tanh nonlinearity applied at the end.
Revised: Subsequently, the output is passed to a single normalized fully-connected layer by LayerNorm (Lei Ba et al., 2016) with the tanh nonlinearity applied at the end.

Original: The target encoder is momentum updated from the state encoder to obtain ¯ z .
Revised: The target encoder is momentum updated from the state encoder to obtain ¯ z .

Original: The action encoder ϕ a is a 3-layer MLP with the ReLU activation.
Revised: The action encoder ϕ a is a 3-layer MLP with the ReLU activation.

Original: For the forward network f s and the backward networks f a and f k , they are 2-layer MLP with the ReLU activation.
Revised: For the forward network f s and the backward networks f a and f k , they are 2-layer MLP with the ReLU activation.

Original: The projection network h is a 2-layer MLP with hidden size of 128 and output size of 64, followed by LayerNorm.
Revised: The projection network h is a 2-layer MLP with hidden size of 128 and output size of 64, followed by LayerNorm.

Original: ¯ h is also momentum updated from h .
Revised: ¯ h is also momentum updated from h .

Original: A.1. CUDC ICM VARY AND CUDC APT VARY
Revised: A.1. CUDC ICM VARY AND CUDC APT VARY

Original: CUDC ICMvary and CUDC APTvary are two variants of the proposed CUDC.
Revised: CUDC ICMvary and CUDC APTvary are two variants of the proposed CUDC.

Original: Given the implementation of ICM and APT in ExORL (Yarats et al., 2022), we added the forward network f s and backward network f k in order to let these models learn more feature representations with the dynamics information.
Revised: Given the implementation of ICM and APT in ExORL (Yarats et al., 2022), we added the forward network f s and backward network f k in order to let these models learn more feature representations with the dynamics information.

Original: Therefore, the environment step k  can be adapted in a similar way as in CUDC.
Revised: Therefore, the temporal distance of k -step can be adapted in a similar way as in CUDC.

Original: k is increased by 1 if the proportion of tuples with low intrinsic rewards is greater than a threshold, i.e. (cid:80) ni 1 [r i < C r ] n > C k .
Revised: k is increased by 1 if the proportion of tuples with low intrinsic rewards is greater than a threshold, i.e. (cid:80) ni 1 [r i < C r ] n > C k .

Original: In this way, we can investigate the effects of adapting environment step on two unsupervised baseline methods.
Revised: In this way, we can investigate the effects of adapting environment step on two unsupervised baseline methods.

Original: A.1. CUDCAPT REWARD
Revised: A.1. CUDC REWARD

Original: CUDC APTreward is a variant of the proposed CUDC.
Revised: CUDC reward is a variant of the proposed CUDC.

Original: Given the implementation of CUDC APTvary in the previous subsection, we extended the state entropy to state-action entropy and meanwhile add a prediction error of the k -step future state as formulated in Equation (3).
Revised: Given the implementation of CUDC APTvary in the previous subsection, we extended the state entropy to state-action entropy and meanwhile add a prediction error of the k -step future state as formulated in Equation (3).

Original: Therefore, agent can be encouraged to explore more diverse state-action space while focusing on the under states with high prediction errors.
Revised: Therefore, agent can be encouraged to explore more diverse state-action space while focusing on the under states with high prediction errors.

Original: In this way, we can investigate the effects of the proposed mixed intrinsic reward.
Revised: In this way, we can investigate the effects of the proposed mixed intrinsic reward

Original:  A.1. CUDCAPT REACH
Revised: . A.1. CUDC REACH

Original: CUDC APTreach is a variant of the proposed CUDC.
Revised: CUDC reach is a variant of the proposed CUDC.

Original: Given the implementation of CUDC APTreward in the previous subsection, we incorporated the full reachability module with the reachable network.
Revised: Given the implementation of CUDC reward in the previous subsection, we incorporated the full reachability module with the reachable network.

Original: As a result, the adaptive update of the environment step can be facilitated by the curiosity weight w outputted by the reachability module and the enhanced representation learning can be carried out in a self-supervised manner.
Revised: As a result, the adaptive update of the environment step can be facilitated by the curiosity weight w outputted by the reachability module and the enhanced representation learning can be carried out in a self-supervised manner.

Original: Compared with the full CUDC model, CUDC APTreach has disabled the regularization of the critic-actor update.
Revised: Compared with the full CUDC model, CUDC reach has disabled the regularization of the critic-actor update.

Original: In this way, we can investigate the effects of the proposed reachability module.
Revised: In this way, we can investigate the effects of the proposed reachability module.

Original: A. H YPERPARAMETER S ETTING A.2. D ATA C OLLECTION
Revised: A. H YPERPARAMETER S ETTING A.2. D ATA C OLLECTION

Original: We provide a full set of common hyperparameters used in baselines as welll as CUDC in Table 2, which closely follows the same settings from ExORL (Yarats et al., 2022) and URLB (Laskin et al., 2021).
Revised: We provide a full set of common hyperparameters used in baselines as welll as CUDC in Table 2, which closely follows the same settings from ExORL (Yarats et al., 2022) and URLB (Laskin et al., 2021).

Original: For the other hyperparameter used in CUDC, they are listed in Table 3.
Revised: For the other hyperparameter used in CUDC, they are listed in Table 3.

Original: It should be noted that Explore2Offline (Lambert et al., 2022) is a concurrent work for data collection, but its source code is not available at the moment.
Revised: It should be noted that Explore2Offline (Lambert et al., 2022) is a concurrent work for data collection, but its source code is not available at the moment.

Original: A.2. O FFLINE RL
Revised: A.2. O FFLINE RL

Original: For the offline RL agent, we follow the findings reported in ExORL that even the vanilla off-policy
Revised: For the offline RL agent, we follow the findings reported in ExORL that even the vanilla off-policy

Original: RL algorithm of TD3 (Fujimoto et al., 2018) can outperform the carefully designed offline RL algorithms when the collected dataset is of high quality.
Revised: RL algorithm of TD3 (Fujimoto et al., 2018) can outperform the carefully designed offline RL algorithms when the collected dataset is of high quality.

Original: Thus, we implement an offline RL of TD 
Revised: Thus, we implement an offline RL of TD [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Walker (

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: Quadruped

Original: [Table] 
Revised: [Table] [Table]

Original: to evaluate the quality of the collected dataset.
Revised: to evaluate the quality of the collected dataset.

Original: The detailed implementation and hyperparamter settings can be found in ExORL
Revised: The detailed implementation and hyperparamter settings can be found in ExORL

Original: [Table]
Revised: [Table]

Original: B P ROOF OF L EMMA 3. 
Revised: B P ROOF OF L EMMA 3.

Original: Proof.
Revised: Proof.

Original: It has been shown in APT Liu & Abbeel (2021b) that the particle-based entropy estimator of the state representation z can be derived as
Revised: It has been shown in APT Liu & Abbeel (2021b) that the particle-based entropy estimator of the state representation z can be derived as

Original: [Equation]
Revised: [Equation]

Original: where b ( K ) represents a bias correlation and v Ki indicates the volume of the hypersphere with a radius of || z i − z K-NN
Revised: where b ( K ) represents a bias correlation and v Ki indicates the volume of the hypersphere with a radius of || z i − z K-NN

Original: i || between the z i and its K -th nearest neighbor z K-NN i .
Revised: i || between the z i and its K -th nearest neighbor z K-NN i .

Original: By substituting v Ki =
Revised: By substituting v Ki =

Original: ||
Revised: ||

Original: z i − z K-NN
Revised: z i − z K-NN

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where Γ is a gamma function and n z is the dimension of z , we can obtain
Revised: where Γ is a gamma function and n z is the dimension of z , we can obtain

Original: [Equation]
Revised: [Equation]

Original: Let u
Revised: Let u

Original: =
Revised: =

Original: ( z s , z a ) represent the state-action representation and we can further substitute z
Revised: ( z s , z a ) represent the state-action representation and we can further substitute z

Original: = u into
Revised: = u into

Original: Equation (6) to obtain
Revised: Equation (6) to obtain

Original: [Equation]
Revised: [Equation]

Original: C A DDITIONAL R ESULTS AND D ISCUSSIONS 
Revised: C A DDITIONAL R ESULTS AND D ISCUSSIONS C. F ULL R ESULTS OF M AIN E XPERIMENTS

Original: We present the performance scores of the four variants of CUDC at 100K and 500K steps in Table 4.

Original: Moreover, the detailed learning curves of the offline RL agent on 4 downstream tasks are shown in

Original: Figure 7.

Original: It can be concluded that adapting the environment step to reach more distant k -step future can substantially improve both sample efficiency and learning capabilities.

Original: Moreover, all proposed components facilitated by the proposed reachability module are necessarily important to yield the improvement.

Original: As a result, the full CUDC model further addressed the instability issue to obtain the minimum standard deviation among all methods.

Original: on 3 out of 4 tasks in sample efficiency (100K) and all four tasks in learning performance (500K) regimes, across 5 random seeds

Original: [Table]

Original: Figure 6 and Figure 7 show the full results on the 12 downstream tasks across 3 domains.
Revised: Figure 5 and Figure 6 show the full results on the 12 downstream tasks across 3 domains.

Original: Figuresummarizes the overall performances in 3 domains.
Revised: Figuresummarizes the overall performances in 3 domains.

Original: They demonstrate that our proposed methods significantly outperform the baseline methods in 3 downstream tasks of Walker and 3 downstream tasks of Quadruped.
Revised: They demonstrate that our proposed methods significantly outperform the baseline methods in 3 downstream tasks of Walker and 3 downstream tasks of Quadruped.

Original: However, in the hardest domain of Jaco Arm, the learning sample efficiency becomes poor in 3 downstream tasks although its learning performance at 500K step catches up with the best baseline method of ICM.
Revised: However, in the hardest domain of Jaco Arm, the learning sample efficiency becomes poor in 3 downstream tasks although its learning performance at 500K step catches up with the best baseline method of ICM.

Original: We explain this by the fact that our method can introduce more complexity and challenges for the agent whenever adapting the environment step to learn a better representation.
Revised: We explain this by the fact that our method can introduce more complexity and challenges for the agent whenever adapting the temporal distance k to learn a better representation.

Original: However, as Jaco Arm is indeed the most challenging domain, the introduced 
Revised: However, as Jaco Arm is indeed the most challenging domain, the introduced complexity cannot be fully coped with this environment.

Revised: Thus, the poor performance in sample efficiency can occur

Original: [Figure]
Revised: [Figure]

Original: complexity cannot be fully coped with this environment. Thus, the poor performance in sample efficiency can occur, and it is interesting to further study on how to cope with hard domain environments in the future works
Revised: and it is interesting to further study on how to cope with hard domain environments in the future works

Original: [Figure]
Revised: [Figure]

Original: 100K and 500K steps for the offline RL agent on each downstream task. CUDC significantly improves the sample-efficiency inout 12 tasks at 100K step and learning performances in all 12 tasks at 500K step.
Revised: [Figure]

Original: Finally, we plot how our proposed method adapts the environment steps during data collection in the Walker domain, as shown in Figure 9.
Revised: [Figure] [Figure] In Figure 8, we plot how our proposed method adapts the k -step during data collection in the Walker domain.

Original: It can be observed that agents take around 450K step to learn and adjust the environment step from k = 3 to k = 4 .
Revised: It can be observed that agents take around 450K step to learn and adjust the temporal distance from k = 3 to k = 4 .

Original: Then, it takes around just 200K steps to increase from k  4 to k = 5 .
Revised: Then, it takes around just 200K steps to increase from k = 4 to k = 5 .

Original: Finally, it takes more than 400K steps to reach k = 6 .
Revised: Finally, it takes more than 400K steps to reach k = 6 .

Original: This increasing behavior implies that at the early phase of training, we should not inject a too challenging k value.
Revised: This increasing behavior implies that at the early phase of training, we should not inject a too challenging k value.

Original: Once the agent has learned enough dynamics information, they are able to learn quickly on more challenging reachability analysis.
Revised: Once the agent has learned enough dynamics information, they are able to learn quickly on more challenging reachability analysis.

Original: However, after a certain stage () k = 5 in this domain), the learning becomes too difficult for the agent.
Revised: However, after a certain stage ( k = 5 in this domain), the learning becomes too difficult for the agent.

Original: Therefore, it is interesting to further adapt this over-difficult knowledge in the future works as well.
Revised: Therefore, it is interesting to further adapt this over-difficult knowledge in the future works as well.

Original: [Figure]
Revised: [Figure]

Revised: In Figure 9, we visualize how the loss of dynamics model (forward and inverse networks) and the contrastive loss of the reachable networks change w.r.t.

Revised: training steps.

Revised: Both losses decrease and converge during learning.

Revised: In particular, they only increase when the step k is adapted to increase.

Revised: After that, both losses decrease quickly.

Revised: The agent can well predict each sampled k -step future state being most reachable from its own current state rather than those from other transition tuples.

Revised: Even without inputting the sequence of intermediate actions or states, the agent can predict the k -step future state accurately.

Revised: During the model training, the feature representations are enhanced as well.

Revised: Therefore, it can be validated that the learned representation contains more semantic latents with dynamics information, with the self-supervised learning of agent’s internal belief.

Revised: [Figure]

Revised: In addition to the offline multi-task learning performance, we compare the quality of the collected datasets by plotting the normalized density of the true reward for the downstream task of Stand in Walker environment.

Revised: The visualization is shown in Figure 10.

Revised: It can be seen that the dataset collected by our proposed CUDC is of higher quality, with larger density for high rewards and a larger proportion at the low reward part.

Revised: Specifically, the mean trajectory reward of CUDC is

Revised: 0.312, which is 69% higher than APS, 21% higher than ProtoRL, 16% higher than ICM, and 10% higher than APT.

Revised: Moreover, the 75% quartile of the trajectory reward for CUDC is 0.498, which is 143% larger than APS, 42% larger than ProtoRL, 19% larger than ICM, 16% larger than APT.

Revised: Therefore, it indicates the effectiveness of our proposed method to collect higher-quality dataset, where increasingly more rewarding states have been visited.

Revised: [Figure]

Revised: C. E VALUATION BY A NOTHER O FFLINE RL A LGORITHM OF CQL

Revised: In this work, we consider the problem setting for offline RL into three main steps: data collection, reward relabeling, and downstream offline learning.

Revised: Our work focuses only on the first step of collecting high-quality dataset and thereby is agnostic to the offline RL algorithms.

Revised: For the main experiments, we chose TD3 (Fujimoto et al., 2018) to evaluate the multi-task downstream learning, since it was concluded in ExORL that the vanilla TD3 algorithm can effectively learn offline and even outperform carefully designed offline RL algorithms by improving the dataset quality.

Revised: To demonstrate that the dataset collected by our proposed method is of high quality than the other methods, we conduct additional experiments using another offline RL algorithm of CQL (Kumar et al., 2020) that regularizes the Q-values during training.

Revised: The results are shown in Figureand our proposed CUDC has also demonstrated improved sample efficiency at 100K learning steps and learning capability at 500K learning steps in all 4 downstream tasks against baseline methods.

Revised: By comparing the performance score at 100K and 500K steps of CUDC against the best baseline method, CUDC has achieved on average 18.2% and 12.3% improvement at respectively 100K and 500K CQL agent learning steps, across 4 downstream tasks.

Revised: Specifically, CUDC is 21% higher than the best baseline of APT at 100K steps on Run task while its learning performance is 15% higher than the best baseline of APS at 500K steps.

Revised: [Figure]

Revised: A v e r

Revised: [Figure]

Revised: E v a l ua t i

Revised: [Figure]

Revised: S c

Revised: o r

Revised: [Figure]

Revised: C. R ESULTS OF A DDING E ACH P ROPOSED C OMPONENT ON TOP OF E ACH O THER

Revised: We present the performance scores of the four variants of CUDC at 100K and 500K steps in Table 4.

Revised: Moreover, the detailed learning curves of the offline RL agent on 4 downstream tasks are shown in

Revised: Figure 6.

Revised: It can be concluded that adapting the temporal distance to reach more distant k -step future can substantially improve both sample efficiency and learning capabilities.

Revised: Moreover, all proposed components facilitated by the proposed reachability module are necessarily important to yield the improvement.

Revised: As a result, the full CUDC model further addressed the instability issue to obtain the minimum standard deviation among all methods

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: C. R ESULTS OF R EMOVING E ACH P ROPOSED C OMPONENT FROM F ULL M ODEL

Revised: To further investigate the effectiveness of each proposed component and quantify the importance of them

Revised: [Figure]

Revised: we carry out additional experiments by respectively removing each proposed component from the full model.

Revised: In particular, the following models are used to collect the task-agnostic dataset for the Walker environment with 5 random seeds.

Revised: • CUDC − Entropy : The proposed intrinsic reward of KNN-based particle entropy of state and action r H ( s t , a t ) is removed, and only the prediction error based reward r E ( s t , a t ) is used.

Revised: • CUDC − PE :

Revised: The proposed intrinsic reward of prediction error r E ( s t , a t ) is removed, and only the KNN-based particle entropy reward of state and action r H ( s t , a t ) is used.

Revised: • CUDC − Regularize : the mechanism of regularizing the backbone DDPG algorithm is removed.

Revised: • CUDC − Vary : k -step is fixed to be 3 throughout learning.

Revised: • CUDC − Reach : The proposed reachability module is removed while k -step is still adapted through the loss of the dynamics model.

Revised: • CUDC − Inverse :

Revised: The inverse networks of predicting the action and step k are removed.

Original: [Figure]
Revised: [Figure]

Revised: v e r

Revised: [Figure]

Revised: v a

Revised: [Figure]

Revised: t

Original: [Figure] 
Revised: [Figure] v age l [Figure] [Table]

Revised: Figure 13 shows the overall offline learning performances.

Revised: We can observe that limiting k =and removing the inverse networks will cause the most significant performance decreases among the evaluated models.

Revised: It implies that adapting the step between current and future states can help to learn rich representation, which plays the most important role in our proposed CUDC to collect high-quality dataset for offline multi-task learning.

Revised: Meanwhile, the inverse networks are necessary to learn the representation that is robust to the uncontrollable features by the agent’s actions and enables the encoders to capture the dynamics information in the learned representation.

Revised: To quantitatively measure the benefits brought by each proposed component

Revised: [Table]

Revised: we compute the relative performance scores at 100K for sample efficiency and 500K for learning capabilities based on the full model.

Revised: The results are summarized in Table 5.

Revised: For the sample efficiency at 100K steps, removing the k-step adaptation has caused the worst performance with a 18.1% decrease in Flip task and an overall 12.5% decrease across all 4 downstream tasks, followed by removing prediction-errorbased reward (

Revised: For the learning capabilities at 500K steps, removing the k-step adaptation has also resulted in the worst performance with a 19.1% decrease in Flip task and an overall 10.7% decrease across alldownstream tasks, followed by removing inverse networks (10.0%), removing entropy-based reward (9.7%) and removing prediction-error-based reward (6.9%).

Revised: Thus, we conclude that adapting how many steps into the future that the dynamics model should predict is most effective to the improved sample efficiency and learning capabilities, while the proposed mixed intrinsic reward is the second most effective.

Revised: C. A BLATION S TUDIES ON THE R ANGE OF V ARYING k -S TEP

Revised: In our work, we set the range of varying k-step from 3 to 6.

Revised: It starts from 3 as we follow the same setting as the ExORL benchmark, where all 8 data collection baselines strictly limit k=3 for the future state in each transition tuple.

Revised: As for the threshold of C w and C k , we did not specifically hypertune these values and they were just set to ensure that k can be varied from 3 to 6 adaptively during the 1M dataset collection process.

Revised: As for the upper bound of 6, it is an optimal end value to obtain the desired performances.

Revised: To support this finding, we carry out a sweep of the upper bound of k from 3 to 8, and present the results in Figure 14.

Revised: Firstly, it can be observed that the learning capabilities at 500K learning steps first increase and then decrease with the increase of the upper bound of k , across all 4 downstream tasks.

Revised: Setting the upper bound of 6 achieves the highest in

Revised: Walk and Run tasks while it is the second highest in Flip and Stand tasks.

Revised: Secondly, there is no clear trend of performance at 100K steps by setting different values of the upper bound for k .

Revised: It can be explained by the complexity caused by varying the k -step for the future states.

Revised: However, we can still observe that setting the upper bound of 6 achieves the highest in 3 tasks.

Revised: Therefore, we believe k should vary from 3 to 6 to learn rich representation with more semantic latents.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]