Original: S EMI -S UPERVISED O FFLINE R EINFORCEMENT L EARNING WITH A CTION -F REE T RAJECTORIES
Revised: S EMI -S UPERVISED O FFLINE R EINFORCEMENT L EARNING WITH A CTION -F REE T RAJECTORIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements.
Revised: Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements.

Original: We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting.
Revised: We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting.

Original: Here, an agent has access to two sets of trajectories: labelled trajectories containing state, action, reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information.
Revised: Here, an agent has access to two sets of trajectories: labelled trajectories containing state, action, reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information.

Original: For this setting, we develop a simple meta-algorithmic pipeline that learns an inversedynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories.
Revised: For this setting, we develop a simple meta-algorithmic pipeline that learns an inversedynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories.

Original: Empirically, we find this simple pipeline to be highly successful — on several D4RL benchmarks (Fu et al., 2020), certain offline RL algorithms can match the performance of variants trained on a fully labelled dataset even when we label only 10% trajectories from the low return regime.
Revised: Empirically, we find this simple pipeline to be highly successful — on several D4RL benchmarks (Fu et al., 2020), certain offline RL algorithms can match the performance of variants trained on a fully labelled dataset even when we label only 10% trajectories from the low return regime.

Original: Finally, we perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e g., inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.
Revised: Finally, we perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e g., inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One of the key challenges with deploying reinforcement learning (RL) agents is its prohibitive sample complexity for real-world applications.
Revised: One of the key challenges with deploying reinforcement learning (RL) agents is its prohibitive sample complexity for real-world applications.

Original: Offline reinforcement learning (RL) can significantly reduce the sample complexity by exploiting logged demonstrations from auxiliary data sources (Levine et al., 2020).
Revised: Offline reinforcement learning (RL) can significantly reduce the sample complexity by exploiting logged demonstrations from auxiliary data sources (Levine et al., 2020).

Original: However, contrary to curated benchmarks in use today, the nature of offline demonstrations in the real world can be highly varied.
Revised: However, contrary to curated benchmarks in use today, the nature of offline demonstrations in the real world can be highly varied.

Original: For example, the demonstrations could be misaligned due to frequency mismatch (Burns et al., 2022), use of different sensors, actuators, or dynamics (Reed et al., 2022; Lee et al., 2022), or lacking partial state (Ghosh et al., 2022; Rafailov et al., 2021; Mazoure et al., 2021), or reward information (Yu et al., 2022).
Revised: For example, the demonstrations could be misaligned due to frequency mismatch (Burns et al., 2022), use of different sensors, actuators, or dynamics (Reed et al., 2022; Lee et al., 2022), or lacking partial state (Ghosh et al., 2022; Rafailov et al., 2021; Mazoure et al., 2021), or reward information (Yu et al., 2022).

Original: Successful offline RL in the real world requires embracing these heterogeneous aspects for maximal data efficiency, similar to learning in humans.
Revised: Successful offline RL in the real world requires embracing these heterogeneous aspects for maximal data efficiency, similar to learning in humans.

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose a new semi-supervised setup for offline RL
Revised: In this work, we propose a new semi-supervised setup for offline RL

Original: Standard offline RL assumes trajectories to be sequences of observations, actions, and rewards.
Revised: Standard offline RL assumes trajectories to be sequences of observations, actions, and rewards.

Original: However, many data sources, such as videos or third-person demonstrations lack direct access to actions.
Revised: However, many data sources, such as videos or third-person demonstrations lack direct access to actions.

Original: Hence, we propose a semi-supervised setup, where an agent’s offline dataset also consists of action-unlabelled trajectories in addition to the aforementioned (action-labelled) trajectories.
Revised: Hence, we propose a semi-supervised setup, where an agent’s offline dataset also consists of action-unlabelled trajectories in addition to the aforementioned (action-labelled) trajectories.

Original: Standard offline
Revised: Standard offline

Original: RL algorithms, such as Conservative Q Learning (CQL; Kumar et al.
Revised: RL algorithms, such as Conservative Q Learning (CQL; Kumar et al.

Original: (2020)) or Decision Transformer (DT; Chen et al.
Revised: (2020)) or Decision Transformer (DT; Chen et al.

Original: (2021)), cannot directly operate on such unlabelled trajectories.
Revised: (2021)), cannot directly operate on such unlabelled trajectories.

Original: At the same time, naively throwing out the unlabelled trajectories can be wasteful, especially when they have high returns.
Revised: At the same time, naively throwing out the unlabelled trajectories can be wasteful, especially when they have high returns.

Original: Our goal in this work is to enable compute and data efficient learning with additional action-unlabelled trajectory logs.
Revised: Our goal in this work is to enable compute and data efficient learning with additional action-unlabelled trajectory logs.

Original: Unlike traditional semi-supervised learning, our setup has a few key differences.
Revised: Unlike traditional semi-supervised learning, our setup has a few key differences.

Original: First, we do not assume that the distribution of the labelled and unlabelled trajectories are necessarily identical.
Revised: First, we do not assume that the distribution of the labelled and unlabelled trajectories are necessarily identical.

Original: In realistic scenarios, we expect these to be different with unlabelled data having higher returns than labelled data e.g., videos of a human professional are easier to obtain than installing actuators for continuous control tasks.
Revised: In realistic scenarios, we expect these to be different with unlabelled data having higher returns than labelled data e.g., videos of a human professional are easier to obtain than installing actuators for continuous control tasks.

Original: We replicate such varied data quality setups in some of our experiments; Figure 1.1 shows an illustration of the difference in returns between the labelled and unlabelled dataset splits for the hopper - medium-expert D4RL dataset.
Revised: We replicate such varied data quality setups in some of our experiments; Figure 1.1 shows an illustration of the difference in returns between the labelled and unlabelled dataset splits for the hopper - medium-expert D4RL dataset.

Original: Second, our end goal goes beyond labeling the actions in the unlabelled trajectories, but rather we intend to use the unlabelled data for learning a downstream policy that is better than the behavioral policies used for generating the offline datasets.
Revised: Second, our end goal goes beyond labeling the actions in the unlabelled trajectories, but rather we intend to use the unlabelled data for learning a downstream policy that is better than the behavioral policies used for generating the offline datasets.

Original: Hence, there are two kinds of generalization challenges: generalizing from the labelled to the unlabelled data distribution and then going beyond the offline data distributions to get closer to the expert distribution.
Revised: Hence, there are two kinds of generalization challenges: generalizing from the labelled to the unlabelled data distribution and then going beyond the offline data distributions to get closer to the expert distribution.

Original: Regular offline RL is concerned only with the latter.
Revised: Regular offline RL is concerned only with the latter.

Original: Finally, we are mainly interested in the case where a significant majority of the trajectories in the offline dataset are unlabelled.
Revised: Finally, we are mainly interested in the case where a significant majority of the trajectories in the offline dataset are unlabelled.

Original: One motivating example for this setup is learning from videos or third-person demos.
Revised: One motivating example for this setup is learning from videos or third-person demos.

Original: There are tremendous amounts of internet videos that can be potentially used to train RL agents, yet they are without action labels and are of varying quality.
Revised: There are tremendous amounts of internet videos that can be potentially used to train RL agents, yet they are without action labels and are of varying quality.

Original: Our paper seeks to answer the following questions:
Revised: Our paper seeks to answer the following questions:

Original: How can we utilize the unlabelled data for improving the performance of offline RL algorithms?
Revised: How can we utilize the unlabelled data for improving the performance of offline RL algorithms?

Original: How does our performance vary as a function of data-centric properties, such as the size and return distributions of labelled and unlabelled datasets?
Revised: How does our performance vary as a function of data-centric properties, such as the size and return distributions of labelled and unlabelled datasets?

Original: How do offline RL algorithms compare in this setup?
Revised: How do offline RL algorithms compare in this setup?

Original: To answer these questions, we propose a meta-algorithmic pipeline to train policies in the semisupervised setup described above.
Revised: To answer these questions, we propose a meta-algorithmic pipeline to train policies in the semisupervised setup described above.

Original: We call our pipeline S
Revised: We call our pipeline S

Original: emi- S upervised O ffline R einforcement
Revised: emi- S upervised O ffline R einforcement

Original: L earning ( SS-ORL ).
Revised: L earning ( SS-ORL ).

Original: SS-ORL contains three simple and scalable steps: (1) train a multi-transition inverse dynamics model on labelled data, which predicts actions based on transition sequences, (2) fill in proxy-actions for unlabelled data, and finally (3) train an offline RL agent on the combined dataset.
Revised: SS-ORL contains three simple and scalable steps: (1) train a multi-transition inverse dynamics model on labelled data, which predicts actions based on transition sequences, (2) fill in proxy-actions for unlabelled data, and finally (3) train an offline RL agent on the combined dataset.

Original: Empirically, we instantiate SS-ORL with CQL (Kumar et al., 2020), DT (Chen et al., 2021), and TD3BC (Fujimoto & Gu, 2021) as the underlying offline RL algorithms respectively, and conduct experiments on the D4RL datasets (Fu et al., 2020).
Revised: Empirically, we instantiate SS-ORL with CQL (Kumar et al., 2020), DT (Chen et al., 2021), and TD3BC (Fujimoto & Gu, 2021) as the underlying offline RL algorithms respectively, and conduct experiments on the D4RL datasets (Fu et al., 2020).

Original: We highlight a few predominant trends from our experimental findings below:
Revised: We highlight a few predominant trends from our experimental findings below:

Original: Given low-quality labelled data, SS-ORL agents can exploit unlabelled data that contains highquality trajectories and thus improve performance.
Revised: Given low-quality labelled data, SS-ORL agents can exploit unlabelled data that contains highquality trajectories and thus improve performance.

Original: The absolute performance of SS-ORL is close to or even matches that of the oracle agents, which have access to complete action information.
Revised: The absolute performance of SS-ORL is close to or even matches that of the oracle agents, which have access to complete action information.

Original: When the labelled data quality is high, utilizing unlabelled data does not bring significant benefits.
Revised: When the labelled data quality is high, utilizing unlabelled data does not bring significant benefits.

Original: The choice of value vs. behavior cloning based methods can significantly affect performance in the semi-supervised setup.
Revised: The choice of value vs. behavior cloning based methods can significantly affect performance in the semi-supervised setup.

Original: In our experiments, CQL and TD3BC are less sensitive to the missing actions compared to DT.
Revised: In our experiments, CQL and TD3BC are less sensitive to the missing actions compared to DT.

Original: They enjoy better absolute performance when the labelled data is of low quality, and their performance gap relative to the oracle agent is also smaller.
Revised: They enjoy better absolute performance when the labelled data is of low quality, and their performance gap relative to the oracle agent is also smaller.

Original: See Appendix H for more details.
Revised: See Appendix H for more details.

Original:  R ELATED W ORK Offline
Revised:  R ELATED W ORK Offline

Original: RL The goal of offline RL is to learn effective policies from fixed datasets which are generated by unknown behavior policies.
Revised: RL The goal of offline RL is to learn effective policies from fixed datasets which are generated by unknown behavior policies.

Original: There are two main categories of model-free offline RL methods: value-based methods and behavior cloning (BC) based methods.
Revised: There are two main categories of model-free offline RL methods: value-based methods and behavior cloning (BC) based methods.

Original: Value-based methods attempt to learn the value functions based on temporal difference (TD) updates.
Revised: Value-based methods attempt to learn the value functions based on temporal difference (TD) updates.

Original: There is a line of work that aims to port existing off-policy value-based online RL methods to the offline setting, with various types of additional regularization components that encourage the learned policy to stay close to the behavior policy.
Revised: There is a line of work that aims to port existing off-policy value-based online RL methods to the offline setting, with various types of additional regularization components that encourage the learned policy to stay close to the behavior policy.

Original: Several representive techniques include specifically tailored policy parameterizations (Fujimoto et al., 2019;
Revised: Several representive techniques include specifically tailored policy parameterizations (Fujimoto et al., 2019;

Original: Ghasemipour et al., 2021), divergence-based regularization on the learned policy (Wu et al., 2019; Jaques et al., 2019; Kumar et al., 2019), and regularized value function estimation (Nachum et al., 2019; Kumar et al., 2020; Kostrikov et al., 2021a; Fujimoto & Gu, 2021;
Revised: Ghasemipour et al., 2021), divergence-based regularization on the learned policy (Wu et al., 2019; Jaques et al., 2019; Kumar et al., 2019), and regularized value function estimation (Nachum et al., 2019; Kumar et al., 2020; Kostrikov et al., 2021a; Fujimoto & Gu, 2021;

Original: Kostrikov et al., 2021b).
Revised: Kostrikov et al., 2021b).

Original: Recently, a growing body of work has tried to formulate offline RL as a supervised learning problem (Chen et al., 2021; Janner et al., 2021; Emmons et al., 2021).
Revised: Recently, a growing body of work has tried to formulate offline RL as a supervised learning problem (Chen et al., 2021; Janner et al., 2021; Emmons et al., 2021).

Original: Compared with the value-based methods, these methods enjoy several appealing properties including algorithmic simplicity and training stability.
Revised: Compared with the value-based methods, these methods enjoy several appealing properties including algorithmic simplicity and training stability.

Original: Generally speaking, these approaches can be viewed as conditional behavior cloning methods (Bain & Sammut, 1995), where the conditioning parameters are related information such as goals or rewards.
Revised: Generally speaking, these approaches can be viewed as conditional behavior cloning methods (Bain & Sammut, 1995), where the conditioning parameters are related information such as goals or rewards.

Original: Similar to value-based methods, these can be extended to the online setup as well (Zheng et al., 2022) and demonstrate excellent performance in hybrid setups involving both offline data and online interactions.
Revised: Similar to value-based methods, these can be extended to the online setup as well (Zheng et al., 2022) and demonstrate excellent performance in hybrid setups involving both offline data and online interactions.

Original: Semi-supervised Learning Semi-supervised learning (SSL) is a sub-area of machine learning that studies approaches to train predictors from a small amount of labelled data combined with a large amount of unlabelled data.
Revised: Semi-supervised Learning Semi-supervised learning (SSL) is a sub-area of machine learning that studies approaches to train predictors from a small amount of labelled data combined with a large amount of unlabelled data.

Original: In supervised learning, predictors only learn from labelled data.
Revised: In supervised learning, predictors only learn from labelled data.

Original: However, labelled training examples often require human annotation efforts and are thus hard to obtain, whereas unlabelled data can be comparatively easy to collect.
Revised: However, labelled training examples often require human annotation efforts and are thus hard to obtain, whereas unlabelled data can be comparatively easy to collect.

Original: The research on semi-supervised learning spans several decades.
Revised: The research on semi-supervised learning spans several decades.

Original: One of the oldest SSL techniques, self-training , was originally proposed in the 1960s (Fralick, 1967).
Revised: One of the oldest SSL techniques, self-training , was originally proposed in the 1960s (Fralick, 1967).

Original: There, a predictor is first trained on the labelled data.
Revised: There, a predictor is first trained on the labelled data.

Original: Then, at each training round, according to certain selection criteria such as model uncertainty, a portion of the unlabelled data is annotated by the predictor and added into the training set for the next round.
Revised: Then, at each training round, according to certain selection criteria such as model uncertainty, a portion of the unlabelled data is annotated by the predictor and added into the training set for the next round.

Original: We refer the readers to Zhu (2005); Chapelle et al.
Revised: We refer the readers to Zhu (2005); Chapelle et al.

Original: (2006); Ouali et al.
Revised: (2006); Ouali et al.

Original: (2020); Van Engelen & Hoos (2020) for comprehensive literature surveys.
Revised: (2020); Van Engelen & Hoos (2020) for comprehensive literature surveys.

Original: Imitation Learning from Observations There have been several works in imitation learning (IL) which do not assume access to the full set of actions, such as BCO (Torabi et al., 2018a),
Revised: Imitation Learning from Observations There have been several works in imitation learning (IL) which do not assume access to the full set of actions, such as BCO (Torabi et al., 2018a),

Original: MoBILE (Kidambi et al., 2021), GAIfO (Torabi et al., 2018b) or third-person IL approaches (Stadie et al., 2017; Sharma et al., 2019).
Revised: MoBILE (Kidambi et al., 2021), GAIfO (Torabi et al., 2018b) or third-person IL approaches (Stadie et al., 2017; Sharma et al., 2019).

Original: The recent work of Baker et al.
Revised: The recent work of Baker et al.

Original: (2022) also considered a setup where a small number of labelled actions are available in addition to a large unlabelled dataset.
Revised: (2022) also considered a setup where a small number of labelled actions are available in addition to a large unlabelled dataset.

Original: A key difference between our work and these is that the IL setup typically assumes that all trajectories are generated by an expert, unlike our offline setup.
Revised: A key difference between our work and these is that the IL setup typically assumes that all trajectories are generated by an expert, unlike our offline setup.

Original: Further, some of these methods even permit reward-free interactions with the environment which is not possible in the offline setup.
Revised: Further, some of these methods even permit reward-free interactions with the environment which is not possible in the offline setup.

Original: Learning from Videos Closely related to IL from observations, several works (Schmeckpeper et al.
Revised: Learning from Videos Closely related to IL from observations, several works (Schmeckpeper et al.

Original: , 2020b;a) consider training agents with human video demonstrations, which are without action annotations.
Revised: , 2020b;a) consider training agents with human video demonstrations, which are without action annotations.

Original: Distinct from out setup, in those works, the offline observational data (videos) are from a different embodiment.
Revised: Distinct from out setup, in those works, the offline observational data (videos) are from a different embodiment.

Original: Moreover, the agents can interact with the environment, and can even collect reward information sometimes.
Revised: Moreover, the agents can interact with the environment, and can even collect reward information sometimes.

Original:  S EMI -S UPERVISED O FFLINE R EINFORCEMENT L EARNING
Revised:  S EMI -S UPERVISED O FFLINE R EINFORCEMENT L EARNING

Original: Preliminaries We model our environment as a Markov decision process (MDP) (Bellman, 1957) denoted by x S , A , p, P, R, γ y , where S is the state space,
Revised: Preliminaries We model our environment as a Markov decision process (MDP) (Bellman, 1957) denoted by x S , A , p, P, R, γ y , where S is the state space,

Original: A is the action space, p p s 1 q is the distribution of the initial state, P p s t ` 1 | s t , a t q is the transition probability distribution, R p s t , a t q is the deterministic reward function, and γ is the discount factor.
Revised: A is the action space, p p s 1 q is the distribution of the initial state, P p s t ` 1 | s t , a t q is the transition probability distribution, R p s t , a t q is the deterministic reward function, and γ is the discount factor.

Original: At each timestep t , the agent observes a state s t P S and executes an action a t P A .
Revised: At each timestep t , the agent observes a state s t P S and executes an action a t P A .

Original: As a response, the environment moves the agent to the next state s t ` 1 „ P p¨| s t , a t q , and also returns the agent a reward r t “ R p s t , a t q .
Revised: As a response, the environment moves the agent to the next state s t ` 1 „ P p¨| s t , a t q , and also returns the agent a reward r t “ R p s t , a t q .

Original: 3. P ROPOSED S ETUP
Revised: 3. P ROPOSED S ETUP

Original: We assume the agent has access to a static offline dataset T offline .
Revised: We assume the agent has access to a static offline dataset T offline .

Original: The dataset consists of trajectories collected by certain unknown policies, which are not necessarily optimal.
Revised: The dataset consists of trajectories collected by certain unknown policies, which are not necessarily optimal.

Original: Let τ denote a trajectory and | τ | denote its length.
Revised: Let τ denote a trajectory and | τ | denote its length.

Original: We assume that all the trajectories in T offline contain complete rewards and states.
Revised: We assume that all the trajectories in T offline contain complete rewards and states.

Original: However, only a small subset of them contain action labels, while most of the trajectories are missing actions.
Revised: However, only a small subset of them contain action labels, while most of the trajectories are missing actions.

Original: We are interested in learning a policy by leveraging the offline dataset without interacting with the environment.
Revised: We are interested in learning a policy by leveraging the offline dataset without interacting with the environment.

Original: This setup is analogous to semi-supervised learning, where actions serve the role of labels .
Revised: This setup is analogous to semi-supervised learning, where actions serve the role of labels .

Original: Hence, we also refer to the complete trajectories as labelled data (denoted by T labelled ) and the action-free trajectories as unlabelled data (denoted by T unlabelled ).
Revised: Hence, we also refer to the complete trajectories as labelled data (denoted by T labelled ) and the action-free trajectories as unlabelled data (denoted by T unlabelled ).

Original: Further, we assume the labelled data are sampled from a distribution P labelled and the unlabelled data are sampled from P unlabelled .
Revised: Further, we assume the labelled data are sampled from a distribution P labelled and the unlabelled data are sampled from P unlabelled .

Original: In general, the two distributions can be different.
Revised: In general, the two distributions can be different.

Original: Practically, one case we are particularly interested in is when P labelled generates low-to-moderate quality trajectories, whereas P unlabelled generates trajectories of diverse qualities including ones with high returns.
Revised: Practically, one case we are particularly interested in is when P labelled generates low-to-moderate quality trajectories, whereas P unlabelled generates trajectories of diverse qualities including ones with high returns.

Original: Our setup shares some similarities with state-only imitation learning (Ijspeert et al., 2002; Bentivegna et al., 2002; Torabi et al., 2019) in the use of action-unlabelled trajectories.
Revised: Our setup shares some similarities with state-only imitation learning (Ijspeert et al., 2002; Bentivegna et al., 2002; Torabi et al., 2019) in the use of action-unlabelled trajectories.

Original: However, there are also some key differences.
Revised: However, there are also some key differences.

Original: In state-only IL, the unlabelled demonstrations are from the same distribution as the labelled demonstrations and correspond to a near-optimal expert policy.
Revised: In state-only IL, the unlabelled demonstrations are from the same distribution as the labelled demonstrations and correspond to a near-optimal expert policy.

Original: In our setting, both
Revised: In our setting, both

Original: 1 Input: trajectories T labelled and T unlabelled , IDM transition size k , offline RL method ORL // train a stochastic
Revised: 1 Input: trajectories T labelled and T unlabelled , IDM transition size k , offline RL method ORL // train a stochastic

Original: 2 p θ Ð argmin θ E a t , s t ´ k : t ` k ` 1 „ T labelled r´ log ϕ θ p a t | s t ´ k : t ` k ` 1 qs // fill in the proxy actions for the unlabelled data 3 T proxy Ð ∅ 4 for each trajectory τ P T unlabelled do p a t Ð mean of N ` µ p θ p s t ´ k : t ` k ` 1 q , Σ p θ p s t ´ k : t ` k ` 1 q ˘ , t “ 1 , . .
Revised: 2 p θ Ð argmin θ E a t , s t ´ k : t ` k ` 1 „ T labelled r´ log ϕ θ p a t | s t ´ k : t ` k ` 1 qs // fill in the proxy actions for the unlabelled data 3 T proxy Ð ∅ 4 for each trajectory τ P T unlabelled do p a t Ð mean of N ` µ p θ p s t ´ k : t ` k ` 1 q , Σ p θ p s t ´ k : t ` k ` 1 q ˘ , t “ 1 , . .

Original: . , | τ | τ proxy Ð τ with proxy actions t p a t u | τ | t “ 1 filled in
Revised: . , | τ | τ proxy Ð τ with proxy actions t p a t u | τ | t “ 1 filled in

Original: T proxy Ð T proxy Ť t τ proxy u // train an offline RL agent using the combined data 8 π Ð policy obtained by training ORL using dataset T labelled Ť T proxy 9 Output: π
Revised: T proxy Ð T proxy Ť t τ proxy u // train an offline RL agent using the combined data 8 π Ð policy obtained by training ORL using dataset T labelled Ť T proxy 9 Output: π

Original: P labelled and P unlabelled can be different from each other and also from the expert policy.
Revised: P labelled and P unlabelled can be different from each other and also from the expert policy.

Original: Further, many state-only imitation learning algorithms (e.g., Gupta et al.
Revised: Further, many state-only imitation learning algorithms (e.g., Gupta et al.

Original: ; Torabi et al.
Revised: ; Torabi et al.

Original: (2018a;b); Liu et al.
Revised: (2018a;b); Liu et al.

Original: (2018); Sermanet et al.
Revised: (2018); Sermanet et al.

Original: (2018)), similar to their original counterparts (e.g.
Revised: (2018)), similar to their original counterparts (e.g.

Original: , Ho & Ermon (2016); Kim et al.
Revised: , Ho & Ermon (2016); Kim et al.

Original: (2020)), permit (reward-free) interactions with the environments.
Revised: (2020)), permit (reward-free) interactions with the environments.

Original: This is not possible in our proposed offline semi-supervised setup where the agents are only provided with T labelled and T unlabelled .
Revised: This is not possible in our proposed offline semi-supervised setup where the agents are only provided with T labelled and T unlabelled .

Original: 3. T RAINING P IPELINE
Revised: 3. T RAINING P IPELINE

Original: RL policies trained on low to moderate quality offline trajectories are often sub-optimal, as many of the trajectories might not have high return and only cover a limited part of the state space.
Revised: RL policies trained on low to moderate quality offline trajectories are often sub-optimal, as many of the trajectories might not have high return and only cover a limited part of the state space.

Original: Our goal is to find a way to combine the action labelled trajectories and the unlabelled action-free trajectories, so that the offline agent can exploit structures in the unlabelled data to improve performance.
Revised: Our goal is to find a way to combine the action labelled trajectories and the unlabelled action-free trajectories, so that the offline agent can exploit structures in the unlabelled data to improve performance.

Original: One natural strategy is to fill in proxy actions for those unlabelled trajectories, and use the annotated data together with the labelled data as a whole to train an offline RL agent.
Revised: One natural strategy is to fill in proxy actions for those unlabelled trajectories, and use the annotated data together with the labelled data as a whole to train an offline RL agent.

Original: Since we assume both the labelled and unlabelled trajectories contain the states, we can train an inverse dynamics model (IDM) ϕ that predicts actions using the states.
Revised: Since we assume both the labelled and unlabelled trajectories contain the states, we can train an inverse dynamics model (IDM) ϕ that predicts actions using the states.

Original: Once we obtain the IDM, we use it to generate the proxy actions for the unlabelled trajectories.
Revised: Once we obtain the IDM, we use it to generate the proxy actions for the unlabelled trajectories.

Original: Finally, we combine those proxy-labelled trajectories with the labelled trajectories, and train an agent using the offline RL algorithm of choice.
Revised: Finally, we combine those proxy-labelled trajectories with the labelled trajectories, and train an agent using the offline RL algorithm of choice.

Original: In particular, we propose a stochastic multi-transition IDM (see Section 3.3), which is favored by our experiments.
Revised: In particular, we propose a stochastic multi-transition IDM (see Section 3.3), which is favored by our experiments.

Original: Our meta-algorithmic pipeline is summarized in Algorithm 1.
Revised: Our meta-algorithmic pipeline is summarized in Algorithm 1.

Original: Remarks.
Revised: Remarks.

Original: The annotation process, which involves training an IDM on the labelled data and generating proxy actions for the unlabelled trajectories, is similar to one step of self-training (Fralick, 1967).
Revised: The annotation process, which involves training an IDM on the labelled data and generating proxy actions for the unlabelled trajectories, is similar to one step of self-training (Fralick, 1967).

Original: A key difference is that in self-training, the predictor is trained in multiple rounds.
Revised: A key difference is that in self-training, the predictor is trained in multiple rounds.

Original: Once an initial predictor is trained, it is used for obtaining annotations on the unlabelled dataset.
Revised: Once an initial predictor is trained, it is used for obtaining annotations on the unlabelled dataset.

Original: Then, a subset of annotated data is selected according to certain criteria, and added into the training set for the next round.
Revised: Then, a subset of annotated data is selected according to certain criteria, and added into the training set for the next round.

Original: As opposed to self-training, we do not retrain the IDM but directly move to the next stage, where we train the agent using the combined data.
Revised: As opposed to self-training, we do not retrain the IDM but directly move to the next stage, where we train the agent using the combined data.

Original: There are a few reasons that we do not employ self-training for IDM.
Revised: There are a few reasons that we do not employ self-training for IDM.

Original: First, it is computationally expensive to execute multiple rounds of training.
Revised: First, it is computationally expensive to execute multiple rounds of training.

Original: More importantly, our end goal is to obtain a downstream policy with improved performance via utilizing the proxy-labelled data.
Revised: More importantly, our end goal is to obtain a downstream policy with improved performance via utilizing the proxy-labelled data.

Original: One commonly used data selection criterion for self-training is based on the model uncertainty.
Revised: One commonly used data selection criterion for self-training is based on the model uncertainty.

Original: There, one adds the proxy-labelled data with sufficiently low predictive uncertainty into the training set for the next round.
Revised: There, one adds the proxy-labelled data with sufficiently low predictive uncertainty into the training set for the next round.

Original: However, we empirically found that such an uncertainty based augmentation strategy did not improve the performance of SS-ORL agents.
Revised: However, we empirically found that such an uncertainty based augmentation strategy did not improve the performance of SS-ORL agents.

Original: See Section 4.3 and Appendix F for the experiment details.
Revised: See Section 4.3 and Appendix F for the experiment details.

Original: 3. S TOCHASTIC M ULTI - TRANSITION I NVERSE D YNAMIC M ODEL
Revised: 3. S TOCHASTIC M ULTI - TRANSITION I NVERSE D YNAMIC M ODEL

Original: In past work (Pathak et al., 2017), the IDM typically maps two subsequent states p s t , s t ` 1 q to a t .
Revised: In past work (Pathak et al., 2017), the IDM typically maps two subsequent states p s t , s t ` 1 q to a t .

Original: We introduce a multi-transition IDM that predicts a t using both transitions before and after timestep t , which we found works better empirically.
Revised: We introduce a multi-transition IDM that predicts a t using both transitions before and after timestep t , which we found works better empirically.

Original: More precisely, our inverse dynamic model which predicts a t using 2 k ` 1 transitions, including the current transition p s t , s t ` 1 q , the previous k transitions that leads to s t , and the next k transitions starting from s t ` 1 .
Revised: More precisely, our inverse dynamic model which predicts a t using 2 k ` 1 transitions, including the current transition p s t , s t ` 1 q , the previous k transitions that leads to s t , and the next k transitions starting from s t ` 1 .

Original: We call k the transition size parameter.
Revised: We call k the transition size parameter.

Original: Let s t ´ k : t ` k ` 1 denote the sequence s min p 0 ,t ´ k q , . . .
Revised: Let s t ´ k : t ` k ` 1 denote the sequence s min p 0 ,t ´ k q , . . .

Original: , s t , s t ` 1 , . . .
Revised: , s t , s t ` 1 , . . .

Original: , s max p| τ | ,t ` k ` 1 q q .
Revised: , s max p| τ | ,t ` k ` 1 q q .

Original: Specifically, we model the distribution of a t as a multivariate Gaussian distribution with a diagonal covariance matrix:
Revised: Specifically, we model the distribution of a t as a multivariate Gaussian distribution with a diagonal covariance matrix:

Original: [Equation]
Revised: [Equation]

Original: Let ϕ θ p a t | s t ´ k : t ` k ` 1 q be the probability density function of N `
Revised: Let ϕ θ p a t | s t ´ k : t ` k ` 1 q be the probability density function of N `

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: ´
Revised: ´

Original: [Equation]
Revised: [Equation]

Original: ` k ` 1 q ˘ .
Revised: ` k ` 1 q ˘ .

Original: Given the labelled trajectories T labelled , we minimize the negative log-likelihood loss E a t , s t ´ k : t ` k ` 1 „ T labelled r´ log ϕ θ p a t | s t ´ k : t ` k ` 1 qs .
Revised: Given the labelled trajectories T labelled , we minimize the negative log-likelihood loss E a t , s t ´ k : t ` k ` 1 „ T labelled r´ log ϕ θ p a t | s t ´ k : t ` k ` 1 qs .

Original: Note that the standard IDM which predicts a t from p s t , s t ` 1 q under the ℓ 2 loss, is a special case subsumed by our model: it is equivalent to the case k “ 0 and the diagonal entries of Σ θ (i.e., the variances of each action dimension) are all the same.
Revised: Note that the standard IDM which predicts a t from p s t , s t ` 1 q under the ℓ 2 loss, is a special case subsumed by our model: it is equivalent to the case k “ 0 and the diagonal entries of Σ θ (i.e., the variances of each action dimension) are all the same.

Original: Choosing k ą 0 allows us to account for non-Markovian behaviour policies and partially observable
Revised: Choosing k ą 0 allows us to account for non-Markovian behaviour policies and partially observable

Original: MDP (POMDP), see Appendix E.1.
Revised: MDP (POMDP), see Appendix E.1.

Original: For all the experiments in this paper, we use k “ 1 .
Revised: For all the experiments in this paper, we use k “ 1 .

Original: We ablate this design choice in Section 4.3.
Revised: We ablate this design choice in Section 4.3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments aim to answer three primary questions:
Revised: Our experiments aim to answer three primary questions:

Original: Can SS-ORL closely track or even match the performance for fully supervised offline reinforcement learning, when only a small subset of trajectories are labelled?
Revised: Can SS-ORL closely track or even match the performance for fully supervised offline reinforcement learning, when only a small subset of trajectories are labelled?

Original: How does the performance of SS-ORL vary as a function of the size and qualities of the labelled and unlabelled datasets?
Revised: How does the performance of SS-ORL vary as a function of the size and qualities of the labelled and unlabelled datasets?

Original: Do different offline RL methods respond differently under varying setups of data size and qualities?
Revised: Do different offline RL methods respond differently under varying setups of data size and qualities?

Original: To answer these questions, we focus on three Gym locomotion tasks hopper , walker , and halfcheetah , and we use the v2 medium-expert , medium and medium-replay datasetsfrom the D4RL benchmark (Fu et al., 2020).
Revised: To answer these questions, we focus on three Gym locomotion tasks hopper , walker , and halfcheetah , and we use the v2 medium-expert , medium and medium-replay datasetsfrom the D4RL benchmark (Fu et al., 2020).

Original: We address the first question in Section 4.1 and the other two in Section 4.2, respectively.
Revised: We address the first question in Section 4.1 and the other two in Section 4.2, respectively.

Original: Finally, we discuss the design choices for SS-ORL in Section 4.3.
Revised: Finally, we discuss the design choices for SS-ORL in Section 4.3.

Original: 4. B ENCHMARKING
Revised: 4. B ENCHMARKING

Original: Data Setup For a given offline dataset, we subsample 10% of the total trajectories from the dataset, whose returns are from the bottom q % , 10 ď q ď 100 .
Revised: Data Setup For a given offline dataset, we subsample 10% of the total trajectories from the dataset, whose returns are from the bottom q % , 10 ď q ď 100 .

Original: We keep the actions for those trajectories, and discard the actions for the rest.
Revised: We keep the actions for those trajectories, and discard the actions for the rest.

Original: We call this setup the coupled setup, since P labelled and P unlabelled will change simultaneously when we vary the value of q .
Revised: We call this setup the coupled setup, since P labelled and P unlabelled will change simultaneously when we vary the value of q .

Original: When q “ 100 , we are uniformly sampling the trajectories and we have P labelled “ P unlabelled .
Revised: When q “ 100 , we are uniformly sampling the trajectories and we have P labelled “ P unlabelled .

Original: Under this setup, we always have 10% trajectories labelled and 90% unlabelled, and the total amount of data used later for training the offline RL agent is the original offline dataset size.
Revised: Under this setup, we always have 10% trajectories labelled and 90% unlabelled, and the total amount of data used later for training the offline RL agent is the original offline dataset size.

Original: This allows us to easily compare our results with results under the standard, fully labelled setup.
Revised: This allows us to easily compare our results with results under the standard, fully labelled setup.

Original: In Section 4.2, we shall decouple the distributions P labelled and
Revised: In Section 4.2, we shall decouple the distributions P labelled and

Original: P unlabelled for a thorough understanding of their individual influences.
Revised: P unlabelled for a thorough understanding of their individual influences.

Original: Inverse Dynamic Model We train an IDM as described in Section 3 with parameter k “ 1 .
Revised: Inverse Dynamic Model We train an IDM as described in Section 3 with parameter k “ 1 .

Original: In other words, the IDM predicts a t using 3 consecutive transitions: p s t ´ 1 , s t , s t ` 1 , s t ` 2 q .
Revised: In other words, the IDM predicts a t using 3 consecutive transitions: p s t ´ 1 , s t , s t ` 1 , s t ` 2 q .

Original: The mean and the covariance matrix are predicted by two independent multilayer perceptrons (MLPs), each contains two hidden layers and 1024 hidden units per layer.
Revised: The mean and the covariance matrix are predicted by two independent multilayer perceptrons (MLPs), each contains two hidden layers and 1024 hidden units per layer.

Original: To prevent overfitting, we randomly sample 10% of the labelled trajectories as the validation set, and use the IDM that yields the best validation error within 100 k training iterations.
Revised: To prevent overfitting, we randomly sample 10% of the labelled trajectories as the validation set, and use the IDM that yields the best validation error within 100 k training iterations.

Original: Offline RL Methods We instantiate Algorithm 1 with DT, CQL and TD3BC (Fujimoto & Gu, 2021) and test their performances.
Revised: Offline RL Methods We instantiate Algorithm 1 with DT, CQL and TD3BC (Fujimoto & Gu, 2021) and test their performances.

Original: Among these methods, DT is a recently proposed conditional behavior cloning (BC) method that uses sequence modeling tools to model the trajectories; CQL is a representative value-based offline RL method; and TD3BC is a hybrid method which adds a BC term to regularize the Q-learning updates.
Revised: Among these methods, DT is a recently proposed conditional behavior cloning (BC) method that uses sequence modeling tools to model the trajectories; CQL is a representative value-based offline RL method; and TD3BC is a hybrid method which adds a BC term to regularize the Q-learning updates.

Original: We refer to those instantiations as SS-DT , SS-CQL and
Revised: We refer to those instantiations as SS-DT , SS-CQL and

Original: SS-TD3BC , respectively.
Revised: SS-TD3BC , respectively.

Original: We defer the implementation details to Appendix A.
Revised: We defer the implementation details to Appendix A.

Original: Results We compare the performance of those SS-ORL agents with corresponding baseline and oracle agents.
Revised: Results We compare the performance of those SS-ORL agents with corresponding baseline and oracle agents.

Original: The baseline agents are trained on the labelled trajectories only, and the oracle agents are trained on the full offline dataset with action labels.
Revised: The baseline agents are trained on the labelled trajectories only, and the oracle agents are trained on the full offline dataset with action labels.

Original: Intuitively, the performances of the baseline and the oracle agents can be considered as the (estimated) lower and upper bounds for the performance of the SS-ORL agents.
Revised: Intuitively, the performances of the baseline and the oracle agents can be considered as the (estimated) lower and upper bounds for the performance of the SS-ORL agents.

Original: For each method, we train 5 instances under different seeds, and
Revised: For each method, we train 5 instances under different seeds, and

Original: [Figure]
Revised: [Figure]

Original: for each instance we run 30 evaluation trajectories.
Revised: for each instance we run 30 evaluation trajectories.

Original: We report the average return and the standard deviation after 200 k iterations.
Revised: We report the average return and the standard deviation after 200 k iterations.

Original: Figure 4.1 plots the results on medium-expert datasets.
Revised: Figure 4.1 plots the results on medium-expert datasets.

Original: For all the three environments and all the three offline RL methods, the SS-ORL agents improve upon the baselines.
Revised: For all the three environments and all the three offline RL methods, the SS-ORL agents improve upon the baselines.

Original: Remarkably, even when the labelled data quality is low, the SS-ORL agents are able to obtain decent returns.
Revised: Remarkably, even when the labelled data quality is low, the SS-ORL agents are able to obtain decent returns.

Original: For example, when q “ 10 , i.e., the labelled trajectories are the bottom 10% trajectories, the average return obtained by SS-TD3BC is 0 .
Revised: For example, when q “ 10 , i.e., the labelled trajectories are the bottom 10% trajectories, the average return obtained by SS-TD3BC is 0 .

Original: 91 and 0 .
Revised: 91 and 0 .

Original: 79 for hopper , walker and halfcheetah .
Revised: 79 for hopper , walker and halfcheetah .

Original: On average, this is 87 .
Revised: On average, this is 87 .

Original: 4% relative to the oracle performance ( 1 . 02 , 1 . 1 and 0 . 89 ).
Revised: 4% relative to the oracle performance ( 1 . 02 , 1 . 1 and 0 . 89 ).

Original: As the value q increases, the labelled data quality increases and the distributions P labelled and
Revised: As the value q increases, the labelled data quality increases and the distributions P labelled and

Original: P unlabelled are getting closer.
Revised: P unlabelled are getting closer.

Original: The performance of the SS-ORL agents also keeps increasing and finally matches the performance of the oracle agents.
Revised: The performance of the SS-ORL agents also keeps increasing and finally matches the performance of the oracle agents.

Original: Similar observations can be found in the results of medium and medium-replay datasets, see Figure C.1 and C.2.
Revised: Similar observations can be found in the results of medium and medium-replay datasets, see Figure C.1 and C.2.

Original: We found relatively suboptimal results for DT on halfcheetah in all cases, consistent with prior results in Zheng et al.
Revised: We found relatively suboptimal results for DT on halfcheetah in all cases, consistent with prior results in Zheng et al.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: We conduct experiments to understand the semi-supervised approach from the perspective of both datasets and learning algorithms.
Revised: We conduct experiments to understand the semi-supervised approach from the perspective of both datasets and learning algorithms.

Original: For a systematic study, we depart from the coupled setup in
Revised: For a systematic study, we depart from the coupled setup in

Original: Section 4.1 and consider a decoupling of the labelled data distributions P labelled and the unlabelled data distribution P unlabelled .
Revised: Section 4.1 and consider a decoupling of the labelled data distributions P labelled and the unlabelled data distribution P unlabelled .

Original: We first vary the quality of the labelled and unlabelled trajectories, and examine how the final performance of those SS-ORL agents changes.
Revised: We first vary the quality of the labelled and unlabelled trajectories, and examine how the final performance of those SS-ORL agents changes.

Original: Next, we vary the size of the labelled and unlabelled trajectories and investigate their influences.
Revised: Next, we vary the size of the labelled and unlabelled trajectories and investigate their influences.

Original: To understand how the value-based methods and the BC methods will potentially react differently under these data setups.
Revised: To understand how the value-based methods and the BC methods will potentially react differently under these data setups.

Original: we report the results of SS-CQL and SS-DT for the aforementioned setups.
Revised: we report the results of SS-CQL and SS-DT for the aforementioned setups.

Original: Last, we ablate the design choice of the transition size k for the proposed IDM.
Revised: Last, we ablate the design choice of the transition size k for the proposed IDM.

Original: For all the experiments we present, the results are aggregated over 5 instances with different seeds.
Revised: For all the experiments we present, the results are aggregated over 5 instances with different seeds.

Original: Quality of Unlabelled Data We divide the trajectories from the hopper - medium-expert dataset into 3 groups, which consist of trajectories whose returns are the bottom 0% to 33%, 33% to 67%, and
Revised: Quality of Unlabelled Data We divide the trajectories from the hopper - medium-expert dataset into 3 groups, which consist of trajectories whose returns are the bottom 0% to 33%, 33% to 67%, and

Original: [Figure]
Revised: [Figure]

Original: (a)
Revised: (a)

Original: We fix the labelled data quality and vary the unlabelled data quality.
Revised: We fix the labelled data quality and vary the unlabelled data quality.

Original: When the labelled data quality is low or moderate, SS-ORL can significantly improve the performance upon the baselines by utilizing high quality unlabelled data
Revised: When the labelled data quality is low or moderate, SS-ORL can significantly improve the performance upon the baselines by utilizing high quality unlabelled data

Original: [Figure]
Revised: [Figure]

Original: Both the sizes of the labelled and unlabelled data are 10% of the offline dataset size.
Revised: Both the sizes of the labelled and unlabelled data are 10% of the offline dataset size.

Original: 67% to 100%, respectively.
Revised: 67% to 100%, respectively.

Original: We refer to them Low , Medium , and High quality groups.
Revised: We refer to them Low , Medium , and High quality groups.

Original: In particular, the High group contains trajectories generated by the expert agents (Fu et al., 2020).
Revised: In particular, the High group contains trajectories generated by the expert agents (Fu et al., 2020).

Original: As before, we report the performance of DT and CQL agents trained on the labelled data only as the baselines.
Revised: As before, we report the performance of DT and CQL agents trained on the labelled data only as the baselines.

Original: We also report the results under the oracle mode, where we fill in the unlabelled trajectories with the true actions, and combine them with the labelled trajectories to train offline RL agents.
Revised: We also report the results under the oracle mode, where we fill in the unlabelled trajectories with the true actions, and combine them with the labelled trajectories to train offline RL agents.

Original: We first report the performance of SS-DT when the labelled data is sampled from the High group, and the unlabelled data are sampled from Low , Med , and High groups, respectively.
Revised: We first report the performance of SS-DT when the labelled data is sampled from the High group, and the unlabelled data are sampled from Low , Med , and High groups, respectively.

Original: Both the size of the labelled and unlabelled trajectories are 10% of the total offline dataset size.
Revised: Both the size of the labelled and unlabelled trajectories are 10% of the total offline dataset size.

Original: The top left panel of
Revised: The top left panel of

Original: Figure 4.2a plots the results.
Revised: Figure 4.2a plots the results.

Original: Clearly, when the labelled data quality is high, training on the labelled data only is sufficient to achieve the expert performance, and adding unlabelled data does not bring extra benefits.
Revised: Clearly, when the labelled data quality is high, training on the labelled data only is sufficient to achieve the expert performance, and adding unlabelled data does not bring extra benefits.

Original: We repeat the same experiment when the labelled data is sampled from Medium and
Revised: We repeat the same experiment when the labelled data is sampled from Medium and

Original: Low , see the top middle and top right panels of Figure 4.2a.
Revised: Low , see the top middle and top right panels of Figure 4.2a.

Original: For those cases, adding unlabelled data with the higher or the same quality 2 improves the performance, whereas the lower quality unlabelled and the same quality group we are simply adding
Revised: For those cases, adding unlabelled data with the higher or the same quality 2 improves the performance, whereas the lower quality unlabelled and the same quality group we are simply adding

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: data is not significantly helpful.
Revised: data is not significantly helpful.

Original: The performance of SS-CQL follows the same trends, see the bottom panels of Figure 4.2a.
Revised: The performance of SS-CQL follows the same trends, see the bottom panels of Figure 4.2a.

Original: To summarize, the experiments provide strong evidence that when the labelled data is of low or moderate quality, SS-ORL is capable to exploit the high quality unlabelled data and remarkably boosts the performance compared with the baselines.
Revised: To summarize, the experiments provide strong evidence that when the labelled data is of low or moderate quality, SS-ORL is capable to exploit the high quality unlabelled data and remarkably boosts the performance compared with the baselines.

Original: The resulting performance is close to that of the oracle agent, and is often optimal (at least 1 ) or near-optimal (close to 1 ).
Revised: The resulting performance is close to that of the oracle agent, and is often optimal (at least 1 ) or near-optimal (close to 1 ).

Original: Quality of Labelled Data Similarly, we fixed the unlabelled data quality and vary the quality of the labelled data.
Revised: Quality of Labelled Data Similarly, we fixed the unlabelled data quality and vary the quality of the labelled data.

Original: Figure 4.2b shows the results.
Revised: Figure 4.2b shows the results.

Original: For both SS-DT and SS-CQL , increasing the labelled data quality raises the performance for all the cases.
Revised: For both SS-DT and SS-CQL , increasing the labelled data quality raises the performance for all the cases.

Original: Size of Labelled Data We train SS-ORL agents where we fix the number of unlabelled trajectories to be 10% of the total number of offline trajectories, and vary the number of labelled trajectories as 10% , 25% , and 50% of the total size.
Revised: Size of Labelled Data We train SS-ORL agents where we fix the number of unlabelled trajectories to be 10% of the total number of offline trajectories, and vary the number of labelled trajectories as 10% , 25% , and 50% of the total size.

Original: Similar to the above experiments, we consider four data quality setups, where the labelled and unlabelled trajectories are sampled from the bottom half (denoted by L ) and top half (denoted by T ) trajectories, respectively.
Revised: Similar to the above experiments, we consider four data quality setups, where the labelled and unlabelled trajectories are sampled from the bottom half (denoted by L ) and top half (denoted by T ) trajectories, respectively.

Original: We consider both hopper - medium-expert and walker - medium-expert datasets.
Revised: We consider both hopper - medium-expert and walker - medium-expert datasets.

Original: To take account of different environments and data setups, we report the 95% stratified bootstrap confidence intervals (CIs) of the interquartile mean 3 of the return for all these cases and training instances (Agarwal et al., 2021).
Revised: To take account of different environments and data setups, we report the 95% stratified bootstrap confidence intervals (CIs) of the interquartile mean 3 of the return for all these cases and training instances (Agarwal et al., 2021).

Original: We use 50000 bootstrap replications to generate the CIs.
Revised: We use 50000 bootstrap replications to generate the CIs.

Original: Compared with some other statistics like the mean or the median, the IQM is robust to outliers and also a good representative of the overall performance.
Revised: Compared with some other statistics like the mean or the median, the IQM is robust to outliers and also a good representative of the overall performance.

Original: The stratified bootstrapping is a handy tool to obtain CIs with descent coverage rate, even if one only have a small number of training instances per setup.
Revised: The stratified bootstrapping is a handy tool to obtain CIs with descent coverage rate, even if one only have a small number of training instances per setup.

Original: We refer the readers to Agarwal et al.
Revised: We refer the readers to Agarwal et al.

Original: (2021) for the complete introduction.
Revised: (2021) for the complete introduction.

Original: Figure 4.3a plots the confidence intervals when we consider all four quality setups, or when the labelled data quality is low or high, respectively.
Revised: Figure 4.3a plots the confidence intervals when we consider all four quality setups, or when the labelled data quality is low or high, respectively.

Original: We found that SS-DT and SS-CQL respond slightly differently.
Revised: We found that SS-DT and SS-CQL respond slightly differently.

Original: Overall, SS-CQL is almost immune to changes in the size of the labelled data, as is
Revised: Overall, SS-CQL is almost immune to changes in the size of the labelled data, as is

Original: SS-DT when the labelled data quality is high.
Revised: SS-DT when the labelled data quality is high.

Original: However, SS-DT ’s performance moderately increases as the labelled size grows when the labelled data quality is low.
Revised: However, SS-DT ’s performance moderately increases as the labelled size grows when the labelled data quality is low.

Original: More detailed results, including the plots of the evaluation curves, CIs of the mean and the median, can be found in Appendix D.
Revised: More detailed results, including the plots of the evaluation curves, CIs of the mean and the median, can be found in Appendix D.

Original: Size of Unlabelled Data As before, we vary the number of labelled data size with the unlabelled data size fixed, and report the 95% stratified bootstrap CIs in Figure 4.3b.
Revised: Size of Unlabelled Data As before, we vary the number of labelled data size with the unlabelled data size fixed, and report the 95% stratified bootstrap CIs in Figure 4.3b.

Original: Similarly, SS-CQL is almost insensitive whereas SS-DT is sensitive when the labelled quality is low.
Revised: Similarly, SS-CQL is almost insensitive whereas SS-DT is sensitive when the labelled quality is low.

Original: Value-based vs. Conditional BC As discussed above, SS-CQL is insensitive to the data size changes, whereas SS-DT is more responsive when the labelled data quality is low.
Revised: Value-based vs. Conditional BC As discussed above, SS-CQL is insensitive to the data size changes, whereas SS-DT is more responsive when the labelled data quality is low.

Original: Regarding the data quality, we are mostly interested in the scenarios where the labelled data quality is low or moderate, see the red and blue curves of Figure 4.2b.
Revised: Regarding the data quality, we are mostly interested in the scenarios where the labelled data quality is low or moderate, see the red and blue curves of Figure 4.2b.

Original: In that regime, if the unlabelled data quality is high (the left column), the distribution shift from the labelled data to the unlabelled data is challenging to handle, and the proxy-actions predicted by the IDM will be less accurate.
Revised: In that regime, if the unlabelled data quality is high (the left column), the distribution shift from the labelled data to the unlabelled data is challenging to handle, and the proxy-actions predicted by the IDM will be less accurate.

Original: There, the absolute performance of SS-CQL is slightly better than SS-DT , with smaller performance gaps compared to the oracle agents.
Revised: There, the absolute performance of SS-CQL is slightly better than SS-DT , with smaller performance gaps compared to the oracle agents.

Original: If the unlabelled data quality is moderate or low (the middle and right columns),
Revised: If the unlabelled data quality is moderate or low (the middle and right columns),

Original: SS-CQL clearly outperforms SS-DT .
Revised: SS-CQL clearly outperforms SS-DT .

Original: Both observations suggest that SS-CQL is less sensitive to the action quality.
Revised: Both observations suggest that SS-CQL is less sensitive to the action quality.

Original: 4. D ESIGN C HOICES
Revised: 4. D ESIGN C HOICES

Original: [Figure]
Revised: [Figure]

Original: Transition Size k of the IDM We train SS-TD3BC , SS-CQL and
Revised: Transition Size k of the IDM We train SS-TD3BC , SS-CQL and

Original: SS-DT agents on the hopper - medium-expert dataset, under the coupled setup as in Section 4.1.
Revised: SS-DT agents on the hopper - medium-expert dataset, under the coupled setup as in Section 4.1.

Original: We consider 6 different values of q : 10 , 30 , 50 , 70 , 90 and 100 .
Revised: We consider 6 different values of q : 10 , 30 , 50 , 70 , 90 and 100 .

Original: For all the 18 setups ( 3 SS-ORL agents anddifferent q values), we train the agents using the multi-transition IDM with k “ 0 , 1 , 2 respectively.
Revised: For all the 18 setups ( 3 SS-ORL agents anddifferent q values), we train the agents using the multi-transition IDM with k “ 0 , 1 , 2 respectively.

Original: As in the previous section, Figure E.1 plots the 95% stratified bootstrapped CIs for the IQM return across all the setups and training instances, which are generated by 50000 bootstrap replications.
Revised: As in the previous section, Figure E.1 plots the 95% stratified bootstrapped CIs for the IQM return across all the setups and training instances, which are generated by 50000 bootstrap replications.

Original: The results favors the choice k “ 1 .
Revised: The results favors the choice k “ 1 .

Original: See Appendix E for more experiment details, such as the average return for each setup and the CIs for the return mean.
Revised: See Appendix E for more experiment details, such as the average return for each setup and the CIs for the return mean.

Original: [Figure]
Revised: [Figure]

Original: Data Augmentation Strategy As discussed in Section 3.2, we consider variants of SS-TD3BC and SS-DT using uncertainty based data augmentation.
Revised: Data Augmentation Strategy As discussed in Section 3.2, we consider variants of SS-TD3BC and SS-DT using uncertainty based data augmentation.

Original: Following Lakshminarayanan et al.
Revised: Following Lakshminarayanan et al.

Original: (2017), we train an ensemble of 3 independent IDMs on T labelled .
Revised: (2017), we train an ensemble of 3 independent IDMs on T labelled .

Original: We generate the proxy actions for the unlabelled trajectories using the combined model, and also estimate the predictive uncertainties.
Revised: We generate the proxy actions for the unlabelled trajectories using the combined model, and also estimate the predictive uncertainties.

Original: We then only add proxy-labelled data whose uncertainties are below p % to the final RL training dataset.
Revised: We then only add proxy-labelled data whose uncertainties are below p % to the final RL training dataset.

Original: Specifically, we test 4 values of p : 25 , 50 , 75 and 95 .
Revised: Specifically, we test 4 values of p : 25 , 50 , 75 and 95 .

Original: We compare the results with standard SS-TD3BC and SS-DT where all the proxy-labelled data are added into the final RL training dataset.
Revised: We compare the results with standard SS-TD3BC and SS-DT where all the proxy-labelled data are added into the final RL training dataset.

Original: Again, we consider both the hopper - medium-expert and walker - medium-expert datasets and use the coupled setup under with 4 different q values: 10 , 30 , 68 , 75 for hopper - medium-expert and 10 , 30 , 54 , 60 for walker - medium-expert .
Revised: Again, we consider both the hopper - medium-expert and walker - medium-expert datasets and use the coupled setup under with 4 different q values: 10 , 30 , 68 , 75 for hopper - medium-expert and 10 , 30 , 54 , 60 for walker - medium-expert .

Original: Figure 4.5 plots the 95% stratified bootstrap CIs of the IQM return across all the setups.
Revised: Figure 4.5 plots the 95% stratified bootstrap CIs of the IQM return across all the setups.

Original: Adding all the proxy-labelled data without filtering outperforms uncertainty based data augmentation; see Appendix F for more details.
Revised: Adding all the proxy-labelled data without filtering outperforms uncertainty based data augmentation; see Appendix F for more details.

Original: Intuitively, to make use of the unlabelled data, most of the SSL pipelines would assume
Revised: Intuitively, to make use of the unlabelled data, most of the SSL pipelines would assume

Original: P labelled and P unlabelled are similar or even the same (Chapelle et al., 2006).
Revised: P labelled and P unlabelled are similar or even the same (Chapelle et al., 2006).

Original: This is not the case in our setup, where P labelled only generates low return trajectories, and all the high return ones come from
Revised: This is not the case in our setup, where P labelled only generates low return trajectories, and all the high return ones come from

Original: P unlabelled .
Revised: P unlabelled .

Original: It remains an open question if self-training with the uncertainty based selection rule can help us generalize to high return trajectories.
Revised: It remains an open question if self-training with the uncertainty based selection rule can help us generalize to high return trajectories.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposed a novel setup for offline RL where the trajectories do not have all of the action information, for which we have introduced a semi-supervised meta-algorithmic pipeline.
Revised: We proposed a novel setup for offline RL where the trajectories do not have all of the action information, for which we have introduced a semi-supervised meta-algorithmic pipeline.

Original: Our experiments identified key properties that enable the agents to learn from unlabelled data and show that near-optimal learning can be done with only 10% of the actions labelled for low-to-moderate quality trajectories.
Revised: Our experiments identified key properties that enable the agents to learn from unlabelled data and show that near-optimal learning can be done with only 10% of the actions labelled for low-to-moderate quality trajectories.

Original: It would be interesting to study other heterogeneous data setups for offline RL in the future, including reward-free or pure state-only settings.
Revised: It would be interesting to study other heterogeneous data setups for offline RL in the future, including reward-free or pure state-only settings.

Original: This work is a step towards a broader goal of empowering robotic systems with the ability to extract meaningful knowledge from copious and ever-growing amounts of unlabelled demonstration data.
Revised: This work is a step towards a broader goal of empowering robotic systems with the ability to extract meaningful knowledge from copious and ever-growing amounts of unlabelled demonstration data.

Original: Beyond simply not having the action labels, many trajectories may be from different robotic systems or tasks and therefore are not directly transferable to the system and task at hand.
Revised: Beyond simply not having the action labels, many trajectories may be from different robotic systems or tasks and therefore are not directly transferable to the system and task at hand.

Original: As we continue building robotic systems to leverage these forms of auxiliary knowledge, we expect that weakly-supervised learning paradigms such as the one explored in this work will be useful.
Revised: As we continue building robotic systems to leverage these forms of auxiliary knowledge, we expect that weakly-supervised learning paradigms such as the one explored in this work will be useful.

Original: A E XPERIMENT D ETAILS
Revised: A E XPERIMENT D ETAILS

Original: In this section, we provide more details about our experiments.
Revised: In this section, we provide more details about our experiments.

Original: For all the offline RL methods we consider, we use our own implementations adopted from the following codebases:
Revised: For all the offline RL methods we consider, we use our own implementations adopted from the following codebases:

Original: DT https://github.com/facebookresearch/online-dt
Revised: DT https://github.com/facebookresearch/online-dt

Original: TD3BC https://github.com/sfujim/TD3_BC
Revised: TD3BC https://github.com/sfujim/TD3_BC

Original: CQL https://github.com/scottemmons/youngs-cql
Revised: CQL https://github.com/scottemmons/youngs-cql

Original: We use the stochastic DT proposed by Zheng et al.
Revised: We use the stochastic DT proposed by Zheng et al.

Original: For offline RL, its performance is similar to the deterministic DT (Chen et al., 2021).
Revised: For offline RL, its performance is similar to the deterministic DT (Chen et al., 2021).

Original: The policy parameter is optimized by the LAMB optimizer (You et al., 2019) with ε “ 10 ´ 8 .
Revised: The policy parameter is optimized by the LAMB optimizer (You et al., 2019) with ε “ 10 ´ 8 .

Original: The log-temperature parameter is optimized by the
Revised: The log-temperature parameter is optimized by the

Original: Adam optimzier (Kingma & Ba, 2014).
Revised: Adam optimzier (Kingma & Ba, 2014).

Original: The architecture and other hyperparameters are listed in
Revised: The architecture and other hyperparameters are listed in

Original: Tabel A.1.
Revised: Tabel A.1.

Original: For TD3BC, we optimize both the critic and actor parameters by the Adam optimizer.
Revised: For TD3BC, we optimize both the critic and actor parameters by the Adam optimizer.

Original: The complete hyperparameters are listed in Table A.2.
Revised: The complete hyperparameters are listed in Table A.2.

Original: For CQL, we also use the Adam optimizer to optimize the critic, actor and the log-temperature parameters.
Revised: For CQL, we also use the Adam optimizer to optimize the critic, actor and the log-temperature parameters.

Original: The architecture of critic and actor networks and the other hyperparameters are listed in Table A.3.
Revised: The architecture of critic and actor networks and the other hyperparameters are listed in Table A.3.

Original: We use batch size 256 and context length 20 for DT, where each batch contains 5120 states.
Revised: We use batch size 256 and context length 20 for DT, where each batch contains 5120 states.

Original: Correspondingly, we use batch size 5120 for CQL and TD3BC.
Revised: Correspondingly, we use batch size 5120 for CQL and TD3BC.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B T HE R ETURN D ISTRIBUTIONS OF THE D4RL D ATASETS
Revised: B T HE R ETURN D ISTRIBUTIONS OF THE D4RL D ATASETS

Original: [Figure]
Revised: [Figure]

Original: C A DDITIONAL E XPERIMENTS U NDER THE C OUPLED S ETUP
Revised: C A DDITIONAL E XPERIMENTS U NDER THE C OUPLED S ETUP

Original: We conduct experiments on the medium and medium-replay datasets of D4RL benchmark, using the same setup as in Section 4.1.
Revised: We conduct experiments on the medium and medium-replay datasets of D4RL benchmark, using the same setup as in Section 4.1.

Original: Figure C.1 and C.2 reports the results.
Revised: Figure C.1 and C.2 reports the results.

Original: The general trend is as the same as that in Figure 4.1.
Revised: The general trend is as the same as that in Figure 4.1.

Original: We note that the results on the halfcheetah - medium dataset are less informative than the others.
Revised: We note that the results on the halfcheetah - medium dataset are less informative than the others.

Original: This is because the data distributions of halfcheetah - medium is very concentrated, similar to a Gaussian distribution with small variance, see Figure B.1.
Revised: This is because the data distributions of halfcheetah - medium is very concentrated, similar to a Gaussian distribution with small variance, see Figure B.1.

Original: In such a case, varying the value of q does not drastically change the labelled data distribution.
Revised: In such a case, varying the value of q does not drastically change the labelled data distribution.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: dataset
Revised: dataset

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: u

Original: [Figure]
Revised: [Figure]

Original: One may notice that for the hopper - medium-replay and walker - medium-replay datasets,
Revised: One may notice that for the hopper - medium-replay and walker - medium-replay datasets,

Original: SS-ORL does not catch up with the oracle as quickly as on the other datasets as q increases.
Revised: SS-ORL does not catch up with the oracle as quickly as on the other datasets as q increases.

Original: Our intuition is that the return distributions of these two datasets concentrate on extremely low values, as shown in Figure B.1.
Revised: Our intuition is that the return distributions of these two datasets concentrate on extremely low values, as shown in Figure B.1.

Original: In our experiments, the labelled trajectories for those two datasets have average return small than 0 .
Revised: In our experiments, the labelled trajectories for those two datasets have average return small than 0 .

Original: even when q “ 70 .
Revised: even when q “ 70 .

Original: In contrast, the return distributions of the other datasets concentrate on larger values.
Revised: In contrast, the return distributions of the other datasets concentrate on larger values.

Original: For the halfcheetah - medium-replay and all the medium and medium-expert datasets, increasing the value of q will greatly change the returns of labelled trajectories, see Table C.1.
Revised: For the halfcheetah - medium-replay and all the medium and medium-expert datasets, increasing the value of q will greatly change the returns of labelled trajectories, see Table C.1.

Original: To demonstrate the performance of SS-ORL on dataset with a more wide return distribution, we consider a subsampled dataset for the walker environment generated as follows.
Revised: To demonstrate the performance of SS-ORL on dataset with a more wide return distribution, we consider a subsampled dataset for the walker environment generated as follows.

Original: Combine the walker - medium-replay and walker - medium datasets.
Revised: Combine the walker - medium-replay and walker - medium datasets.

Original: Let R min and R max denote the minimum and maximum return in the dataset.
Revised: Let R min and R max denote the minimum and maximum return in the dataset.

Original: We divide the trajectories into 40 bins, where the maximum returns within each bin are linear spaced between R min and R max .
Revised: We divide the trajectories into 40 bins, where the maximum returns within each bin are linear spaced between R min and R max .

Original: Let n i be the number trajectories in bin i .
Revised: Let n i be the number trajectories in bin i .

Original: We randomly sample
Revised: We randomly sample

Original: 1000 trajectories.
Revised: 1000 trajectories.

Original: To sample a trajectory, we first first sample a bin i P r 1 , . .
Revised: To sample a trajectory, we first first sample a bin i P r 1 , . .

Original: , 40 s with weights proportional to 1 { n i , then sample a trajectory uniformly at random from the sampled bin.
Revised: , 40 s with weights proportional to 1 { n i , then sample a trajectory uniformly at random from the sampled bin.

Original: Figure C.3 plots the return distribution of the subsampled dataset.
Revised: Figure C.3 plots the return distribution of the subsampled dataset.

Original: It is wide and has 3 modes.
Revised: It is wide and has 3 modes.

Original: We run the same experiments as before on this subsampled dataset, and Figure C.4 plots the results.
Revised: We run the same experiments as before on this subsampled dataset, and Figure C.4 plots the results.

Original: We can see that SS-ORL methods can catch up with the oracle agents even when q is small.
Revised: We can see that SS-ORL methods can catch up with the oracle agents even when q is small.

Original: D I NFLUENCES OF THE L ABELLED AND U NLABELLED D ATA S IZE
Revised: D I NFLUENCES OF THE L ABELLED AND U NLABELLED D ATA S IZE

Original: Figure D.1 plots the average return of SS-DT and SS-CQL when we fix the number of unlabelled trajectories and vary the number of labelled trajectories.
Revised: Figure D.1 plots the average return of SS-DT and SS-CQL when we fix the number of unlabelled trajectories and vary the number of labelled trajectories.

Original: We found that there is a bad seed for
Revised: We found that there is a bad seed for

Original: SS-CQL when both labelled and unlabelled trajectories are sampled from L and the size of labelled data is 10% , so that the result there the bottom right panel) exhibits large variance.
Revised: SS-CQL when both labelled and unlabelled trajectories are sampled from L and the size of labelled data is 10% , so that the result there the bottom right panel) exhibits large variance.

Original: Correspondingly,
Revised: Correspondingly,

Original: Figure D.2 plots the 95% stratified bootstrap CIs for the median, mean, interquartile mean, and the optimality gap of the return of SS-DT and SS-CQL .
Revised: Figure D.2 plots the 95% stratified bootstrap CIs for the median, mean, interquartile mean, and the optimality gap of the return of SS-DT and SS-CQL .

Original: Similarly, Figure D.3 and D.4 plots the results when we vary the number of unlabelled trajectories, while the number of unlabelled ones is fixed.
Revised: Similarly, Figure D.3 and D.4 plots the results when we vary the number of unlabelled trajectories, while the number of unlabelled ones is fixed.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: (a) Results of SS-DT .
Revised: (a) Results of SS-DT .

Original: [Figure]
Revised: [Figure]

Original: E T RANSITION S IZE k F OR THE M ULTI - TRANSITION I NVERSE D YNAMIC M ODEL E. T HEORY
Revised: E T RANSITION S IZE k F OR THE M ULTI - TRANSITION I NVERSE D YNAMIC M ODEL E. T HEORY

Original: Let β denote the behaviour policy.
Revised: Let β denote the behaviour policy.

Original: When k “ 0 , the IDM is modeling
Revised: When k “ 0 , the IDM is modeling

Original: [Equation]
Revised: [Equation]

Original: For the cases where k ą 0 , w.l.o.g, we assume k “ 1 .
Revised: For the cases where k ą 0 , w.l.o.g, we assume k “ 1 .

Original: The IDM is modeling
Revised: The IDM is modeling

Original: [Equation]
Revised: [Equation]

Original: where in the last line we used the fact that the policy β can only generate actions based on previous states, the Markovian transition property P p s t ` 1 | a t , s t , s t ` 2 , s t ´ 1 q “ P p s t ` 1 | a t , s t q , and also the induced property P p s t ` 1 | s t , s t ´ 1 q “ P p s t ` 1 | s t ` 2 , s t , s t ´ 1 q .
Revised: where in the last line we used the fact that the policy β can only generate actions based on previous states, the Markovian transition property P p s t ` 1 | a t , s t , s t ` 2 , s t ´ 1 q “ P p s t ` 1 | a t , s t q , and also the induced property P p s t ` 1 | s t , s t ´ 1 q “ P p s t ` 1 | s t ` 2 , s t , s t ´ 1 q .

Original: If the behaviour policy β is Markovian, we have that β p a t | s t q “ β p a t | s t , s t ´ 1 q , and as a consequence
Revised: If the behaviour policy β is Markovian, we have that β p a t | s t q “ β p a t | s t , s t ´ 1 q , and as a consequence

Original: [Equation]
Revised: [Equation]

Original: where C “
Revised: where C “

Original: P p s t ` 1 | s t q P p s t ` 1 | s t ,s t ´ 1 q is independent of the action a t .
Revised: P p s t ` 1 | s t q P p s t ` 1 | s t ,s t ´ 1 q is independent of the action a t .

Original: Therefore, the probabilities that the IDMs with k “ 0 and k “ 1 are modeling, are equivalent up to a state-only dependent scaling.
Revised: Therefore, the probabilities that the IDMs with k “ 0 and k “ 1 are modeling, are equivalent up to a state-only dependent scaling.

Original: The cases where k ě 2 can be derived analogously.
Revised: The cases where k ě 2 can be derived analogously.

Original: In practice, the offline dataset might contain trajectories generated by multiple behaviour policies and it is unknown if any of them is Markovian.
Revised: In practice, the offline dataset might contain trajectories generated by multiple behaviour policies and it is unknown if any of them is Markovian.

Original: Therefore, choosing k ą 0 allows us to take into account past information before timestep t .
Revised: Therefore, choosing k ą 0 allows us to take into account past information before timestep t .

Original: For future, we do not need anything beyond s t ` 1 for a MDP, but our formulation is general purpose to account for POMDP as well, where both past and future partial observations might be needed to infer the action a t .
Revised: For future, we do not need anything beyond s t ` 1 for a MDP, but our formulation is general purpose to account for POMDP as well, where both past and future partial observations might be needed to infer the action a t .

Original: To summarize, choosing k ě 0 is more general and have been shown to be favorable in the empirical experiments presented in next section.
Revised: To summarize, choosing k ě 0 is more general and have been shown to be favorable in the empirical experiments presented in next section.

Original: E. E MPIRICAL E XPERIMENTS
Revised: E. E MPIRICAL E XPERIMENTS

Original: We train SS-TD3BC , SS-CQL and SS-DT with 3 IDM transition size: k = 0 , 1 and 2 on the hopper - medium-expert dataset.
Revised: We train SS-TD3BC , SS-CQL and SS-DT with 3 IDM transition size: k = 0 , 1 and 2 on the hopper - medium-expert dataset.

Original: We use the coupled setup described in Section 4.1, withdifferent values of q .
Revised: We use the coupled setup described in Section 4.1, withdifferent values of q .

Original: Table E.1 reports the performance of those agents for each case.
Revised: Table E.1 reports the performance of those agents for each case.

Original: In addition to the interquartile mean considered in Section 4.3, we also consider 3 other statistics of the return across all the setups: the mean, the median and the optimality gap.
Revised: In addition to the interquartile mean considered in Section 4.3, we also consider 3 other statistics of the return across all the setups: the mean, the median and the optimality gap.

Original: Figure 4.3 plots the 95% stratified bootstrap confidence intervals for all the four statistics, genereated by 50000 bootstrap replications.
Revised: Figure 4.3 plots the 95% stratified bootstrap confidence intervals for all the four statistics, genereated by 50000 bootstrap replications.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: F D ATA A UGMENTATION S TRATEGY
Revised: F D ATA A UGMENTATION S TRATEGY

Original: Following Lakshminarayanan et al.
Revised: Following Lakshminarayanan et al.

Original: (2017), we train an ensemble of 3 independent IDMs on T labelled .
Revised: (2017), we train an ensemble of 3 independent IDMs on T labelled .

Original: Each individual IDM models the action as a diagonal Gaussian distribution (see Equation (1))
Revised: Each individual IDM models the action as a diagonal Gaussian distribution (see Equation (1))

Original: N p µ i , Σ
Revised: N p µ i , Σ

Original: i q , i “ 1 , 2 , 3 .
Revised: i q , i “ 1 , 2 , 3 .

Original: The ensemble models the action using a equally weighted Gaussian mixture of these three distributions.
Revised: The ensemble models the action using a equally weighted Gaussian mixture of these three distributions.

Original: We predict the action by the mixture
Revised: We predict the action by the mixture

Original: ’s mean and predict the uncertainty by the mixture’s variance; both can be written in close form.
Revised: ’s mean and predict the uncertainty by the mixture’s variance; both can be written in close form.

Original: We conduct experiments for SS-DT and SS-TD3BC , where we only add proxy-labelled data whose uncertainties are below p % to the final RL training dataset.
Revised: We conduct experiments for SS-DT and SS-TD3BC , where we only add proxy-labelled data whose uncertainties are below p % to the final RL training dataset.

Original: Specifically, we test 4 values of p : 25 , 50 , 75 and 95 .:w
Revised: Specifically, we test 4 values of p : 25 , 50 , 75 and 95 .:w

Original: We compare the results with standard SS-DT and SS-TD3BC where all the proxy-labelled data are added into the final RL training dataset.
Revised: We compare the results with standard SS-DT and SS-TD3BC where all the proxy-labelled data are added into the final RL training dataset.

Original: We consider both the hopper - medium-expert and walker - medium-expert datasets.
Revised: We consider both the hopper - medium-expert and walker - medium-expert datasets.

Original: We use the coupled setup described in Section 4.1, where we consider 4 different values of q : 10 , 30 , 68 , 75 for hoppermedium-expert and 10 , 30 , 54 , 60 for walker - medium-expert .
Revised: We use the coupled setup described in Section 4.1, where we consider 4 different values of q : 10 , 30 , 68 , 75 for hoppermedium-expert and 10 , 30 , 54 , 60 for walker - medium-expert .

Original: Table F.1 reports the average return and standard deviation obtained by SS-DT and SS-TD3BC under different data augmentation strategies, when trained on the hopper - medium-expert dataset.
Revised: Table F.1 reports the average return and standard deviation obtained by SS-DT and SS-TD3BC under different data augmentation strategies, when trained on the hopper - medium-expert dataset.

Original: The results on the walker - medium-expert dataset are reported in Table F.2.
Revised: The results on the walker - medium-expert dataset are reported in Table F.2.

Original: It is easy to see that uncertainty based data augmentation degrades the performance, compared with adding all the proxylabelled data without filtering.
Revised: It is easy to see that uncertainty based data augmentation degrades the performance, compared with adding all the proxylabelled data without filtering.

Original: Overall, the latter performs consistently well across different setups.
Revised: Overall, the latter performs consistently well across different setups.

Original: Figure F.1 plots the 95% stratified bootstrap CIs for this experiments.
Revised: Figure F.1 plots the 95% stratified bootstrap CIs for this experiments.

Original: All the statistics favor the no filtering strategy.
Revised: All the statistics favor the no filtering strategy.

Original: [Table]
Revised: [Table]

Original: standard deviation) of SS-ORL agents trained on the hoppermedium-expert dataset under the coupled setup, using different data augmentation strategies.
Revised: standard deviation) of SS-ORL agents trained on the hoppermedium-expert dataset under the coupled setup, using different data augmentation strategies.

Original: Results aggregated over 5 training instances.
Revised: Results aggregated over 5 training instances.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: G C OMPARISON WITH GATO UNDER THE C OUPLED S ETUP
Revised: G C OMPARISON WITH GATO UNDER THE C OUPLED S ETUP

Original: Inspired the multi-task and multi-modal generalist agent proposed by Reed et al.
Revised: Inspired the multi-task and multi-modal generalist agent proposed by Reed et al.

Original: (2022), we consider a GATO type of variant of DT that can incorporate the unlabelled data into policy training.
Revised: (2022), we consider a GATO type of variant of DT that can incorporate the unlabelled data into policy training.

Original: GATO is trained on the labelled and unlabelled data simultaneously.
Revised: GATO is trained on the labelled and unlabelled data simultaneously.

Original: The implementation details are:
Revised: The implementation details are:

Original: • We form the same input sequence as DT , where we fill in zeros for the missing actions for unlabelled trajectories.
Revised: • We form the same input sequence as DT , where we fill in zeros for the missing actions for unlabelled trajectories.

Original: • For the labelled trajectories, GATO predicts the actions, states and rewards; for the unlabelled ones, GATO only predicts the states and rewards.
Revised: • For the labelled trajectories, GATO predicts the actions, states and rewards; for the unlabelled ones, GATO only predicts the states and rewards.

Original: • We use the stochastic policy as in online decision transformer (Zheng et al., 2022) to predict the actions.
Revised: • We use the stochastic policy as in online decision transformer (Zheng et al., 2022) to predict the actions.

Original: • We use deterministic predictors for the states and rewards, which are single linear layers built on top of the Transformer outputs.
Revised: • We use deterministic predictors for the states and rewards, which are single linear layers built on top of the Transformer outputs.

Original: Let g t “ ř | τ | r t 1 t 1 “ t be the return-to-go of a trajectory τ at timestep t .
Revised: Let g t “ ř | τ | r t 1 t 1 “ t be the return-to-go of a trajectory τ at timestep t .

Original: Let H P labelled θ denotes the policy entropy included on the labelled data distribution.
Revised: Let H P labelled θ denotes the policy entropy included on the labelled data distribution.

Original: For simplicity, we assume the context length of
Revised: For simplicity, we assume the context length of

Original: GATO is 1 .
Revised: GATO is 1 .

Original: We refer the readers to Zheng et al.
Revised: We refer the readers to Zheng et al.

Original: for the formulation with a general context length and more details.
Revised: for the formulation with a general context length and more details.

Original: Equation (5) shows the training objective of GATO .
Revised: Equation (5) shows the training objective of GATO .

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: we use significantly outperforms
Revised: we use significantly outperforms

Original: GATO only slightly improves
Revised: GATO only slightly improves

Original: [Figure]
Revised: [Figure]

Original: parameters.
Revised: parameters.

Original: The constant ν , λ s and λ r are prefixed hyper-parameters, where ν is the target policy entropy, and λ s and λ r are regularization parameters used to balance the losses for actions, states, and rewards.
Revised: The constant ν , λ s and λ r are prefixed hyper-parameters, where ν is the target policy entropy, and λ s and λ r are regularization parameters used to balance the losses for actions, states, and rewards.

Original: We use ν “ ´ dim p A q as for DT (see Appendix A).
Revised: We use ν “ ´ dim p A q as for DT (see Appendix A).

Original: To choose the regularization parameters λ s and λ r for GATO , we test 16 combinations where λ s and λ r are 1 . 0 , 0 . 1 , 0 . 01 and 0 .
Revised: To choose the regularization parameters λ s and λ r for GATO , we test 16 combinations where λ s and λ r are 1 . 0 , 0 . 1 , 0 . 01 and 0 .

Original: 001 respectively.
Revised: 001 respectively.

Original: We run experiments as in Section 4.1 for q “ 10 , 30 , 50 , 70 , 90 , 100 , and compute the confidence intervals for the aggregated results.
Revised: We run experiments as in Section 4.1 for q “ 10 , 30 , 50 , 70 , 90 , 100 , and compute the confidence intervals for the aggregated results.

Original: Figure G.2 shows that λ s “ 0 . 01 and λ r “ 0 . 1 yield the best performance.
Revised: Figure G.2 shows that λ s “ 0 . 01 and λ r “ 0 . 1 yield the best performance.

Original: Figure G.1 compares the performance of GATO (with λ s “ 0 . 01 and λ r “ 0 . 1 ) and SS-ORL agents.
Revised: Figure G.1 compares the performance of GATO (with λ s “ 0 . 01 and λ r “ 0 . 1 ) and SS-ORL agents.

Original: It is clear that SS-ORL agents outperform GATO .
Revised: It is clear that SS-ORL agents outperform GATO .

Original: H P ERFORMANCE G AP OF SS-ORL A GENTS
Revised: H P ERFORMANCE G AP OF SS-ORL A GENTS

Original: For a chosen offline RL method, the relative performance gap between the corresponding SS-ORL and oracle agents illustrates how sensitive this offline RL is to missing actions:
Revised: For a chosen offline RL method, the relative performance gap between the corresponding SS-ORL and oracle agents illustrates how sensitive this offline RL is to missing actions:

Original: [Equation]
Revised: [Equation]

Original: We consider the coupled setup as in Section 4.1.
Revised: We consider the coupled setup as in Section 4.1.

Original: For each of the 9 datasets ( hopper , walker , halfcheetah with medium-expert , medium , and medium-replay datasets), we compute the relative performance gap for SS-CQL , SS-DT and SS-TD3BC , trained with 6 different values of q : 10 , 30 , 50 , 70 , 90 and 100 .
Revised: For each of the 9 datasets ( hopper , walker , halfcheetah with medium-expert , medium , and medium-replay datasets), we compute the relative performance gap for SS-CQL , SS-DT and SS-TD3BC , trained with 6 different values of q : 10 , 30 , 50 , 70 , 90 and 100 .

Original: Table H.1 reports the aggregate results over seeds.
Revised: Table H.1 reports the aggregate results over seeds.

Original: On average, SS-CQL and SS-TD3BC have smaller relative performance gap, suggesting that
Revised: On average, SS-CQL and SS-TD3BC have smaller relative performance gap, suggesting that

Original: CQL and TD3BC are less sensitive to the missing actions.
Revised: CQL and TD3BC are less sensitive to the missing actions.

Original: [Table]
Revised: [Table]