Original: T HE P ROVABLE B ENEFITS OF U NSUPERVISED D ATA S HARING FOR O FFLINE R EINFORCEMENT L EARNING
Revised: T HE P ROVABLE B ENEFITS OF U NSUPERVISED D ATA S HARING FOR O FFLINE R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised methods play a vital role in fueling the progress of deep learning using supervision from the data itself, reducing the need for expensive annotations.
Revised: Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations.

Original: Like self-supervised methods in other areas of machine learning, we would like methods that can leverage reward-free data to improve the performance ofoffline RL, but it is unclear how to conductsuch self-supervised RL in a principled way.
Revised: However, the question of how to conduct self-supervised ofﬂine reinforcement learning (RL) in a principled way remains unclear.

Original: In this paper, we examine the theoretical benefit of unlabeled data inthe context of linear MDPs and propose a novel and P rovable D ata S haring algorithm, which we refer to as PDS, to utilize such unlabeled data for offline RL.
Revised: In this paper, we address this issue by investigating the theoretical beneﬁts of utilizing reward free data in linear Markov Decision Processes (MDPs) within a semi supervised setting Further, we propose a novel, P rovable D ata S haring algorithm

Original: PDS utilizes additional penalties upon the reward function learned from labeled data to avoid potential overestimation of the reward.
Revised: PDS ) to utilize such reward -free data for ofﬂine RL. PDS uses additional penalties on the reward function learned from labeled data to prevent overestimation, ensuring a conservative algorithm.

Original: We show that such a penalty is crucial to keep the algorithm conservative, and PDS achieves a provable benefit from unlabeled data under mild conditions. We conduct extensive experiments on various offline RL tasks and show that PDS can significantly improve offline RL algorithms with unlabeled data. 
Revised: Our results on various ofﬂine RL tasks demonstrate that PDS signiﬁcantly improves the performance of ofﬂine RL algorithms with reward -free data. Overall, our work provides a promising approach to leveraging the beneﬁts of unlabeled data in of ﬂine RL while maintaining theoretical guarantees

Revised: We believe our ﬁndings will contribute to developing more robust self-supervised

Revised: RL methods

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Offline reinforcement learning (RL) offers a promising framework to learn performant sequential policies with pre-collected datasets, which is highly preferredto online RL in many real-world problems where active data collection and exploration is expensive, unsafe, or infeasible (Swaminathan & Joachims, 2015; Shalev-Shwartz et al., 2016; Singh et al., 2020).
Revised: Ofﬂine reinforcement learning (RL) is a promising framework for learning sequential policies with pre-collected datasets. It is highly preferred in many real-world problems where active data collection and exploration is expensive, unsafe, or infeasible (Swaminathan & Joachims, 2015; ShalevShwartz et al., 2016; Singh et al., 2020).

Original: However,this approach can still be expensive since labeling large datasets with rewards can be costly and requires huge human efforts (Singh et al., 2019; Wirth et al., 2017).
Revised: However, labeling large datasets with rewards can be costly and require signiﬁcant human effort (Singh et al., 2019; Wirth et al., 2017).

Original: On the contrary, unlabeled data can be cheap and abundant, encouraging us to learn from self-supervisions with unlabeled data.
Revised: In contrast, unlabeled data can be cheap and abundant, making self-supervised learning with unlabeled data an attractive alternative.

Original: Such self-supervised methods have achieved great success in computer vision and natural language processing tasks like Brown et al. Devlin et al. ; Chen et al.
Revised: While self-supervised methods have achieved great success in computer vision and natural language processing tasks (Brown et al., 2020; Devlin et al., 2018; Chen et al., 2020), their potential beneﬁts in ofﬂine RL are less explored.

Original: but it is less discussed in the context of offline reinforcement learning. Some prior works (Ho & Ermon, 2016; Reddy et al., 2019; Kostrikov et al., 2019) have explored demonstration-based approaches, which remove the need for reward annotations but  requirethe samples to be near optimal.
Revised: Several prior works (Ho & Ermon, 2016; Reddy et al., 2019; Kostrikov et al., 2019) have explored demonstration-based approaches to eliminate the need for reward annotations, but these approaches require samples to be near-optimal.

Original: There is also a line of workthat focuses on data sharing from different datasets (Kalashnikov et al., 2021; Yu et al., 2021a) but assumes that the dataset can be relabeled with oracle reward functions for the target task.
Revised: Another line of work focuses on data sharing from different datasets (Kalashnikov et al., 2021; Yu et al., 2021a). Still, it assumes that the dataset can be relabeled with oracle reward functions for the target task.

Original: These assumptions can be strong in real-world problems, whereboth expert trajectories and reward labeling are expensive.
Revised: These settings can be unrealistic in real-world problems where expert trajectories and reward labeling are expensive.

Original: It is important but non-trivial to incorporate unlabeled datasets into offline RL  due to RL problems’ sequential and dynamic nature.
Revised: Incorporating reward-free datasets into ofﬂine RL is important but challenging due to the sequential and dynamic nature of RL problems.

Original: Prior work (Yu et al., 2022) has shown that learning to predictthe rewards can be challenging and that the simple approach of setting the reward to zero can achieve good results.
Revised: Prior work (Yu et al., 2022) has shown that learning to predict rewards can be difﬁcult, and simply setting the reward to zero can achieve good results.

Original: However, it is unclear what affects the performance of reward prediction methods  and whether reward-free data canbe provably beneficial to offline RL, which naturally leads to the following question:
Revised: However, it’s unclear how reward-prediction methods affect performance and whether reward-free data can provably beneﬁt ofﬂine RL. This naturally leads to the following question:

Original: How can we leverage reward-free data to improve the performance of offline RL algorithms in a principled way?
Revised: How can we leverage reward-free data to improve the performance of ofﬂine RL algorithms in a principled way?

Original: To answer this question, we start with the analysis of the theoretical benefit of using unlabeled data in linear MDPs.
Revised: To answer this question, we conduct a theoretical analysis of the beneﬁts of utilizing unlabeled data in linear MDPs.

Original: Intuitively, unlabeled data helps us gain more information about the dynamics of the MDP, leaving only uncertainty over reward functions.As a result, the problem is reduced to a linear bandit problem with uncertainty only on reward functions when the unlabeled data goes to infinity. Based on this analysis, we may develop a model-based approach and add pessimism over models and rewards, respectively, but it can be computationally extensive.
Revised: Our analysis reveals that although unlabeled data can provide information about the dynamics of the MDP, it cannot reduce the uncertainty over reward functions.

Original: To sidestep this issue, we propose a model-free method, which we refer to as provable data sharing (PDS), whichsimply adds uncertainty penaltieson learned rewards tokeep the algorithm conservative.
Revised: Based on this insight, we propose a model-free method named Provable Data Sharing (PDS), which adds uncertainty penalties to the learned reward functions to maintain a conservative algorithm.

Revised: By doing so,

Original:  We show that our method can cooperate with model-free offline methods while being simple and provably efficient. Theoretical analysis shows that our method can improve performance bounds under mild conditions. In practice, our method also achieves superior performance.
Revised: PDS can effectively leverage the beneﬁts of unlabeled data for ofﬂine RL while ensuring theoretical guarantees. We demonstrate that PDS can cooperate with model-free ofﬂine methods while being simple and efﬁcient.

Original: We conduct extensive experiments on various environments, including single-task domains like MuJoCo (Todorov et al., 2012) and kitchen, as well as multi-task domains like AntMaze and Meta-World (Yu et al., 2020a).
Revised: We conduct extensive experiments on various environments, including single-task domains like MuJoCo (Todorov et al., 2012) and Kitchen (Gupta et al., 2019), as well as multi-task domains like AntMaze and Meta-World (Yu et al., 2020a).

Original: Results show that our algorithm improves significantly over previous methods like UDS (Yu et al., 2022) and naive reward prediction methods.
Revised: The results show that PDS improves signiﬁcantly over previous methods like UDS (Yu et al., 2022) and naive reward prediction methods.

Original: Our main contribution is the provable data sharing (PDS) algorithm, a novel method to utilize unsupervised data for offline RL with theoretical guarantees.
Revised: Our main contribution is the Provable Data Sharing (PDS) algorithm, a novel method for utilizing unsupervised data in ofﬂine RL that provides theoretical guarantees.

Original: PDSis a model-free method that adds uncertainty penalties upon learned reward functions and can be easily integrated with existing offline RL algorithms.
Revised: PDS adds uncertainty penalties to the learned reward functions and can be easily integrated with existing ofﬂine RL algorithms.

Original: We show that PDS can achieveprovable improvement in linear MDPs under mild conditions. We also show that our method achieves superior performancein practice with extensive experiments on varioustasks like locomotion, navigation, and manipulation tasks. 
Revised: Our experimental results demonstrate that PDS can achieve superior performance on various locomotion, navigation, and manipulation tasks. Overall, our work provides a promising approach to leveraging the beneﬁts of unlabeled data in ofﬂine RL while maintaining theoretical guarantees and contributing to the development of more robust self-supervised RL methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Offline Reinforcement Learning Current offline RL methods (Levine et al., 2020) can be roughly divided into policy constraint-based, uncertainty estimation-based, and model-based approaches.
Revised: Ofﬂine Reinforcement Learning Current ofﬂine RL methods (Levine et al., 2020) can be roughly divided into policy constraint-based, uncertainty estimation-based, and model-based approaches.

Original: Policy constraint methods aim to keep the policy close to the behavior under a probabilistic distance (Siegel et al., 2020;
Revised: Policy constraint methods aim to keep the policy close to the behavior under a probabilistic distance (Siegel et al., 2020;

Original: Yang et al., 2021; Kumar et al., 2020; Fujimoto & Gu, 2021; Fujimoto et al., 2019; Nair et al., 2020; Hu et al., 2022;Peng et al., 2019; Kostrikov et al., 2021; Ma et al., 2021).
Revised: Yang et al., 2021; Kumar et al., 2020; Fujimoto & Gu, 2021; Yang et al., 2022; Fujimoto et al., 2019; Hu et al., 2022; Kostrikov et al., 2021; Ma et al., 2021).

Original: Uncertainty estimation-based methods attempt to consider the Q-value prediction’s confidence using dropout or ensemble techniques (An et al., 2021; Wu et al., 2021).
Revised: Uncertainty estimation-based methods attempt to consider the Q-value prediction’s conﬁdence using dropout or ensemble techniques (An et al., 2021; Wu et al., 2021).

Original: Last, model-based methods incorporates the uncertainty in the model space for conservative offline learning (Yu et al., 2020b; 2021b; Kidambi et al., 2020).
Revised: Last, model-based methods incorporates the uncertainty in the model space for conservative ofﬂine learning (Yu et al., 2020b; 2021b; Kidambi et al., 2020).

Original: Offline Data Sharing Prior works have demonstrated that data sharing across tasks can be beneficial by designing sophisticated data-sharing protocols (Yu et al., 2021a).
Revised: Ofﬂine Data Sharing Prior works have demonstrated that data sharing across tasks can be beneﬁcial by designing sophisticated data-sharing protocols (Yu et al., 2021a).

Original: For example, previous studies have explored developing data sharing strategies by human effort (Kalashnikov et al., 2021), inverse RL (Reddy et al., 2019; Li et al., 2020), and estimated Q-values (Yu et al., 2021a).
Revised: For example, previous studies have explored developing data sharing strategies by human effort (Kalashnikov et al., 2021), inverse RL (Reddy et al., 2019; Li et al., 2020), and estimated Q-values (Yu et al., 2021a).

Original: However, these data sharing works must assume the dataset can be relabeled with oracle rewards for the target task, which is a strong assumption since the high cost of reward labeling.
Revised: However, these data sharing works must assume the dataset can be relabeled with oracle rewards for the target task, which is a strong assumption since the high cost of reward labeling.

Original: Therefore, effectively incorporating the unlabeled data into the offline RL algorithms is essential.
Revised: Therefore, effectively incorporating the unlabeled data into the ofﬂine RL algorithms is essential.

Original: To solve this issue, some recent work (Yu et al., 2022) proposes simply applying zero rewards to unlabeled data.
Revised: To solve this issue, some recent work (Yu et al., 2022) proposes simply applying zero rewards to unlabeled data.

Original: In this work, we propose a principle way to leverage unlabeled data without the strong assumption of reward relabeling.
Revised: In this work, we propose a principle way to leverage unlabeled data without the strong assumption of reward relabeling.

Original: Reward Prediction It is widely observed that reward shaping and intrinsic rewards can accelerate learning in online RL (Mataric, 1994;
Revised: Reward Prediction It is widely observed that reward shaping and intrinsic rewards can accelerate learning in online RL (Mataric, 1994;

Original: Ng et al., 1999; Wu & Tian, 2017; Song et al., 2019;
Revised: Ng et al., 1999; Wu & Tian, 2017; Song et al., 2019;

Original: Guo et al., 2016; Abel et al., 2021).
Revised: Guo et al., 2016; Abel et al., 2021).

Original: There are also extensive works that studies automatically designing reward functions using inverse RL (Ng et al., 2000; Fu et al., 2017).
Revised: There are also extensive works that studies automatically designing reward functions using inverse RL (Ng et al., 2000; Fu et al., 2017).

Original: However, there is less attention on the offline setting where online interaction is not allowed and the trajectories may not be optimal.
Revised: However, there is less attention on the ofﬂine setting where online interaction is not allowed and the trajectories may not be optimal.

Original:  P RELIMINARIES 3. L INEAR MDP S AND P ERFORMANCE M ETRIC
Revised:  P RELIMINARIES 3. L INEAR MDP S AND P ERFORMANCE M ETRIC

Original: We consider infinite-horizon discounted Markov Decision Processes (MDPs), defined by the tuple ( S , A , P , r, γ ) , with state space S , action space A , discount factor γ ∈ [0 , 1) , transition function
Revised: We consider inﬁnite-horizon discounted Markov Decision Processes (MDPs), deﬁned by the tuple ( S , A , P , r, γ ) , with state space S , action space A , discount factor γ ∈ [0 , 1) , transition function

Original: P : S ×
Revised: P : S ×

Original: A → ∆( S ) , and reward function r : S ×
Revised: A → ∆( S ) , and reward function r : S ×

Original: A → [0 , r max ] .
Revised: A → [0 , r max ] .

Original: To make things more concrete, we consider the linear MDP (Yang & Wang, 2019; Jin et al., 2020) as follows, where the transition kernel and expected reward function are linear with respect to a feature map.
Revised: To make things more concrete, we consider the linear MDP (Yang & Wang, 2019; Jin et al., 2020) as follows, where the transition kernel and expected reward function are linear with respect to a feature map.

Original: Definition 3.1 (Linear MDP) .
Revised: Deﬁnition 3.1 (Linear MDP) .

Original: We say an episodic MDP ( S , A , P , r, γ ) is a linear MDP with known feature map ϕ : S ×
Revised: We say an episodic MDP ( S , A , P , r, γ ) is a linear MDP with known feature map φ : S ×

Original: A → R d if there exist unknown measures µ = ( µ 1 , . . . , µ d ) over S and an unknown vector θ ∈ R d such that
Revised: A → R d if there exist unknown measures µ = ( µ 1 , . . . , µ d ) over S and an unknown vector θ ∈ R d such that

Original: [Equation]
Revised: [Equation]

Original: for all ( s, a, s ′ ) ∈ S ×
Revised: for all ( s, a, s (cid:48) ) ∈ S ×

Original: A × S .
Revised: A × S .

Original: And we assume ∥ ϕ ( s, a ) ∥ 2 ≤ 1 for all ( s, a, s ′ )
Revised: And we assume (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 for all ( s, a, s (cid:48) )

Original: ∈ S ×
Revised: ∈ S ×

Original: A × S and max {∥ µ ( S ) ∥ 2 , ∥ θ ∥ 2 } ≤ √ d , where ∥ µ ( S ) ∥ ≡ (cid:82) S ∥ µ ( s ) 
Revised: A × S and max {(cid:107) µ ( S ) (cid:107) 2 , (cid:107) θ (cid:107) 2 } ≤ √ d , where (cid:107) µ ( S ) (cid:107) ≡ (cid:82) S (cid:107) µ ( s ) (cid:107) ds .

Original: ∥ ds . A policy π : S → ∆( A ) specifies a decision-making strategy in which the agent chooses actions adaptively based on the current state, i.e., a t ∼ π ( · | s t ) .
Revised: A policy π : S → ∆( A ) speciﬁes a decision-making strategy in which the agent chooses actions adaptively based on the current state, i.e., a t ∼ π ( · | s t ) .

Original: The value function V π : S → R and the action-value function (Q-function) Q π :
Revised: The value function V π : S → R and the action-value function (Q-function) Q π :

Original: S ×
Revised: S ×

Original: A → R are defined as
Revised: A → R are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is with respect to the trajectory τ induced by policy π .
Revised: where the expectation is with respect to the trajectory τ induced by policy π .

Original: We define the Bellman operator as
Revised: We deﬁne the Bellman operator as

Original: [Equation]
Revised: [Equation]

Original: We use π ∗ , Q ∗ , and V ∗ to denote the optimal policy, optimal Q-function, and optimal value function, respectively.
Revised: We use π ∗ , Q ∗ , and V ∗ to denote the optimal policy, optimal Q-function, and optimal value function, respectively.

Original: We have the Bellman optimality equation
Revised: We have the Bellman optimality equation

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the optimal policy π ∗ satisfies
Revised: Meanwhile, the optimal policy π ∗ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where the maximum is taken over all functions mapping from S to distributions over A .
Revised: where the maximum is taken over all functions mapping from S to distributions over A .

Original: We aim to learn a policy that maximizes the expected cumulative reward.
Revised: We aim to learn a policy that maximizes the expected cumulative reward.

Original: Correspondingly, we define the performance metric as
Revised: Correspondingly, we deﬁne the performance metric as

Original: [Equation]
Revised: [Equation]

Original: 3. P ROVABLE O FFLINE A LGORITHMS
Revised: 3. P ROVABLE O FFLINE A LGORITHMS

Original: In this section, we consider pessimistic value iteration (PEVI; Jin et al., 2021) as the backbone algorithm, and we provide a detailed description in Algorithm 3 in Appendix A.
Revised: In this section, we consider pessimistic value iteration (PEVI; Jin et al., 2021) as the backbone algorithm, described in Algorithm 2.

Original: It is a representative model-free offline algorithm with theoretical guarantees.
Revised: It is a representative model-free ofﬂine algorithm with theoretical guarantees.

Original: PEVI uses negative bonus Γ( · , · ) over standard Q -value estimation (cid:98) Q ( · , · ) =
Revised: PEVI uses negative bonus Γ( · , · ) over standard Q -value estimation (cid:98) Q ( · , · ) =

Original: ( (cid:98) B (cid:98) V )( · ) to reduce potential bias due to finite data, where (cid:98) B is some empirical estimation of B from dataset D .
Revised: ( (cid:98) B (cid:98) V )( · ) to reduce potential bias due to ﬁnite data, where (cid:98) B is some empirical estimation of B from dataset D .

Original: Please refer to Appendix A for  details of the PEVI algorithm.
Revised: Please refer to Appendix A.1 for more details of the PEVI algorithm.

Original: We use the  notion of ξ -uncertainty quantifier as follows to formalize the idea of pessimism.
Revised: We use the following notion of ξ -uncertainty quantiﬁer as follows to formalize the idea of pessimism.

Original: Definition 3.2 ( ξ -Uncertainty Quantifier) .
Revised: Deﬁnition 3.2 ( ξ -Uncertainty Quantiﬁer) .

Original: We say Γ : S ×
Revised: We say Γ : S ×

Original: A → R is a ξ -uncertainty quantifier for (cid:98) B and (cid:98) V if with probability 1 − ξ , for all ( s, a ) ∈ S × A , 
Revised: A → R is a ξ -uncertainty quantiﬁer for (cid:98) B and (cid:98) V if with probability 1 − ξ , for all ( s, a ) ∈ S × A , [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3. U NSUPERVISED D ATA S HARING
Revised: 3. U NSUPERVISED D ATA S HARING

Original: We consider unsupervised data sharing in offline reinforcement learning.
Revised: We consider unsupervised data sharing in ofﬂine reinforcement learning.

Original: We first characterize the quality of the dataset with the notion of coverage coefficient (Uehara & Sun, 2021), defined as below [Equation]
Revised: We ﬁrst characterize the quality of the dataset with the notion of coverage coefﬁcient (Uehara & Sun, 2021), deﬁned as below.

Original: Definition 3.3.
Revised: Deﬁnition 3.3.

Original: The coverage coefficient C † of a dataset D = { ( s τ , a τ , r τ ) }
Revised: The coverage coefﬁcient C † of a dataset D = { ( s τ , a τ , r τ ) }

Original: Nτ =1 is defined as
Revised: Nτ =1 is deﬁned as

Original: C † = sup

Original: [Equation]
Revised: [Equation]

Original: The coverage coefficient C †  represents the maximum ratio between the density of empirical stateaction distribution and the density induced from the optimal policy.
Revised: The coverage coefﬁcient C † is common in ofﬂine RL literature (Uehara & Sun, 2021; Jin et al., 2021; Rashidinejad et al., 2021), which represents the maximum ratio between the density of empirical state-action distribution and the density induced from the optimal policy.

Original: Intuitively, it represents the quality of the dataset.
Revised: Intuitively, it represents the quality of the dataset.

Original: For example, the expert dataset has a high coverage ratio while the random dataset may have a low ratio.
Revised: For example, the expert dataset has a high coverage ratio while the random dataset may have a low ratio.

Original: We denote D 0 as the origin labeled dataset, with coverage coefficient C † 0 and size N 0 .
Revised: We denote D 0 as the origin labeled dataset, with coverage coefﬁcient C † 0 and size N 0 .

Original: And we denote the unlabeled dataset as D 1 , with coverage coefficient C † 1 and size N 1 .
Revised: And we denote the unlabeled dataset as D 1 , with coverage coefﬁcient C † 1 and size N 1 .

Original: Note that it is possible that the unlabeled data comes from multiple sources, such as multi-task settings, and we still use D 1 to represent the combined dataset ∪ Mi =1 D i  for simplicity.
Revised: Note that it is possible that the unlabeled data comes from multiple sources, such as multi-task settings, and we still use D 1 to represent the combined dataset ∪ Mi =1 D i from M tasks for simplicity.

Original:  P ROVABLE U NSUPERVISED D ATA S HARING
Revised:  P ROVABLE U NSUPERVISED D ATA S HARING

Original: How can we leverage unlabeled data for offline RL?
Revised: How can we leverage reward-free data for ofﬂine RL?

Original: At first sight, we may learn the reward function from labeled data via the following regression
Revised: A naive approach is to learn the reward function from labeled data via the following regression

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where f θ ( s τ , a τ ) =
Revised: where f θ ( s τ , a τ ) =

Original: ϕ ( s τ , a τ ) ⊤ θ in linear MDPs.
Revised: φ ( s τ , a τ ) (cid:62) θ in linear MDPs.

Original: Then we can use this learned reward function to label unsupervised data.
Revised: Then we can use this learned reward function to label unsupervised data.

Original: However, Yu et al. (2022) observes that using such a naive reward prediction method can lead to suboptimal performances, which is even worse than using all zeros in some scenarios. This is due tothe potential overestimation of the predicted reward r (cid:98) θ , which ruins the pessimism in offline algorithms.
Revised: However, this approach can lead to suboptimal performance due to overestimation of the predicted reward r (cid:98) θ (Yu et al., 2022), which undermines the pessimism in ofﬂine algorithms.

Original: To address this issue, we first analyze the uncertainty in learned reward functions. We propose simply adding penalties for such uncertainty to leverage unlabeled data, which leads to a provable data-sharing algorithm, which we refer to as PDS.
Revised: To address this issue, we propose a data-sharing algorithm called Provable Data Sharing (PDS). We start by analyzing the uncertainty in learned reward functions and add penalties for such uncertainty to leverage unlabeled data.

Original: As we will show in Section 4.2,  PDS has a provable performance bound consisting of two parts. The first part is the offline error, which is tightened compared to no data sharing because of additional data, and the second part comes from the uncertainties in rewards.
Revised: In Section 4.2, we show that PDS has a provable performance bound consisting of two parts: the ofﬂine error, which is tightened compared to no data sharing due to additional data, and the error from reward bias.

Original: As long as the unlabeled dataset has mediocre size or quality, the performance bound is provably better than no data sharing.
Revised: The performance bound of PDS is provably better than no data sharing as long as the unlabeled dataset has mediocre size or quality.

Original: We  extend our algorithm in linear MDPs to general settings in Section 4.3 and propose to use ensembles forthe reward uncertainty estimation.
Revised: We also extend our algorithm in linear MDPs to general settings in Section 4.3 and propose using ensembles for reward uncertainty estimation.

Original: We use IQL (Kostrikov et al., 2021) as the backbone offline algorithm, which leads to Algorithm 2. The algorithm is simple and can be easily integrated with other model-free offline algorithms.
Revised: We demonstrate the effectiveness of PDS by integrating it with IQL (Kostrikov et al., 2021), and present Algorithm 3, which is simple and can be easily integrated with other model-free ofﬂine algorithms.

Original: 4. P ROVABLE D ATA S HARING
Revised: 4. P ROVABLE D ATA S HARING

Original: We begin by analyzing the uncertainty in learned reward functions.
Revised: To address the issue of potential overestimation of predicted rewards, we ﬁrst analyze the uncertainty in learned reward functions.

Original: In the context of linear MDPs, the reward function can be learned via linear regression, and the uncertainty of the parameters is characterized by the following lemma.
Revised: In the context of linear MDPs, the reward function can be learned via linear regression, and the uncertainty of the parameters is characterized by the elliptical conﬁdence region, as shown in Lemma 4.1.

Revised: This conﬁdence region is important as it allows us to give a more accurate estimation of the reward function while keeping the overall algorithm pessimistic.

Original: Lemma 4.1 ( Abbasi-Yadkori et al.
Revised: Lemma 4.1 ( Abbasi-Yadkori et al.

Revised: Let α = √ ν + r max · (cid:113) 2 log 1 δ + d log (1 + N 0 νd ) , Λ = νI + (cid:80)

Revised: [Equation]

Original: Let α = √
Revised: = [Equation] [Equation] τ

Original: [Equation]
Revised: [Equation]

Revised: (cid:62) ,

Original: [Equation]
Revised: [Equation]

Original:  where (cid:98) θ is the minimizer in Equation (7) , then we have P ( θ ⋆ ∈ C ( δ )) ≥ 1 − δ , where θ ⋆ is the true parameter for the reward function.
Revised: (8) where (cid:98) θ is the minimizer in Equation (7) , then we have P ( θ (cid:63) ∈ C ( δ )) ≥ 1 − δ , where θ (cid:63) is the true parameter for the reward function.

Original: Proof.
Revised: Proof.

Original: Please refer to Theorem 2 in Abbasi-Yadkori et al.
Revised: Please refer to Theorem 2 in Abbasi-Yadkori et al.

Original: (2011) for detailed proof.
Revised: (2011) for detailed proof.

Original: Lemma 4.1 is a standard ellipsoid argument and shows that with high probability, the true parameters lie in the range ofan ellipsoid near the learned parameters. The diameter of the ellipsoid only depends on the quality and size of labeled data.
Revised: Lemma 4.1 provides a useful insight: the uncertainty of the learned reward function in linear MDPs only depends on the quality and size of labeled data.

Original: This motivates us to give a more accurate estimation of the reward function to keep the overall algorithm pessimistic while keeping the reward bias small.
Revised: Based on this insight, we propose a two-phase algorithm that guarantees a provable performance bound, as shown in Theorem 4.3.

Original: The simple reward prediction method ruins the pessimistic estimation of the algorithm and UDS(Yu et al., 2022) may induce atoo-large reward bias.
Revised: The simple reward prediction method can compromise the pessimistic estimation of the algorithm, while UDS may result in a reward bias that is too large.

Original: As we will show in Theorem 4.3, we can add pessimism over the reward function and plug in other model-free algorithms like PEVI (Jin et al., 2021), and we will have a provable performance bound.

Original: Based on this, we design an algorithm consisting of two phases. In the first phase, we construct a pessimistic reward estimator by finding the reward function in the confidence set that leads to the lowest optimal value.
Revised: Our algorithm consists of two phases: in the ﬁrst phase, we construct a pessimistic reward estimator that ﬁnds the reward function in the conﬁdence set that leads to the lowest optimal value.

Original: In the second phase, our algorithm conducts standard offline RL with the given pessimistic reward function.
Revised: In the second phase, we conduct standard ofﬂine RL with the given pessimistic reward function.

Original: The overall algorithm is summarized in Algorithm 1. It is still computationally challenging to find the best parameter in the confidence set as in Equation 11, which is a bi-level optimization problem since it requires computing a value estimation for each θ and minimizing over θ . To solve this challenge, we propose simply using a pessimistic estimation, which keeps the  pessimistic property of the offline algorithms.
Revised: To solve the challenge of ﬁnding the best parameter in the conﬁdence set, which is a bi-level optimization problem, we propose using a simpler method that maintains the pessimistic property of the ofﬂine algorithms. This method involves using a pessimistic estimation, which allows us to keep the algorithm pessimistic while avoiding the computational challenges of the bi-level optimization problem.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where Λ = νI + (cid:80) N 0 τ =1 ϕ ( s τ , a τ ) ϕ ( s τ , a τ ) ⊤ .
Revised: where Λ = νI + (cid:80) N 0 τ =1 φ ( s τ , a τ ) φ ( s τ , a τ ) (cid:62) .

Original: We use such a pessimistic estimation because Equation (9) is a lower bound of reward functions in the confidence set C ( δ ) , which is guaranteed by the following lemma:
Revised: We adopt the pessimistic estimation in Equation (9) because it provides a lower bound for reward functions in the conﬁdence set C ( δ ) , as guaranteed by the following lemma derived from CauchySchwartz inequalities.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: For any θ ∈ C ( δ ) ,
Revised: For any θ ∈ C ( δ ) ,

Original: [Equation]
Revised: [Equation]

Original: This lemma follows directly from Cauchy-Schwartz inequalities.

Original: Then it is easy to see that with a high probability of 1 − δ , Equation (9) does not overestimate the true reward function.

Original: Whenthe amount of labeled data is limited, or the distributional shift between  labeled and unlabeled dataset is large, Equation (9) will degrade to 0, which is exactly the UDS  (Yu et al., 2022) algorithm.
Revised: When labeled data is scarce or there is a signiﬁcant distributional shift between the labeled and unlabeled data, Equation (9) degenerates to 0, which is equivalent to the UDS algorithm (Yu et al., 2022). MDP

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) V θ is the estimated value function from Algorithm 3 and dataset D 0 ∪ D 1 with reward relabeled with parameter θ .[Equation]
Revised: where (cid:98) V θ is the estimated value function from Algorithm 2 and dataset D 0 ∪ D 1 with reward relabeled with parameter θ .

Original: [Equation]
Revised: [Equation]

Original: 4. T HEORETICAL A NALYSIS
Revised: 4. T HEORETICAL A NALYSIS

Original: In this subsection, we analyze how PDS improves the performance bound with unlabeled data.
Revised: The following subsection analyzes how the provable data-sharing (PDS) algorithm can enhance the performance bound by leveraging unlabeled data.

Revised: To be speciﬁc, we present the following theorem.

Original: Theorem 4.3 (Performance Bound for PDS) .
Revised: Theorem 4.3 (Performance Bound for PDS) .

Original: Suppose the dataset D 0 , D 1 have positive coverage coefficients C † 0 , C † 1 , and the underlying MDP is a linear MDP.
Revised: Suppose the dataset D 0 , D 1 have positive coverage coefﬁcients C † 0 , C † 1 , and the underlying MDP is a linear MDP.

Original: In Algorithm 1, we set λ
Revised: In Algorithm 1, we set λ

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is an absolute constant and δ ∈ (0 , 1) is the confidence parameter.
Revised: where c > 0 is an absolute constant and δ ∈ (0 , 1) is the conﬁdence parameter.

Original: Then with probability 1 − 2 δ , the policy (cid:98) π generated by PDS satisfies for all s ∈ S ,
Revised: Then with probability 1 − 2 δ , the policy (cid:98) π generated by PDS satisﬁes for all s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Please refer to Appendix B for detailed proof.
Revised: Please refer to Appendix B for detailed proof.

Original: The performance bound of PDS consists of two terms.
Revised: The performance bound of PDS is composed of two terms.

Original: The first is the offline error, which inherits from offline algorithms.
Revised: The ﬁrst is the ofﬂine error, which is inherited from ofﬂine algorithms.

Original: Compared to the bound without data sharing, this bound is improved due to additional unlabeled data with size N 1 and coverage C † 1 .
Revised: This bound is improved when additional unlabeled data with size N 1 and coverage C † 1 is available.

Original: The second term is  reward error, which comes from uncertainties in the rewards.
Revised: The second term is the reward bias, which arises due to uncertainties in the rewards.

Original: Note that the second term is the same as the performance bound of a linear bandit with rewards in the range [0 , r max / (1 − γ )] .
Revised: Notably, this term is equivalent to the performance bound of a linear bandit with rewards in the range [0 , r max / (1 − γ )] .

Original: This means that when the number of unlabeled data goes to infinity, the uncertainty of the dynamics becomes zero, and the RL problem degenerates to linear bandits.
Revised: As the number of unlabeled data approaches inﬁnity, the uncertainty of the dynamics decreases to zero, and the RL problem becomes a linear bandit problem.

Original: This theorem reflects the performance bound in the worst case and improving the bound means improving the worst-case performance.
Revised: The theorem demonstrates that PDS outperforms UDS, which suffers from a constant reward bias, and naive reward prediction methods, which lack pessimism and therefore do not offer such guarantees. Moreover, we demonstrate the tightness of the bound by constructing an “adversarial” dataset that matches the bound’s suboptimality (see Appendix F).

Original: To better see the benefit of unlabeled data, we define the suboptimality bound ratio (SBR) of an offline algorithm A as the ratio of the suboptimality bound from the policy learned with additional unlabeled data to the suboptimality bound of the policy learned with only labeled data:
Revised: To better understand the beneﬁts of unlabeled data, we deﬁne the suboptimality bound ratio (SBR) of an ofﬂine algorithm A as the ratio of the suboptimality bound obtained by the policy learned with additional unlabeled data to the suboptimality bound of the policy learned using labeled data alone. Mathematically, the SBR of A is given by:

Original: [Equation]
Revised: [Equation]

Original: (13) where SubOpt is an upper bound for the suboptimality.
Revised: where SubOpt is the tight upper bound on suboptimality.

Original: Note that the ratio compares the bound of suboptimality rather than the suboptimality itself.

Original: Intuitively, SBR measures the benefit of unlabeled data to the offline algorithm. The smaller the SBR, the more benefit the unlabeled data brings to the offline algorithm.
Revised: The SBR provides a measure of the beneﬁt of unlabeled data to the ofﬂine algorithm, with a smaller SBR indicating a greater beneﬁt from the unlabeled data.

Original: Applying the definition above to PDS, we have the following corollary.
Revised: Applying this deﬁnition to PDS, we obtain the following corollary.

Original: Corollary 4.4 (Informal) .
Revised: Corollary 4.4 (Informal) .

Original: The SBR of PDS satisfies
Revised: The SBR of PDS satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where c is the constant in Theorem 4.3 and we ignore the logarithmic factors.
Revised: where c is the constant in Theorem 4.3 and we ignore the logarithmic factors.

Original: When does unlabeled data improve the performance of offline algorithms?
Revised: When does unlabeled data improve the performance of ofﬂine algorithms?

Original: Based on Corollary 4.4, we can analyze the relative performance of PDS under different conditions.
Revised: Corollary 4.4 allows us to analyze the relative performance of PDS under different conditions.

Original: The first term  depends on the qualities and amounts of both labeled and unlabeled datasets.
Revised: The ﬁrst term of the bound depends on the qualities and amounts of both labeled and unlabeled datasets.

Original: It will be sufficiently small if the unlabeled dataset has a mediocre number of samples or data quality.
Revised: If the unlabeled dataset has a mediocre number of samples or data quality, the ﬁrst term will be sufﬁciently small.

Original: The second term affects the asymptotic performance when the unlabeled data goes to infinity. It depends on the discount factor and the dimension of the problem.
Revised: The second term affects the asymptotic performance when the unlabeled data approaches inﬁnity, and it depends on the discount factor and the dimension of the problem.

Original: The longer the horizon or, the larger the problem dimension, the better PDS improves over no data-sharing algorithms asymptotically.
Revised: PDS improves over no data-sharing algorithms asymptotically in larger problems or longer horizons.

Original: Please refer to Appendix F for a more detailed discussion.
Revised: For a more detailed discussion, please refer to Appendix E.

Original: 4. P RACTICAL I MPLEMENTATION
Revised: 4. P RACTICAL I MPLEMENTATION

Original: This subsection discusses the practical implementation of PDS in general MDPs.
Revised: This subsection outlines the practical implementation of PDS in general MDPs.

Original: For the uncertainty estimation, we choose to use L ensembles{ θ 1 , . .. , θ L } , which is learned via Equation (9).
Revised: We employ L ensembles θ 1 , . . , θ L to estimate uncertainty, which are learned using Equation (7).

Original: Then we use the following pessimistic estimation:[Equation]
Revised: To estimate pessimistic rewards, we use the following pessimistic estimation:

Original: [Equation]
Revised: [Equation]

Original: where µ ( s, a ) =
Revised: where µ ( s, a ) =

Original: 1 L (cid:80) L i =
Revised: 1 L (cid:80) Li =

Original: 1 f θ i [Equation] [Equation] L 
Revised: 1 f θ i ( s, a ) , σ ( s, a ) = (cid:113) 1 L (cid:80) Li

Original: [Equation]{+=
+}
Revised: {+=
+}

Original: [Equation] f [Equation] , a )) 2 are the mean and standard deviation, respectively.
Revised: ( f θ i ( s, a ) − µ ( s, a )) 2 are the mean and standard deviation, respectively.

Original:  k is a hyperparameter  to control the amount of pessimism.
Revised: Here, k is a hyperparameter used to control the amount of pessimism.

Original: Note that we can also use the minimum over L ensembles for the pessimistic estimation, which is linked to Equation (15) following An et al.
Revised: We can also use the minimum over L ensembles for the pessimistic estimation, which is linked to Equation (15) following An et al.

Original: ; Royston (1982) as 
Revised: ; Royston (1982) as shown in Equation (16):

Original: [Equation]
Revised: [Equation]

Original: where Φ is the CDF of the standard Gaussian distribution.
Revised: where Φ is the CDF of the standard Gaussian distribution.

Revised: The appropriate value of k for each domain can be difﬁcult to determine.

Revised: To address this issue, we observe that the amount of pessimism required for different domains is proportional to the difference in mean rewards between labeled and unlabeled data.

Revised: Leveraging this insight, we propose a simple and efﬁcient automatic mechanism for adjusting the value of the k parameter.

Revised: Speciﬁcally, we suggest a method that adjusts k based on the difference in mean rewards, as given by Equation (17):

Revised: [Equation]

Original: There aremore advanced estimation methods like the dropout-based method (Wu et al., 2021) and the model-based method (Yu et al., 2020b), but we stick to the ensemble method for simplicity.
Revised: where k = a · max( µ − (cid:98) µ, 0) | µ | + (cid:15) . where µ = 1 N 0 (cid:80) N 0 i =1 µ ( s i , a i ) , (cid:98) µ = 1 N 1 (cid:80) N 1 i =1 (cid:98) µ ( s i , a i ) are the mean reward of labeled and (predicted) unlabeled data, respectively. We use a = 25 and L = 10 in all experiments.

Original: Then we can plug in any model-free offline algorithms.
Revised: Then we can plug in any model-free ofﬂine algorithms.

Original: Here we use IQL (Kostrikov et al., 2021) as the backbone offline algorithm for illustration, but we emphasize that it can be easily integrated with other model-free algorithms.
Revised: Here we use IQL (Kostrikov et al., 2021) as the backbone ofﬂine algorithm, but we emphasize that it can be easily integrated with other modelfree algorithms.

Original: The overall algorithm is summarized in Algorithm 2. [Equation] [Equation] [Table]
Revised: The details of our algorithm is summarized in Algorithm 3 in Appendix A.2.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: end

Original: [Table]

Original: E XPERIMENTS
Revised: XPERIMENTS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we aim to evaluate the effectiveness of pessimistic reward estimation and answer the following questions: (1) How does PDS perform compared to the naive reward prediction and unlabeled data sharing (UDS) methods in single locomotion and manipulation tasks?
Revised: we aim to evaluate the effectiveness of pessimistic reward estimation and answer the following questions: (1) How does PDS perform compared to the naive reward prediction and unlabeled data sharing (UDS) methods in single locomotion and manipulation tasks?

Original: (2) How does
Revised: (2) How does

Original: PDS behave in multi-task offline RL settings?
Revised: PDS behave in multi-task ofﬂine RL settings compared to baselines?

Original: (3) Why and when is PDS effective?
Revised: (3) What makes PDS effective?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Single-task domains and datasets.
Revised: Single-task domains and datasets.

Original: To answer Question (1), we perform empirical evaluations on the hopper, walker2d
Revised: To address Question (1), we empirically evaluate the PDS algorithm on the hopper, walker2d, and kitchen tasks from the D4RL benchmark suite (Fu et al., 2020). We use 50 labeled trajectories with varying amounts of unlabeled data of different sizes and qualities. This experimental setup is motivated by real-world problems where labeled data is often scarce

Original: [Table]
Revised: [Table]

Original: and kitchen tasks adopted from D4RL (Fu et al., 2020). We consider
Revised: and additional unlabeled data may be readily available.

Original: [Table]
Revised: [Table]

Original: This setup resembles practical problems where the labeled data is scarce.

Original: Multi-task domains and datasets.
Revised: Multi-task domains and datasets.

Original: As for Question (2), we consider several multi-task domains.
Revised: We investigate Question (2) by evaluating PDS on several multi-task domains.

Original: (1) The first set of domains  isthe Meta-World (Yu et al., 2020a). We consider the same setup as in CDS (Yu et al., 2021a) and  four tasks: door open , door close , drawer open  and drawer close .
Revised: The ﬁrst set of domains we consider is Meta-World (Yu et al., 2020a), where we adopt the same setup as in CDS (Yu et al., 2021a) and evaluate PDS on four tasks: door open , door close , drawer open , and drawer close .

Original: Each task has a sparse reward of 1 when the success condition is met andotherwise.

Original: The second domain is the AntMaze task in D4RL, which consists of mazes of two sizes (medium and large) with 3 and 7 tasks corresponding to different goal positions, respectively. Following Yu et al.
Revised: The second domain is the AntMaze task in D4RL, which consists of mazes of two sizes (medium and large) and includes 3 and 7 tasks, respectively, corresponding to different goal positions.

Original: (2021a; 2022), we construct two set of tasks, undirected and directed .

Original: The trajectories in the undirected dataset is randomly and uniformly divided into different tasks, while the directed dataset is associated with the goal closest to the final state of the trajectory.

Original: Please refer to Appendix E for detailed description.
Revised: For a detailed description of the experimental setting, please refer to Appendix D.

Original: Comparisons.
Revised: Comparisons.

Original: We adopt the state-of-the-art unlabeled data sharing method (UDS) (Yu et al., 2022) as our baseline, and we combine UDS with IQL (Kostrikov et al., 2021), the same underlying offline RL method as PDS for a fair comparison.
Revised: To ensure a fair comparison, we combine UDS with IQL (Kostrikov et al., 2021), the same underlying ofﬂine RL method as PDS.

Original: We also train  naive reward prediction method (Prediction), sharing all true rewards method (Oracle), and adapt them with IQL.
Revised: In addition to UDS, we train a naive reward prediction method and the sharing-all-true-rewards method (Oracle), and adapt them with IQL.

Original: For more details on experimental settings and hyper-parameter settings, please refer to Appendix G.
Revised: In all experiments, we set the hyperparameters a = 25 and L = 10 for our method.

Original: 5. E XPERIMENTAL R ESULTS
Revised: 5. E XPERIMENTAL R ESULTS

Original: Results of Question (1).
Revised: Results of Question (1).

Original: We evaluate each method on the hopper, walker2d, and kitchen domains. As shown in Table 3, PDS outperforms other methods on most tasks and achieves competitive or better performance than the oracle method.
Revised: We evaluated each method on the hopper, walker2d, and kitchen domains and found that PDS outperformed the other methods on most tasks and achieved competitive or better performance than the oracle method (Table 3).

Original: Particularly, PDS performs well when the labeled and unlabeled datasets have different data qualities. We hypothesize that this is because PDS successfully captures the uncertainties induced by this distribution shift and keeps the algorithm pessimistic.
Revised: Notably, PDS performed well when the labeled and unlabeled datasets had different data qualities, which we attribute to its ability to capture the uncertainties induced by this distribution shift and maintain a pessimistic algorithm.

Original: The prediction method performs well when the unlabeled dataset has a good quality, and UDS performs well when the unlabeled data has low quality.
Revised: The prediction method performed well when the unlabeled dataset had high quality, and UDS performed well when the unlabeled data had low quality.

Original: PDS takes the better of both worlds and achieves a better performance.
Revised: PDS combined the strengths of both methods and achieved superior performance.

Original: Results of Question (2).
Revised: Results of Question (2).

Original: Multi-task settings exhibit more significant distributional shifts between labeleddata and unlabeled data due to the different goals of the tasks.
Revised: Multi-task settings exhibit greater distributional shifts between labeled and unlabeled data due to the differing task goals.

Original: To evaluate the performance of PDS on multi-task settings, we evaluate each method on the AntMaze andthe Meta-World domain, and the result is shown in Table 1 and Table 2. On Meta-World experiments, the performance of PDS is comparable to the oracle method and outperforms other methods.
Revised: We evaluated PDS and the other methods on the AntMaze and Meta-World domains (Tables 2 and ?? ) and found that PDS’s performance was comparable to the oracle method and outperformed the other methods.

Original: UDS performs relatively poorly onthis dataset. We hypothesize that this is because the dataset quality is relatively high, and labeling the  dataset  with zeros induces a large reward bias.
Revised: UDS performed relatively poorly on the Meta-World dataset, possibly due to the high dataset quality, which made labeling with zeros induce a large reward bias.

Original: On the multi-task AntMaze domain, 
Revised: On the multi-task AntMaze domain, PDS outperformed both

Original: PDS outperforms both UDS and the naive prediction method. The performance gap between PDS and other baselines is larger on the diverse dataset.
Revised: UDS and the naive prediction method, especially on the diverse dataset.

Original: This aligns with our observation on single-task domains that PDS performs better when the distribution shift between datasets is larger.[Figure] [Figure]
Revised: These results aligned with our observation on single-task domains that PDS performs better when the distribution shift between datasets is larger.

Original: Results of Question (3).
Revised: Results of Question (3).

Original: To investigate the effect of uncertainty weights, we conduct experiments on the hopper and walker2d tasks with various penalty weight k , shown in Figure 1.
Revised: We conducted experiments on the hopper and walker2d tasks with various penalty weights k to investigate the effect of uncertainty weights in PDS (Figure 1).

Original: The results show that PDS interpolates between UDS and the reward prediction method. PDS degenerates to naive reward prediction if k is small and performs like UDS when k is relatively large. PDS offers a better trade-off to balance the conservation and generalization of reward estimators and thus achieves better performance.
Revised: The results shows that PDS can interpolate between UDS and the reward prediction method and offered a better trade-off to balance the conservation and generalization of reward estimators, resulting in better performance.

Original: We plot the standard deviation of state values learned by IQL between different runs to investigate further why PDS improves over naive prediction methods.

Original: The result in Figureshows that PDS reduces the variance from reward prediction and is close to the oracle method. This indicates that PDS successfully reduces the uncertainties from the variance of reward predictors while keeping the reward bias small.
Revised: Also, PDS reduces the variance from reward prediction and is close to the oracle method, indicating its ability to reduce the uncertainties from the variance of reward predictors while keeping the reward bias small, as shown in Figure 2. [Figure]

Original:  Discussion between PDS, UDS and  Prediction: PDS is the generalization of  UDS and Prediction methods since UDS is PDS with k = ∞ , and Prediction is PDS with k = 0 .
Revised: [Figure] Discussion of PDS, UDS, and Reward Prediction: PDS can be seen as a generalization of both UDS and the reward prediction method. UDS sets the penalty weight k to inﬁnity, while the reward prediction method sets it to zero.

Original: Further, UDS induces a high bias, and Prediction ruins the pessimism of offline algorithms.
Revised: However, UDS introduces a high reward bias [Figure] and the reward prediction method ruins the pessimism of ofﬂine algorithms.

Original: On the contrary, PDS achieves a trade-off between bias and pessimism by adaptively adjusting k .
Revised: In contrast, PDS offers a trade-off between bias and pessimism by adaptively adjusting k .

Original: Therefore, it is not surprising that tuning k yielding better results than either baseline.
Revised: To verify that overestimation is the main factor for the suboptimal performance of the reward prediction method, we conduct additional ablation studies, as shown in Appendix G.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper investigates the theoretical benefit of using unlabeled data in offline reinforcement learning.
Revised: In this paper, we show that incorporating reward-free data into ofﬂine reinforcement learning can yield signiﬁcant performance improvements.

Original: We show that unlabeled data helps us gain more information about the dynamics of the MDP, reduce the problem to linear bandits, and improve the  performance bound accordingly.
Revised: Our theoretical analysis reveals that unlabeled data provides additional information about the MDP’s dynamics, reducing the problem to linear bandits in the limit and improving performance bounds therefore.

Original: Based on this analysis, we propose a new method, PDS, which adds uncertainty penalties on learned rewards to keep the algorithm conservative.
Revised: Building upon these insights, we propose a new algorithm, PDS, that leverages this information by incorporating uncertainty penalties on learned rewards to ensure a conservative approach.

Original: We show that our method had provable guarantees in theory and achieves superior performance in practice.
Revised: Our method has provable guarantees in theory and achieves superior performance in practice.

Original: Integrating representation learning methods for offline unsupervised data sharing is an interesting future direction. It is also an interesting direction to extend our analysis into more general settings than linear MDPs like generalized linear MDPs (Wang et al., 2019) and low-rank MDPs (Ayoub et al., 2020; Jiang et al., 2017).  
Revised: In future work, it may be interesting to explore how PDS can be further improved with representation learning methods, and to extend our analysis to more general settings, such as generalized linear MDPs (Wang et al., 2019) and low-rank MDPs (Ayoub et al., 2020; Jiang et al., 2017).  A CKNOWLEDGEMENTS

Original:  A A LGORITHM D ETAILS 
Revised: This work is supported in part by Science and Technology Innovation 2030 - “New Generation Artiﬁcial Intelligence” Major Project (No. 2018AAA0100904) and the National Natural Science Foundation of China (62176135). A A LGORITHM D ETAILS A. P ESSIMISTIC V ALUE I TERATION (PEVI,(J IN ET AL ., 2021))

Original: In this section, we describe the details of PEVI algorithm.
Revised: In this section, we describe the details of PEVI algorithm.

Original: In linear MDPs, we can construct (cid:98) B (cid:98) V and Γ based on D as follows, where (cid:98) B (cid:98) V is the empirical estimation for B (cid:98) V .
Revised: In linear MDPs, we can construct (cid:98) B (cid:98) V and Γ based on D as follows, where (cid:98) B (cid:98) V is the empirical estimation for B (cid:98) V .

Original: For a given dataset D = { ( s τ , a τ , r τ ) } Nτ =1 , we define the empirical mean squared Bellman error (MSBE) as
Revised: For a given dataset D = { ( s τ , a τ , r τ ) } Nτ =1 , we deﬁne the empirical mean squared Bellman error (MSBE) as

Original: [Equation]
Revised: [Equation]

Original: Here λ > 0 is the regularization parameter.
Revised: Here λ > 0 is the regularization parameter.

Original: Note that (cid:98) w has the closed form
Revised: Note that (cid:98) w has the closed form

Original: [Equation]
Revised: [Equation]

Original: where Λ = λI +
Revised: where Λ = λI +

Original: N (cid:88)τ
Revised: N (cid:88)

Original: = ϕ

Original: ( s τ , a τ ) ϕ

Original: ( s

Original: [Equation]
Revised: [Equation]

Original: a [Equation] ) ⊤ .
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Then we simply let
Revised: Then we simply let

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, we construct Γ based on D as
Revised: Meanwhile, we construct Γ based on D as

Original: [Equation]
Revised: [Equation]

Original: Here β > 0 is the scaling parameter.
Revised: Here β > 0 is the scaling parameter.

Original: The overall PEVI algorithm is summarized in Algorithm 3.
Revised: The overall PEVI algorithm is summarized in Algorithm 2.

Original: +
Revised: 1: Require : Dataset D = { ( s τ , a τ , r τ , s τ +1 ) } Tτ =1 . 2: Initialization: Set (cid:98) V ( · ) ← 0 and construct Γ( · , · ) .

Original: = converged do 4:
Revised: 3: while not converged do 4:

Original: Construct ( (cid:98) B (cid:98) V )( · , · ) 5:
Revised: Construct ( (cid:98) B (cid:98) V )( · , · ) 5:

Original: Set (cid:98) Q ( · , · ) ← ( (cid:98) B (cid:98) V )( · , · ) − Γ( · , · ) .
Revised: Set (cid:98) Q ( · , · ) ← ( (cid:98) B (cid:98) V )( · , · ) − Γ( · , · ) .

Original: Set (cid:98) π ( · | · ) ← argmax π E π (cid:104) (cid:98) Q ( · , · ) (cid:105) .
Revised: Set (cid:98) π ( · | · ) ← argmax π E π (cid:104) (cid:98) Q ( · , · ) (cid:105) .

Original: Set (cid:98) V ( · ) ← E (cid:98) π (cid:104) (cid:98) Q ( · , · ) (cid:105) .
Revised: Set (cid:98) V ( · ) ← E (cid:98) π (cid:104) (cid:98) Q ( · , · ) (cid:105) .

Original: 8: end while 9: Return (cid:98) V , (cid:98) π
Revised: 8: end while 9: Return (cid:98) V , (cid:98) π

Revised: A. IQL WITH PROVABLE DATA SHARING

Revised: In this section, we give a detailed description of our IQL+PDS algorithm.

Revised: [Equation]

Revised: [Equation]

Original:  B P ROOF OF T HEOREM 4. 
Revised: [Equation] [Equation] [Equation] B P ROOF OF T HEOREM 4.

Original: Proof.
Revised: Proof.

Original: From Equation (11) in Algorithm 1, we have
Revised: From Equation (11) in Algorithm 1, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) θ is the pessimistic estimation of θ .
Revised: where (cid:101) θ is the pessimistic estimation of θ .

Original: Let E 1 be the event θ ⋆ ∈ C ( δ ) , then we have P ( E 1 ) ≥ 

Revised: Let E 1 be the event θ (cid:63) ∈ C ( δ ) , then we have P ( E 1 ) ≥

Original: −
Revised: −

Original: δ from Lemma 4.1.
Revised: δ from Lemma 4.1.

Original: Let E 2 be the event where the following inequality holds,
Revised: Let E 2 be the event where the following inequality holds,

Original: [Equation]
Revised: [Equation]

Original: (21) then we have P ( E 2 ) ≥ 1 − δ from Lemma C.3.
Revised: then we have P ( E 2 ) ≥ 1 − δ from Lemma C.3.

Original: Condition on E 1 ∩ E 2 , we have.
Revised: Condition on E 1 ∩ E 2 , we have.

Original: [Equation]
Revised: [Equation]

Original: where the first inequality follows from Lemma C.2.
Revised: where the ﬁrst inequality follows from Lemma C.2.

Original: The second inequality follows from Equation (20), and the last inequality follows from Lemma C.5 and C.1.
Revised: The second inequality follows from Equation (21), and the last inequality follows from Lemma C.

Original: From the union bound, we have that the above inequality holds with a probability of 1 − 2 δ .
Revised: From the union bound, we have that the above inequality holds with a probability of 1 − 2 δ .

Original: C A DDTIONAL L EMMAS AND M ISSING P ROOFS
Revised: C A DDTIONAL L EMMAS AND M ISSING P ROOFS

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Under the event in Lemma C.3, we have
Revised: Under the event in Lemma C.3, we have

Original: [Equation]
Revised: [Equation]

Original: with probability 1 − δ , for all ∥ θ ∥ 22 ≤ d .
Revised: with probability 1 − δ , for all (cid:107) θ (cid:107) 22 ≤ d .

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From the definition of (cid:98) Q ( s, a ) and (cid:98) V ( s )
Revised: From the deﬁnition of (cid:98) Q ( s, a ) and (cid:98) V ( s )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Under the condition of Lemma C
Revised: Under the condition of Lemma C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Here the first inequality follows from the fact that (cid:98) π ( ·| s ) = argmax π (cid:68) (cid:98) Q ( · , · ) , π ( ·| s ) (cid:69) and the second inequality follows from Equation (24) [Equation]
Revised: Here the ﬁrst inequality follows from the fact that (cid:98) π ( ·| s ) = argmax π (cid:68) (cid:98) Q ( · , · ) , π ( ·| s ) (cid:69) and the second inequality follows from Equation (25).

Original: By the Cauchy-Schwarz inequality, we have
Revised: By the Cauchy-Schwarz inequality, we have

Original: [Equation]
Revised: [Equation]

Original: for all s ∈ S .
Revised: for all s ∈ S .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Here { λ j ( s ) } dj =1 are the eigenvalues of Σ π ∗ ,s for all s ∈ S , the second inequality follows from
Revised: Here { λ j ( s ) } dj =1 are the eigenvalues of Σ π ∗ ,s for all s ∈ S , the second inequality follows from

Original: Equation (25).
Revised: Equation (26).

Original: Meanwhile, by Definition 3.1, we have ∥ ϕ ( s, a ) ∥ ≤ 1 for all ( s, a ) ∈ S × A .
Revised: Meanwhile, by Deﬁnition 3.1, we have (cid:107) φ ( s, a ) (cid:107) ≤ 1 for all ( s, a ) ∈ S × A .

Original: By Jensen’s inequality, we have
Revised: By Jensen’s inequality, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 27) for all s ∈ S .
Revised: for all s ∈ S .

Original: As Σ π ∗ ,s is positive semidefinite, we have λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] .
Revised: As Σ π ∗ ,s is positive semideﬁnite, we have λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] .

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] for all x ∈ S , where the second inequality follows from the fact that λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] , while the third inequality follows from the choice of the scaling parameter β > 0 .
Revised: for all x ∈ S , where the second inequality follows from the fact that λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] , while the third inequality follows from the choice of the scaling parameter β > 0 .

Original: Then we have the conclusion in Lemma C.1.
Revised: Then we have the conclusion in Lemma C.1.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Under the event in Lemma C.3, we have
Revised: Under the event in Lemma C.3, we have

Original: [Equation]
Revised: [Equation]

Original: (29) with probability 1 − δ , for all ∥ θ ∥ 22 ≤ d .
Revised: with probability 1 − δ , for all (cid:107) θ (cid:107) 22 ≤ d .

Original: Proof.
Revised: Proof.

Original: Similar to the proof of Lemma C.1, let
Revised: Similar to the proof of Lemma C.1, let

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: (31) we have

Original: [Equation]
Revised: [Equation]

Original: Then under the condition of Lemma C. [Equation] it holds that
Revised: Then under the condition of Lemma C.3, it holds that

Original: [Equation]
Revised: [Equation]

Original: Then we have the result immediately.
Revised: Then we have the result immediately.

Original: Lemma C.3 ( ξ -Quantifiers) .
Revised: Lemma C.3 ( ξ -Quantiﬁers) .

Original: Let [Equation] [Equation]
Revised: Let λ = 1 ,

Original: [Equation]
Revised: [Equation]

Original: Then Γ = β · (cid:0) ϕ ( s, a ) ⊤ Λ − 1 ϕ ( s [Equation] a ) (cid:1) 1 / 2 are ξ -quantifiers with probability at least 1 − ξ .
Revised: Then Γ = β · (cid:0) φ ( s, a ) (cid:62) Λ − 1 φ ( s, a ) (cid:1) 1 / 2 are ξ -quantiﬁers with probability at least 1 − ξ .

Original: That is, let
Revised: That is, let

Original: E 2 be the event that the following inequality holds,
Revised: E 2 be the event that the following inequality holds,

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]

Original: Then we have P ( E 2 ) ≥ 1 − ε .
Revised: Then we have P ( E 2 ) ≥ 1 − ε .

Original: Proof.
Revised: Proof.

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Then we bound (i) and (ii), respectively.
Revised: Then we bound (i) and (ii), respectively.

Original: For (i), we have
Revised: For (i) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: 35) where the first inequality follows from Cauchy-Schwartz inequality.
Revised: ) where the ﬁrst inequality follows from Cauchy-Schwartz inequality.

Original: The second inequality follows from the fact that || Λ − 1 || op ≤ λ − 1 and Lemma C.4.
Revised: The second inequality follows from the fact that || Λ − 1 || op ≤ λ − 1 and Lemma C.4.

Original: For notation simplicity, let ϵ τ
Revised: For notation simplicity, let (cid:15) τ

Original: = r τ + γ [Equation] V ( s 
Revised: = r τ + γ (cid:98) V ( s τ +1 ) − φ (cid:62) τ w , then we have

Original: [Equation]
Revised: [Equation]

Original: 1 ) − ϕ ⊤ τ w , then we have

Original: [Equation]
Revised: [Equation]

Original: The term (iii) is depend on the randomness of the data collection process of D .
Revised: The term (iii) is depend on the randomness of the data collection process of D .

Original: To bound this term, we resort to uniform concentration inequalities to upper bound
Revised: To bound this term, we resort to uniform concentration inequalities to upper bound

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: , where

Original: V ( R, B, λ ) =

Original: { V ( s ; w, β, Σ) : S → [0 , V max ] with || w || ≤ R, β ∈ [0 , B ] , Σ ⪰ λ · I }

Original: [Equation]
Revised: [Equation]

Original: (37) where V ( s ; w, β, Σ) = max a { ϕ ( s, a ) ⊤ w − β · (cid:112) ϕ ( s, a ) ⊤ Σ − 1 ϕ ( s, a ) } .
Revised: where V ( s ; w, β, Σ) = max a { φ ( s, a ) (cid:62) w − β · (cid:112) φ ( s, a ) (cid:62) Σ − 1 φ ( s, a ) } .

Original: For all ϵ > 0 , let
Revised: For all (cid:15) > 0 , let

Original: N ( ϵ ; R, B, λ ) be the minimal cover if V ( R, B, λ ) .
Revised: N ( (cid:15) ; R, B, λ ) be the minimal cover if V ( R, B, λ ) .

Original: That is, for any function V ∈ V ( R, B, λ ) , there exists a function V † ∈ N ( ϵ ; R, B, λ ) , such that
Revised: That is, for any function V ∈ V ( R, B, λ ) , there exists a function V † ∈ N ( (cid:15) ; R, B, λ ) , such that

Original: [Equation]
Revised: [Equation]

Original: Let R 

Revised: Let R

Original: =
Revised: =

Original: V max (cid:112) Nd/λ, B 

Revised: V max (cid:112) Nd/λ, B

Original: = 2 β , it is easy to show that at each iteration, (cid:98) V u ∈ V ( R 0 , B 0 , λ ) .
Revised: = 2 β , it is easy to show that at each iteration, (cid:98) V u ∈ V ( R 0 , B 0 , λ ) .

Original: From the definition of B , we have
Revised: From the deﬁnition of B , we have

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Let ϵ † τ = r
Revised: Let (cid:15) † τ = r

Original: ( s τ , a τ ) + γV † ( s τ +1 ) − B V † ( s, a ) , we have (iii
Revised: ( s τ , a τ ) + γV † ( s τ +1 ) − B V † ( s, a ) , we have (iii

Original: [Equation]
Revised: [Equation]

Original: = || N (cid:88) [Equation] =
Revised: |

Original: [Equation]
Revised: [Equation]

Original: It remains to bound || (cid:80)
Revised: It remains to bound || (cid:80)

Original: Nτ =1 ϕ τ ϵ † τ || 2Λ − 1 .
Revised: Nτ =1 φ τ (cid:15) † τ || 2Λ − 1 .

Original: From the assumption for data collection process, it is easy to show that E
Revised: From the assumption for data collection process, it is easy to show that E

Original: D [ ϵ τ | F τ − 1 ] = 0 , where F τ − 1 = σ ( { ( s i , a i ) τi =1 ∪ ( r i , s i +1 )
Revised: D [ (cid:15) τ | F τ − 1 ] = 0 , where F τ − 1 = σ ( { ( s i , a i ) τi =1 ∪ ( r i , s i +1 )

Original: τi =1 } ) is the σ -algebra generated by the variables from the first τ step.
Revised: τi =1 } ) is the σ -algebra generated by the variables from the ﬁrst τ step.

Original: Moreover, since ϵ τ ≤ 2 V max , we have ϵ τ are 2 V max -sub-Gaussian conditioning on F τ − 1 .
Revised: Moreover, since (cid:15) τ ≤ 2 V max , we have (cid:15) τ are 2 V max -sub-Gaussian conditioning on F τ − 1 .

Original: Then we invoke Lemma C.7 with M 

Revised: Then we invoke Lemma C.7 with M

Original: = λ · I and
Revised: = λ · I and

Original: M k = λ · I + (cid:80) kτ =1 ϕ ( s τ , a τ ) ϕ ( s τ , a τ ) ⊤ .
Revised: M k = λ · I + (cid:80) kτ =1 φ ( s τ , a τ ) φ ( s τ , a τ ) (cid:62) .

Original: For the fixed function V : S → [0 , V max ] , we have
Revised: For the ﬁxed function V : S → [0 , V max ] , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: (41) for all δ ∈ (0 , 1) .
Revised: for all δ ∈ (0 , 1) .

Original: Note that ∥ ϕ ( s, a ) ∥ ≤ 1 for all ( s, a ) ∈ S
Revised: Note that (cid:107) φ ( s, a ) (cid:107) ≤ 1 for all ( s, a ) ∈ S

Original: × A by Definition 3.1.
Revised: × A by Deﬁnition 3.1.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ op denotes the matrix operator norm
Revised: where (cid:107) · (cid:107) op denotes the matrix operator norm.

Original: Hence, it holds that det(Λ) ≤ ( λ + N ) d and det( λ · I ) =
Revised: Hence, it holds that det(Λ) ≤ ( λ + N ) d and det( λ · I ) =

Original: λ d , which implies
Revised: λ d , which implies

Original: [Equation]
Revised: [Equation]

Revised: ≤

Original: Therefore, we conclude the proof of Lemma C.3.
Revised: [Equation] Therefore [Equation] we conclude the proof of Lemma C

Revised: [Equation]

Original: Applying Lemma C.3 and the union bound
Revised: Applying Lemma C.3 and the union bound

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: P D

Original: [Equation]
Revised: [Equation]

Original: Recall that [Equation]
Revised: Recall that (cid:98) V ∈ V

Original:  where {+R 
 = V max (cid:112) Nd/λ, B+}
Revised: ( R 0 , B 0 , λ ) , where {+R 
 = V max (cid:112) Nd/λ, B+}

Original: [Equation]
Revised: [Equation]

Original: Here c > 0 is an absolute constant, ξ ∈ (0 , 1) is the confidence parameter, and ζ = log(2 dV max /ξ ) is specified in Algorithm 3.
Revised: Here c > 0 is an absolute constant, ξ ∈ (0 , 1) is the conﬁdence parameter, and ζ = log(2 dV max /ξ ) is speciﬁed in Algorithm 2.

Original: Applying Lemma C.6 with ε = dV max /N , we have
Revised: Applying Lemma C.6 with ε = dV max /N , we have

Original: [Equation]
Revised: [Equation]

Original:  By setting δ = ξ/ |N ( ε ) | , we have that with probability at least 1 − ξ , 
Revised: [Equation] By setting δ = ξ/ |N ( ε ) | , we have that with probability at least 1 − ξ , [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here the last inequality follows from simple algebraic inequalities.
Revised: Here the last inequality follows from simple algebraic inequalities.

Original: We set c ≥ 1 to be sufficiently large, which ensures that 36 + 8 · log(64 c 2 ) ≤ c 2 / 4 on the right-hand side of Equation (45).
Revised: We set c ≥ 1 to be sufﬁciently large [Equation] which ensures that 36 + 8 · log(64 c 2 ) ≤ c 2 / 4 on the right-hand side of Equation (46).

Original: By
Revised: By

Original: Equations (36) and (45), it holds that
Revised: Equations (37) and (46) [Equation] it holds that

Original: [Equation]
Revised: [Equation]

Revised: By Equations (20), (35)

Original: [Equation]
Revised: [Equation]

Original: By Equations (19), (34), (35), and (46), for all ( s, a ) ∈ S × A , it holds that
Revised: [Equation] and (47) [Equation] for all ( s, a ) ∈ S × A , it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − ξ .
Revised: with probability at least 1 − ξ .

Original: Therefore, we conclude the proof of Lemma C.3.
Revised: Therefore, we conclude the proof of Lemma C.3.

Original: Lemma C.4 (Bounded weight of value function) .
Revised: Lemma C.4 (Bounded weight of value function) .

Original: Let V max =
Revised: Let V max =

Original: r max / (1 − γ ) .
Revised: r max / (1 − γ ) .

Original: For any function
Revised: For any function

Original: V : S → [0 , V max ] , we have
Revised: V : S → [0 , V max ] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For (cid:98) w , we have
Revised: For (cid:98) w , we have

Original: [Equation]
Revised: [Equation]

Original: =
Revised: ≤ V

Original: [Equation]
Revised: [Equation]

Original: Λ −
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: ϕ τ
Revised: ≤ V

Original: [Equation]
Revised: [Equation]

Original: ϕ
Revised: (cid:114) N λ [Equation] Λ −

Original: [Equation]
Revised: [Equation]

Revised: = φ

Original: [Equation]
Revised: [Equation]

Revised: (cid:62) τ

Original: [Equation]
Revised: [Equation]

Revised: ≤ V max (cid:114) Nd λ .

Original: Lemma C.5.
Revised: Lemma C.5.

Original: For policy π ∗ and any reward function parameter θ ∈ C ( δ ) , we have
Revised: For policy π ∗ and any reward function parameter θ ∈ C ( δ ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the definition, we have
Revised: From the deﬁnition [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Where the first inequality follows from the Cauchy-Schwartz inequality and the second inequality uses the fact that θ ∈ C ( δ ) .
Revised: Where the ﬁrst inequality follows from the Cauchy-Schwartz inequality and the second inequality uses the fact that θ ∈ C ( δ ) .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the second to last inequality follows similarly as Equation(25) (26)  and the last inequality follows from the fact that α = 2 r max √ dζ 2 .
Revised: where the second to last inequality follows similarly as Equation (26) (27) and the last inequality follows from the fact that α = 2 r max √ dζ 2 .

Original: Note that such a choice of α is sufficient for Lemma 4.1 to hold since
Revised: Note that such a choice of α is sufﬁcient for Lemma 4.1 to hold since

Original: [Equation]
Revised: [Equation]

Original: where the inequalities holds for sufficiently small δ > 0 and d ≥ 2 .
Revised: where the inequalities holds for sufﬁciently small δ > 0 and d ≥ 2 .

Original: C. T ECHNICAL L EMMAS
Revised: C. T ECHNICAL L EMMAS

Original: Lemma C.6 ( ε -Covering Number (Jin et al., 2020)) .
Revised: Lemma C.6 ( ε -Covering Number (Jin et al., 2020)) .

Original: For all h ∈ [ H ] and all ε > 0 , we have
Revised: For all h ∈ [ H ] and all ε > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma C.6.
Revised: Proof of Lemma C.6.

Original: See Lemma D.6 in Jin et al.
Revised: See Lemma D.6 in Jin et al.

Original: (2020) for detailed proof.
Revised: (2020) for detailed proof.

Original: Lemma C.7 (Concentration of Self-Normalized Processes (Abbasi-Yadkori et al., 2011)) .
Revised: Lemma C.7 (Concentration of Self-Normalized Processes (Abbasi-Yadkori et al., 2011)) .

Original: Let {F t } ∞ t
Revised: Let {F t } ∞ t

Original: =
Revised: =

Original: be a filtration and { ϵ t } ∞ t =1 be an R -valued stochastic process such that ϵ t is F t -measurable for all t ≥ 1 .
Revised: be a ﬁltration and { (cid:15) t } ∞ t =1 be an R -valued stochastic process such that (cid:15) t is F t -measurable for all t ≥ 1 .

Original: Moreover, suppose that conditioning on F t − 1 , ϵ t is a zero-mean and σ -sub-Gaussian random variable for all t ≥ 1 , that is,
Revised: Moreover, suppose that conditioning on F t − 1 , (cid:15) t is a zero-mean and σ -sub-Gaussian random variable for all t ≥ 1 , that is,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, let { ϕ t } ∞ t =1 be an R d -valued stochastic process such that ϕ t is F t − 1 -measurable for all t ≥ 1 .
Revised: Meanwhile, let { φ t } ∞ t =1 be an R d -valued stochastic process such that φ t is F t − 1 -measurable for all t ≥ 1 .

Original: Also, let
Revised: Also, let

Original: M 0 ∈ R
Revised: M 0 ∈ R

Original: d × d be a deterministic positive-definite matrix and
Revised: d × d be a deterministic positive-deﬁnite matrix and

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 1 .
Revised: for all t ≥ 1 .

Original: For all δ > 0 , it holds that
Revised: For all δ > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 1 with probability at least 1 − δ .
Revised: for all t ≥ 1 with probability at least 1 − δ .

Original: Proof.
Revised: Proof.

Original: See Theorem 1 of Abbasi-Yadkori et al.
Revised: See Theorem 1 of Abbasi-Yadkori et al.

Original: (2011) for a detailed proof.
Revised: (2011) for a detailed proof.

Original: D C ONSTRUCTION OF A DVERSARIAL E XAMPLES

Original: Construction of Adversarial Examples: In this section, we show that there is an MDP and an “unfortunate” dataset such that the suboptimality of Algorithm 3 matches the performance bound in

Original: Theorem 4.3.

Original: We first focus on the case without data sharing.

Original: The same techniques can be used to match the reward-learning bound.

Original: We only need to show that all inequalities in Theorem 4.3 can become equality.

Original: Suppose we have an MDP with one state and N > d actions.

Original: d of them are optimal actions, with feature map

Original: [Equation]

Original: and let the optimal policy be uniform over d actions.

Original: Such construction makes Σ π ∗ ,s =

Original: I so that inequalities in Equation (25) ∼ (27) become equalities.

Original: Then we let samples from other actions match the confidence upper bound while samples from the optimal action match the confidence lower bound, and we also need all the samples to be symmetric over different dimensions (this is required by the Cauchy-Schwitz inequality used in the proof), such that the confidence bound inequalities in

Original: Lemma C.3 also become equalities.

Original: Then, in this case, the suboptimality bound is matched and the bound in Theorem 4.3 is tight.

Original: E E XPERIMENTAL S ETTING E. M ULTI - TASK A NTMAZE
Revised: D E XPERIMENTAL S ETTINGS D. M ULTI - TASK A NTMAZE

Original: We first divide the source dataset (e.g., antmaze-medium-diverse-v2) into the directed dataset or the undirected dataset.
Revised: We ﬁrst divide the source dataset (e.g., antmaze-medium-diverse-v2) into the directed dataset or the undirected dataset.

Original: The trajectories in the undirected dataset are randomly and uniformly divided into different tasks.
Revised: The trajectories in the undirected dataset are randomly and uniformly divided into different tasks.

Original: In contrast, the directed dataset is associated with the goal closest to the final state of the trajectory.
Revised: In contrast, the directed dataset is associated with the goal closest to the ﬁnal state of the trajectory.

Original: Then, for each subtask (e.g., goal=(0.0, 16.0)), we relabel the corresponding undirected or undirected dataset according to the goal.
Revised: Then, for each subtask (e.g., goal=(0.0, 16.0)), we relabel the corresponding undirected or undirected dataset according to the goal.

Original: Finally, we keep the rewards in the target task dataset (labeled dataset) while setting the rewards in the other task dataset to 0 (unlabeled dataset).
Revised: Finally, we keep the rewards in the target task dataset (labeled dataset) while setting the rewards in the other task dataset to 0 (unlabeled dataset).

Original: We visualize the directed or undirected dataset in Figure 3.
Revised: We visualize the directed or undirected dataset in Figure 3.

Original: We can find that the distribution shift issue in the directed dataset is more severe than the undirected dataset, which further challenges the unlabeled data sharing algorithms.
Revised: We can ﬁnd that the distribution shift issue in the directed dataset is more severe than the undirected dataset, which further challenges the unlabeled data sharing algorithms.

Original: [Figure]
Revised: D. M ULTI - TASK M ETA -W ORLD

Original: We consider the same setup as in CDS (Yu et al., 2021a) and four tasks: door open , door close , drawer open and drawer close , which is shown in Figure 4.
Revised: We consider the same setup as in CDS (Yu et al., 2021a) and four tasks: door open , door close , drawer open and drawer close , which is shown in Figure 4.

Original: We use medium-replay datasets with 152K transitions for task door open and drawer close and use 10 expert trajectories for task door close and drawer open.
Revised: We use medium-replay datasets with 152K transitions for task door open and drawer close and use 10 expert trajectories for task door close and drawer open.

Original: [Figure]
Revised: [Figure]

Original: F M ORE DISCUSSION ON THE RELATIVE PERFORMANCE OF UDS
Revised: E M ORE DISCUSSION ON THE RELATIVE PERFORMANCE OF UDS

Original: Based on the discussion in the main text, we can summarize the factors that affect the relative performance of PDS algorithms as follows scenarios large number of unlabeled data [Table] unlabeled 
Revised: Based on the discussion in the main text, we can summarize the factors that affect the relative performance of PDS algorithms as follows scenarios large number of unlabeled data high quality of unlabeled data long horizon large dimension ﬁnite-sample term

Original: [Table]
Revised: [Table]

Original: However, it is worth noting that the asymptotic term is dependent on the backbone offline algorithm we choose.
Revised: However, it is worth noting that the asymptotic term is dependent on the backbone ofﬂine algorithm we choose.

Original: For model-based algorithms like Uehara & Sun (2021), the performance bound scale as O ( d ) and thus the problem’s dimension does not affect the asymptotic term.
Revised: For model-based algorithms like Uehara & Sun (2021), the performance bound scale as O ( d ) and thus the problem’s dimension does not affect the asymptotic term.

Original: The dependence over the discount factor is also dependent on the choice of the backbone algorithm.
Revised: The dependence over the discount factor is also dependent on the choice of the backbone algorithm.

Original: However, it is known that the lower bound of offline RL algorithms in linear MDPs scales as O ((1 − γ ) 1 . 5 ) so this asymptotic term must scales at least as O ((1 − γ ) 0 . 5 ) and thus negatively depends on the discount factor.
Revised: However, it is known that the lower bound of ofﬂine RL algorithms in linear MDPs scales as O ((1 − γ ) 1 . 5 ) so this asymptotic term must scales at least as O ((1 − γ ) 0 . 5 ) and thus negatively depends on the discount factor.

Original: That is, the larger the discount factor, the better the relative performance of PDS.
Revised: That is, the larger the discount factor, the better the relative performance of PDS.

Original: G D ISCUSSION AND IMPROVEMENT ABOUT THE PARAMETER k G. H OW TO SELECT THE PARAMETER k ?
Revised: F D ISCUSSION ON THE P ERFORMANCE B OUND F. C ONSTRUCTION OF A DVERSARIAL E XAMPLES

Original: PDS is the generalization of UDS and prediction. We can improve the performance of PDS by selecting the suitable parameter k .
Revised: In this section, we show that there is an MDP and an “unfortunate” dataset such that the suboptimality of Algorithm 3 matches the performance bound in Theorem 4.3.

Original: We give the guidance for selecting k as follows:
Revised: We ﬁrst focus on the case without data sharing.

Original: Suppose the quality of the labeled and unlabeled data is similar (e
Revised: The same techniques can be used to match the reward-learning bound. We only need to show that all inequalities in Theorem 4.3 can become equality. Suppose we have an

Original: g., both labeled and unlabeled data come from the Walker2d/Hopper-medium datasets or AntmazeDirected/Undirected datasets).
Revised: MDP with one state and N > d actions. d of them are optimal actions, with feature map φ ( s, a i ) = (cid:124) (cid:123)(cid:122) (cid:125) i − 1 zeros , √ d, 0 , . . . ) and let the optimal policy be uniform over d actions.

Original: In that case, we find the uncertainty of the reward prediction σ ( s, a ) is low.
Revised: Such construction makes Σ π ∗ ,s = I so that inequalities in Equation (25) ∼ (27) become equalities.

Original: Therefore, we select a smaller k (e.g., k = 5 ). On the other hand, suppose the quality of the labeled and unlabeled data differs significantly.
Revised: Then we let samples from other actions match the conﬁdence upper bound while samples from the optimal action match the conﬁdence lower bound, and we also need all the samples to be symmetric over different dimensions (this is required by the Cauchy-Schwitz inequality used in the proof), such that the conﬁdence bound inequalities in Lemma C.3 also become equalities.

Original: In that case, the uncertainty of the reward prediction is high, andwe will select k according to the quality of the unlabeled data. Specifically, suppose the labeled data is higher quality than the unlabeled data ( e
Revised: Then, in this case, the suboptimality bound is matched and the bound in Theorem 4.3 is tight. F. R EWARD B IAS OF UDS

Original: g.,

Original: la beled data comes from

Original: the Walker2d-Expert

Original: datasets while the unlabeled data comes from the Walker2d-Random datasets).
Revised: In this section, we show that UDS suffers from a constant reward bias.

Original: We select a higher k to penalize the prediction uncertainty (e.g., k = 15 ). On the contrary, suppose the labeled data is lower quality than the unlabeled data (e g., labeled data comes from the Walker2d-Random datasets while the unlabeled data comes from the Walker2d-Expert datasets We select a lower k
Revised: Lemma F.1.

Original: (e.g., k = 10 )

Original: since we prefer the high reward in thehigh-quality unlabeled data. We conduct ablation
Revised: By labeling all rewards in the unlabeled dataset to zeros, we have

Original: studies in

Original: Figure 1 to verify the above statement

Original: G. I S THERE EXISTS A UNIVERSAL CONSTANT THAT WORKS ACROSS DIFFERENT DOMAINS ?

Original: Although we have the guidance about k in the subsection G.1, we need to select the appropriate k for each domain, which is somewhat cumbersome.

Original: To solve this issue, we propose a simple yet efficient automatically parameter-adjustment mechanism.

Original: We first define the reward mean of the labeled data as µ = 1 N 0 (cid:80) N 0 i =1 µ ( s i , a i ) , and the prediction reward mean is denoted as (cid:98) µ = 1 N 1 (cid:80) N 1 i =1 (cid:98) µ ( s i , a i ) .

Original: We observe that the quality-difference between the labeled and unlabeled datasets, is highly susceptible to the scale of µ − (cid:98) µ .

Original: Specifically, compared with the labeled data, if the quality of the unlabeled data is poor, (cid:98) µ is small and µ − (cid:98) µ > 0 .

Original: On the contrary, µ − (cid:98) µ < 0 if the unlabeled data is high quality and the labeled data is low quality.

Original: Based on the above observation, we adopt a linear function related to µ − (cid:98) µ to measure the quality of the unlabeled data.

Original: Next, we use the standard deviation σ ( s, a ) to capture the uncertainty.

Original: In summary, we adopt the following function to automatically adjust k :

Original: [Equation]
Revised: [Equation]

Original: where k = a · max( µ − (cid:98) µ, 0) [Equation] µ [Equation]
Revised: That is, the reward bias does not vanish as long as the ratio of labeled data size and unlabeled data size keeps constant.

Original: Here we use the minimum over L ensembles rather than the mean value to impose the additional pessimistic penalties. We use a = 25 and L = 10 in all experiments. The experimental results in Table 5 demonstrate that the PDS with the automatic parameter-adjustment mechanism achieves strong performance across various domains. The experimental results also show that there is a universal parameter a to adaptively adjust k that generally works better than the baselines.
Revised: Proof.

Original: [Table]
Revised: [Equation]

Original: H H OW BAD IS THE REWARD PREDICTION BASELINE ACTUALLY ?
Revised: G H OW BAD IS THE REWARD PREDICTION BASELINE ACTUALLY ?

Original: We conduct ablation studies for the reward prediction method from three aspects, including model size, ensemble number, and early stopping, which are shown in Table 6, Table 7 and Table 8.
Revised: We conduct ablation studies for the reward prediction method from three aspects, including model size, ensemble number, and early stopping, which are shown in Table 5, Table 6 and Table 7.

Original: For the model size, 256*3 denotes the 256 hidden neurons with three hidden layers.
Revised: For the model size, 256*3 denotes the 256 hidden neurons with three hidden layers.

Original: As for the early stopping, Epoch Number = 3 denotes traversing the entire training dataset 3 times.
Revised: As for the early stopping, Epoch Number = 3 denotes traversing the entire training dataset 3 times.

Original: (We find the
Revised: (We ﬁnd the

Original: Epoch Number=3 is enough to reduce the prediction error to a small range, and increasing epochs leads to overfitting.)
Revised: Epoch Number=3 is enough to reduce the prediction error to a small range, and increasing epochs leads to overﬁtting.)

Original: We conduct experiments in a setting where the quality of the reward labeled and reward-free data differs significantly.
Revised: We conduct experiments in a setting where the quality of the reward labeled and reward-free data differs signiﬁcantly.

Original: For example, walker2d-expert(50K)-random(0.1M) denotes 50K reward-labeled data from expert datasets and 0.1M reward-free data from random datasets.
Revised: For example, walker2d-expert(50K)-random(0.1M) denotes 50K reward-labeled data from expert datasets and 0.1M reward-free data from random datasets.

Original: We set the PDS as the default parameter in all comparisons, including the model size 256*2, the training epoch 3, and the ensemble number 10.
Revised: We set the PDS as the default parameter in all comparisons, including the model size 256*2, the training epoch 3, and the ensemble number 10.

Original: All experimental results adopt the normalized score metric averaged over five seeds with standard deviation.
Revised: All experimental results adopt the normalized score metric averaged over ﬁve seeds with standard deviation.

Original: The experimental results show that fine-tuning reward prediction can improve its performance in some cases, Nevertheless, a well-tuned reward prediction method still performs poorly compared to PDS in general.
Revised: The experimental results show that ﬁne-tuning reward prediction can improve its performance in some cases, Nevertheless, a well-tuned reward prediction method still performs poorly compared to PDS in general.

Original: We hypothesize that this is because the ”test” (reward-free) dataset may have a large distributional shift from the ”training” (reward-labeled) dataset, violating the i.i.d.
Revised: We hypothesize that this is because the ”test” (reward-free) dataset may have a large distributional shift from the ”training” (reward-labeled) dataset, violating the i.i.d.

Original: assumption in supervised learning.
Revised: assumption in supervised learning.

Original: [Table] [Table]
Revised: Model Size (Reward Prediction)

Original: [Table]
Revised: [Equation]

Original: In this section
Revised: PDS

Original: [Table]
Revised: [Table]

Original: we show that UDS suffers from a constant reward bias

Original: [Table]
Revised: [Table]

Original: Lemma I.1.

Original: By labeling all rewards in the unlabeled dataset to zeros, we have

Original: [Equation]
Revised: [Table]

Original: That is, the reward bias does not vanish as long as the ratio of labeled data size and unlabeled data size keeps constant. Proof.
Revised: parameter epoch number

Original: [Equation]
Revised: [Table]