Original: T HE P ROVABLE B ENEFITS OF U NSUPERVISED D ATA S HARING FOR O FFLINE R EINFORCEMENT L EARNING
Revised: T HE P ROVABLE B ENEFITS OF U NSUPERVISED D ATA S HARING FOR O FFLINE R EINFORCEMENT L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations.
Revised: Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations.

Original: However, the question of how to conduct self-supervised ofﬂine reinforcement learning (RL) in a principled way remains unclear.
Revised: However, the question of how to conduct self-supervised offline reinforcement learning (RL) in a principled way remains unclear.

Original: In this paper, we address this issue by investigating the theoretical beneﬁts of utilizing reward free data in linear Markov
Revised: In this paper, we address this issue by investigating the theoretical benefits of utilizing reward-free data in linear Markov

Original: Decision Processes (MDPs) within a semi supervised setting
Revised: Decision Processes (MDPs) within a semi-supervised setting.

Original: Further, we propose a novel, P rovable D ata S haring algorithm PDS ) to utilize such reward
Revised: Further, we propose a novel, P rovable D ata S haring algorithm (PDS) to utilize such reward-free data for offline RL.

Original: -free data for ofﬂine RL.

Original: PDS uses additional penalties on the reward function learned from labeled data to prevent overestimation, ensuring a conservative algorithm.
Revised: PDS uses additional penalties on the reward function learned from labeled data to prevent overestimation, ensuring a conservative algorithm.

Original: Our results on various ofﬂine RL tasks demonstrate that PDS signiﬁcantly improves the performance of ofﬂine RL algorithms with reward -free data.
Revised: Our results on various offline RL tasks demonstrate that PDS significantly improves the performance of offline RL algorithms with reward-free data.

Original: Overall, our work provides a promising approach to leveraging the beneﬁts of unlabeled data in of ﬂine RL while maintaining theoretical guarantees
Revised: Overall, our work provides a promising approach to leveraging the benefits of unlabeled data in offline RL while maintaining theoretical guarantees.

Original: We believe our ﬁndings will contribute to developing more robust self-supervised RL methods
Revised: We believe our findings will contribute to developing more robust self-supervised RL methods

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Ofﬂine reinforcement learning (RL) is a promising framework for learning sequential policies with pre-collected datasets.
Revised: Offline reinforcement learning (RL) is a promising framework for learning sequential policies with pre-collected datasets.

Original: It is highly preferred in many real-world problems where active data collection and exploration is expensive, unsafe, or infeasible (Swaminathan & Joachims, 2015;
Revised: It is highly preferred in many real-world problems where active data collection and exploration is expensive, unsafe, or infeasible (Swaminathan & Joachims, 2015;

Original: ShalevShwartz et al., 2016; Singh et al., 2020).
Revised: ShalevShwartz et al., 2016; Singh et al., 2020).

Original: However, labeling large datasets with rewards can be costly and require signiﬁcant human effort (Singh et al., 2019;
Revised: However, labeling large datasets with rewards can be costly and require significant human effort (Singh et al., 2019;

Original: Wirth et al., 2017).
Revised: Wirth et al., 2017).

Original: In contrast, unlabeled data can be cheap and abundant, making self-supervised learning with unlabeled data an attractive alternative.
Revised: In contrast, unlabeled data can be cheap and abundant, making self-supervised learning with unlabeled data an attractive alternative.

Original: While self-supervised methods have achieved great success in computer vision and natural language processing tasks (Brown et al., 2020; Devlin et al., 2018; Chen et al., 2020), their potential beneﬁts in ofﬂine RL are less explored.
Revised: While self-supervised methods have achieved great success in computer vision and natural language processing tasks (Brown et al., 2020; Devlin et al., 2018; Chen et al., 2020), their potential benefits in offline RL are less explored.

Original: Several prior works (Ho & Ermon, 2016; Reddy et al., 2019;
Revised: Several prior works (Ho & Ermon, 2016; Reddy et al., 2019;

Original: Kostrikov et al., 2019) have explored demonstration-based approaches to eliminate the need for reward annotations, but these approaches require samples to be near-optimal.
Revised: Kostrikov et al., 2019) have explored demonstration-based approaches to eliminate the need for reward annotations, but these approaches require samples to be near-optimal.

Original: Another line of work focuses on data sharing from different datasets (Kalashnikov et al., 2021; Yu et al., 2021a).
Revised: Another line of work focuses on data sharing from different datasets (Kalashnikov et al., 2021; Yu et al., 2021a).

Original: Still, it assumes that the dataset can be relabeled with oracle reward functions for the target task.
Revised: Still, it assumes that the dataset can be relabeled with oracle reward functions for the target task.

Original: These settings can be unrealistic in real-world problems where expert trajectories and reward labeling are expensive.
Revised: These settings can be unrealistic in real-world problems where expert trajectories and reward labeling are expensive.

Original: Incorporating reward-free datasets into ofﬂine RL is important but challenging due to the sequential and dynamic nature of RL problems.
Revised: Incorporating reward-free datasets into offline RL is important but challenging due to the sequential and dynamic nature of RL problems.

Original: Prior work (Yu et al., 2022) has shown that learning to predict rewards can be difﬁcult, and simply setting the reward to zero can achieve good results.
Revised: Prior work (Yu et al., 2022) has shown that learning to predict rewards can be difficult, and simply setting the reward to zero can achieve good results.

Original: However, it’s unclear how reward-prediction methods affect performance and whether reward-free data can provably beneﬁt ofﬂine RL.
Revised: However, it’s unclear how reward-prediction methods affect performance and whether reward-free data can provably benefit offline RL.

Original: This naturally leads to the following question:
Revised: This naturally leads to the following question:

Original: How can we leverage reward-free data to improve the performance of ofﬂine RL algorithms in a principled way?
Revised: How can we leverage reward-free data to improve the performance of offline RL algorithms in a principled way?

Revised: contribution.

Original: To answer this question, we conduct a theoretical analysis of the beneﬁts of utilizing unlabeled data in linear MDPs.
Revised: To answer this question, we conduct a theoretical analysis of the benefits of utilizing unlabeled data in linear MDPs.

Original: Our analysis reveals that although unlabeled data can provide information about the dynamics of the MDP, it cannot reduce the uncertainty over reward functions.
Revised: Our analysis reveals that although unlabeled data can provide information about the dynamics of the MDP, it cannot reduce the uncertainty over reward functions.

Original: Based on this insight, we propose a model-free method named Provable Data Sharing (PDS), which adds uncertainty penalties to the learned reward functions to maintain a conservative algorithm.
Revised: Based on this insight, we propose a model-free method named Provable Data Sharing (PDS), which adds uncertainty penalties to the learned reward functions to maintain a conservative algorithm.

Original: By doing so,
Revised: By doing so,

Original: PDS can effectively leverage the beneﬁts of unlabeled data for ofﬂine RL while ensuring theoretical guarantees.
Revised: PDS can effectively leverage the benefits of unlabeled data for offline RL while ensuring theoretical guarantees.

Original: We demonstrate that PDS can cooperate with model-free ofﬂine methods while being simple and efﬁcient.
Revised: We demonstrate that PDS can cooperate with model-free offline methods while being simple and efficient.

Original: We conduct extensive experiments on various environments, including single-task domains like MuJoCo (Todorov et al., 2012) and Kitchen (Gupta et al., 2019), as well as multi-task domains like AntMaze and Meta-World (Yu et al., 2020a).
Revised: We conduct extensive experiments on various environments, including single-task domains like MuJoCo (Todorov et al., 2012) and Kitchen (Gupta et al., 2019), as well as multi-task domains like AntMaze and Meta-World (Yu et al., 2020a).

Original: The results show that PDS improves signiﬁcantly over previous methods like UDS (Yu et al., 2022) and naive reward prediction methods.
Revised: The results show that PDS improves significantly over previous methods like UDS (Yu et al., 2022) and naive reward prediction methods.

Original: Our main contribution is the Provable Data Sharing (PDS) algorithm, a novel method for utilizing unsupervised data in ofﬂine RL that provides theoretical guarantees.
Revised: Our main contribution is the Provable Data Sharing (PDS) algorithm, a novel method for utilizing unsupervised data in offline RL that provides theoretical guarantees.

Original: PDS adds uncertainty penalties to the learned reward functions and can be easily integrated with existing ofﬂine RL algorithms.
Revised: PDS adds uncertainty penalties to the learned reward functions and can be easily integrated with existing offline RL algorithms.

Original: Our experimental results demonstrate that PDS can achieve superior performance on various locomotion, navigation, and manipulation tasks.
Revised: Our experimental results demonstrate that PDS can achieve superior performance on various locomotion, navigation, and manipulation tasks.

Original: Overall, our work provides a promising approach to leveraging the beneﬁts of unlabeled data in ofﬂine RL while maintaining theoretical guarantees and contributing to the development of more robust self-supervised RL methods.
Revised: Overall, our work provides a promising approach to leveraging the benefits of unlabeled data in offline RL while maintaining theoretical guarantees and contributing to the development of more robust self-supervised RL methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Ofﬂine Reinforcement Learning Current ofﬂine RL methods (Levine et al., 2020) can be roughly divided into policy constraint-based, uncertainty estimation-based, and model-based approaches.
Revised: Offline Reinforcement Learning Current offline RL methods (Levine et al., 2020) can be roughly divided into policy constraint-based, uncertainty estimation-based, and model-based approaches.

Original: Policy constraint methods aim to keep the policy close to the behavior under a probabilistic distance (Siegel et al., 2020;
Revised: Policy constraint methods aim to keep the policy close to the behavior under a probabilistic distance (Siegel et al., 2020;

Original: Yang et al., 2021; Kumar et al., 2020; Fujimoto & Gu, 2021;
Revised: Yang et al., 2021; Kumar et al., 2020; Fujimoto & Gu, 2021;

Original: Yang et al., 2022; Fujimoto et al., 2019; Hu et al., 2022; Kostrikov et al., 2021; Ma et al.,  2021).
Revised: Yang et al., 2022; Fujimoto et al., 2019; Hu et al., 2022; Kostrikov et al., 2021; Ma et al., 2021; Wang et al., 2021).

Original: Uncertainty estimation-based methods attempt to consider the Q-value prediction’s conﬁdence using dropout or ensemble techniques (An et al., 2021; Wu et al., 2021).
Revised: Uncertainty estimation-based methods attempt to consider the Q-value prediction’s confidence using dropout or ensemble techniques (An et al., 2021; Wu et al., 2021).

Original: Last, model-based methods incorporates the uncertainty in the model space for conservative ofﬂine learning (Yu et al., 2020b; 2021b; Kidambi et al., 2020).
Revised: Last, model-based methods incorporates the uncertainty in the model space for conservative offline learning (Yu et al., 2020b; 2021b; Kidambi et al., 2020).

Original: Ofﬂine Data Sharing Prior works have demonstrated that data sharing across tasks can be beneﬁcial by designing sophisticated data-sharing protocols (Yu et al., 2021a).
Revised: Offline Data Sharing Prior works have demonstrated that data sharing across tasks can be beneficial by designing sophisticated data-sharing protocols (Yu et al., 2021a).

Original: For example, previous studies have explored developing data sharing strategies by human effort (Kalashnikov et al., 2021), inverse RL (Reddy et al., 2019; Li et al., 2020), and estimated Q-values (Yu et al., 2021a).
Revised: For example, previous studies have explored developing data sharing strategies by human effort (Kalashnikov et al., 2021), inverse RL (Reddy et al., 2019; Li et al., 2020), and estimated Q-values (Yu et al., 2021a).

Original: However, these data sharing works must assume the dataset can be relabeled with oracle rewards for the target task, which is a strong assumption since the high cost of reward labeling.
Revised: However, these data sharing works must assume the dataset can be relabeled with oracle rewards for the target task, which is a strong assumption since the high cost of reward labeling.

Original: Therefore, effectively incorporating the unlabeled data into the ofﬂine RL algorithms is essential.
Revised: Therefore, effectively incorporating the unlabeled data into the offline RL algorithms is essential.

Original: To solve this issue, some recent work (Yu et al., 2022) proposes simply applying zero rewards to unlabeled data.
Revised: To solve this issue, some recent work (Yu et al., 2022) proposes simply applying zero rewards to unlabeled data.

Original: In this work, we propose a principle way to leverage unlabeled data without the strong assumption of reward relabeling.
Revised: In this work, we propose a principle way to leverage unlabeled data without the strong assumption of reward relabeling.

Original: Reward Prediction It is widely observed that reward shaping and intrinsic rewards can accelerate learning in online RL (Mataric, 1994;
Revised: Reward Prediction It is widely observed that reward shaping and intrinsic rewards can accelerate learning in online RL (Mataric, 1994;

Original: Ng et al., 1999; Wu & Tian, 2017; Song et al., 2019;
Revised: Ng et al., 1999; Wu & Tian, 2017; Song et al., 2019;

Original: Guo et al., 2016; Abel et al., 2021).
Revised: Guo et al., 2016; Abel et al., 2021).

Original: There are also extensive works that studies automatically designing reward functions using inverse RL (Ng et al., 2000; Fu et al., 2017).
Revised: There are also extensive works that studies automatically designing reward functions using inverse RL (Ng et al., 2000; Fu et al., 2017).

Original: However, there is less attention on the ofﬂine setting where online interaction is not allowed and the trajectories may not be optimal.
Revised: However, there is less attention on the offline setting where online interaction is not allowed and the trajectories may not be optimal.

Original:  P RELIMINARIES 3. L INEAR MDP S AND P ERFORMANCE M ETRIC
Revised:  P RELIMINARIES 3. L INEAR MDP S AND P ERFORMANCE M ETRIC

Original: We consider inﬁnite-horizon discounted Markov Decision Processes (MDPs), deﬁned by the tuple ( S , A , P , r, γ ) , with state space S , action space A , discount factor γ ∈ [0 , 1) , transition function
Revised: We consider infinite-horizon discounted Markov Decision Processes (MDPs), defined by the tuple ( S , A , P , r, γ ) , with state space S , action space A , discount factor γ ∈ [0 , 1) , transition function

Original: P : S ×
Revised: P : S ×

Original: A → ∆( S ) , and reward function r : S ×
Revised: A → ∆( S ) , and reward function r : S ×

Original: A → [0 , r max ] .
Revised: A → [0 , r max ] .

Original: To make things more concrete, we consider the linear MDP (Yang & Wang, 2019; Jin et al., 2020) as follows, where the transition kernel and expected reward function are linear with respect to a feature map.
Revised: To make things more concrete, we consider the linear MDP (Yang & Wang, 2019; Jin et al., 2020) as follows, where the transition kernel and expected reward function are linear with respect to a feature map.

Original: Deﬁnition 3.1 (Linear MDP) .
Revised: Definition 3.1 (Linear MDP) .

Original: We say an episodic MDP ( S , A , P , r, γ ) is a linear MDP with known feature map φ : S ×
Revised: We say an episodic MDP ( S , A , P , r, γ ) is a linear MDP with known feature map ϕ : S ×

Original: A → R d if there exist unknown measures µ = ( µ 1 , . . . , µ d ) over S and an unknown vector θ ∈ R d such that
Revised: A → R d if there exist unknown measures µ = ( µ 1 , . . . , µ d ) over S and an unknown vector θ ∈ R d such that

Original: [Equation]
Revised: [Equation]

Original: for all ( s, a, s (cid:48) ) ∈ S ×
Revised: for all ( s, a, s ′ ) ∈ S ×

Original: A × S .
Revised: A × S .

Original: And we assume (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 for all ( s, a, s (cid:48) )
Revised: And we assume ∥ ϕ ( s, a ) ∥ 2 ≤ 1 for all ( s, a, s ′ )

Original: ∈ S ×
Revised: ∈ S ×

Original: A × S and max {(cid:107) µ ( S ) (cid:107) 2 , (cid:107) θ (cid:107) 2 } ≤ √ d , where (cid:107) µ ( S ) (cid:107) ≡ (cid:82) S (cid:107) µ ( s )(cid:107) ds .
Revised: A × S and max {∥ µ ( S ) ∥ 2 , ∥ θ ∥ 2 } ≤ √ d , where ∥ µ ( S ) ∥ ≡ (cid:82) S ∥ µ ( s )

Original:  A policy π : S → ∆( A ) speciﬁes a decision-making strategy in which the agent chooses actions adaptively based on the current state, i.e., a t ∼ π ( · | s t ) .
Revised: ∥ ds . A policy π : S → ∆( A ) specifies a decision-making strategy in which the agent chooses actions adaptively based on the current state, i.e., a t ∼ π ( · | s t ) .

Original: The value function V π : S → R and the action-value function (Q-function) Q π :
Revised: The value function V π : S → R and the action-value function (Q-function) Q π :

Original: S ×
Revised: S ×

Original: A → R are deﬁned as
Revised: A → R are defined as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is with respect to the trajectory τ induced by policy π .
Revised: where the expectation is with respect to the trajectory τ induced by policy π .

Original: We deﬁne the Bellman operator as
Revised: We define the Bellman operator as

Original: [Equation]
Revised: [Equation]

Original: We use π ∗ , Q ∗ , and V ∗ to denote the optimal policy, optimal Q-function, and optimal value function, respectively.
Revised: We use π ∗ , Q ∗ , and V ∗ to denote the optimal policy, optimal Q-function, and optimal value function, respectively.

Original: We have the Bellman optimality equation
Revised: We have the Bellman optimality equation

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the optimal policy π ∗ satisﬁes
Revised: Meanwhile, the optimal policy π ∗ satisfies

Original: [Equation]
Revised: [Equation]

Original: where the maximum is taken over all functions mapping from S to distributions over A .
Revised: where the maximum is taken over all functions mapping from S to distributions over A .

Original: We aim to learn a policy that maximizes the expected cumulative reward.
Revised: We aim to learn a policy that maximizes the expected cumulative reward.

Original: Correspondingly, we deﬁne the performance metric as
Revised: Correspondingly, we define the performance metric as

Original: [Equation]
Revised: [Equation]

Original: 3. P ROVABLE O FFLINE A LGORITHMS
Revised: 3. P ROVABLE O FFLINE A LGORITHMS

Original: In this section, we consider pessimistic value iteration (PEVI; Jin et al., 2021) as the backbone algorithm, described in Algorithm 2.
Revised: In this section, we consider pessimistic value iteration (PEVI; Jin et al., 2021) as the backbone algorithm, described in Algorithm 2.

Original: It is a representative model-free ofﬂine algorithm with theoretical guarantees.
Revised: It is a representative model-free offline algorithm with theoretical guarantees.

Original: PEVI uses negative bonus Γ( · , · ) over standard Q -value estimation (cid:98) Q ( · , · ) =
Revised: PEVI uses negative bonus Γ( · , · ) over standard Q -value estimation (cid:98) Q ( · , · ) =

Original: ( (cid:98) B (cid:98) V )( · ) to reduce potential bias due to ﬁnite data, where (cid:98) B is some empirical estimation of B from dataset D .
Revised: ( (cid:98) B (cid:98) V )( · ) to reduce potential bias due to finite data, where (cid:98) B is some empirical estimation of B from dataset D .

Original: Please refer to Appendix A.1 for more details of the PEVI algorithm.
Revised: Please refer to Appendix A.1 for more details of the PEVI algorithm.

Original: We use the following notion of ξ -uncertainty quantiﬁer as follows to formalize the idea of pessimism.
Revised: We use the following notion of ξ -uncertainty quantifier as follows to formalize the idea of pessimism.

Original: Deﬁnition 3.2 ( ξ -Uncertainty Quantiﬁer) .
Revised: Definition 3.2 ( ξ -Uncertainty Quantifier) .

Original: We say Γ : S ×
Revised: We say Γ : S ×

Original: A → R is a ξ -uncertainty quantiﬁer for (cid:98) B and (cid:98) V if with probability 1 − ξ , for all ( s, a ) ∈ S × A ,[Equation]
Revised: A → R is a ξ -uncertainty quantifier for (cid:98) B and (cid:98) V if with probability 1 − ξ , for all ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: 3. U NSUPERVISED D ATA S HARING
Revised: 3. U NSUPERVISED D ATA S HARING

Original: We consider unsupervised data sharing in ofﬂine reinforcement learning.
Revised: We consider unsupervised data sharing in offline reinforcement learning.

Original: We ﬁrst characterize the quality of the dataset with the notion of coverage coefﬁcient (Uehara & Sun, 2021), deﬁned as below.
Revised: We first characterize the quality of the dataset with the notion of coverage coefficient (Uehara & Sun, 2021), defined as below.

Original: Deﬁnition 3.3.
Revised: Definition 3.3.

Original: The coverage coefﬁcient C † of a dataset D = { ( s τ , a τ , r τ ) }
Revised: The coverage coefficient C † of a dataset D = { ( s τ , a τ , r τ ) }

Original: Nτ =1 is deﬁned as
Revised: Nτ =1 is defined as

Original: [Equation]
Revised: [Equation]

Original: The coverage coefﬁcient C † is common in ofﬂine RL literature (Uehara & Sun, 2021; Jin et al., 2021;
Revised: The coverage coefficient C † is common in offline RL literature (Uehara & Sun, 2021; Jin et al., 2021;

Original: Rashidinejad et al., 2021), which represents the maximum ratio between the density of empirical state-action distribution and the density induced from the optimal policy.
Revised: Rashidinejad et al., 2021), which represents the maximum ratio between the density of empirical state-action distribution and the density induced from the optimal policy.

Original: Intuitively, it represents the quality of the dataset.
Revised: Intuitively, it represents the quality of the dataset.

Original: For example, the expert dataset has a high coverage ratio while the random dataset may have a low ratio.
Revised: For example, the expert dataset has a high coverage ratio while the random dataset may have a low ratio.

Original: We denote D 0 as the origin labeled dataset, with coverage coefﬁcient C † 0 and size N 0 .
Revised: We denote D 0 as the origin labeled dataset, with coverage coefficient C † 0 and size N 0 .

Original: And we denote the unlabeled dataset as D 1 , with coverage coefﬁcient C † 1 and size N 1 .
Revised: And we denote the unlabeled dataset as D 1 , with coverage coefficient C † 1 and size N 1 .

Original: Note that it is possible that the unlabeled data comes from multiple sources, such as multi-task settings, and we still use D 1 to represent the combined dataset ∪ Mi =1 D i from M tasks for simplicity.
Revised: Note that it is possible that the unlabeled data comes from multiple sources, such as multi-task settings, and we still use D 1 to represent the combined dataset ∪ Mi =1 D i from M tasks for simplicity.

Original:  P ROVABLE U NSUPERVISED D ATA S HARING
Revised:  P ROVABLE U NSUPERVISED D ATA S HARING

Original: How can we leverage reward-free data for ofﬂine RL?
Revised: How can we leverage reward-free data for offline RL?

Original: A naive approach is to learn the reward function from labeled data via the following regression
Revised: A naive approach is to learn the reward function from labeled data via the following regression

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where f θ ( s τ , a τ ) =
Revised: where f θ ( s τ , a τ ) =

Original: φ ( s τ , a τ ) (cid:62) θ in linear MDPs.
Revised: ϕ ( s τ , a τ ) ⊤ θ in linear MDPs.

Original: Then we can use this learned reward function to label unsupervised data.
Revised: Then we can use this learned reward function to label unsupervised data.

Original: However, this approach can lead to suboptimal performance due to overestimation of the predicted reward r (cid:98) θ (Yu et al., 2022), which undermines the pessimism in ofﬂine algorithms.
Revised: However, this approach can lead to suboptimal performance due to overestimation of the predicted reward r (cid:98) θ (Yu et al., 2022), which undermines the pessimism in offline algorithms.

Original: To address this issue, we propose a data-sharing algorithm called Provable Data Sharing (PDS).
Revised: To address this issue, we propose a data-sharing algorithm called Provable Data Sharing (PDS).

Original: We start by analyzing the uncertainty in learned reward functions and add penalties for such uncertainty to leverage unlabeled data.
Revised: We start by analyzing the uncertainty in learned reward functions and add penalties for such uncertainty to leverage unlabeled data.

Original: In Section 4.2, we show that PDS has a provable performance bound consisting of two parts: the ofﬂine error, which is tightened compared to no data sharing due to additional data, and the error from reward bias.
Revised: In Section 4.2, we show that PDS has a provable performance bound consisting of two parts: the offline error, which is tightened compared to no data sharing due to additional data, and the error from reward bias.

Original: The performance bound of PDS is provably better than no data sharing as long as the unlabeled dataset has mediocre size or quality.
Revised: The performance bound of PDS is provably better than no data sharing as long as the unlabeled dataset has mediocre size or quality.

Original: We also extend our algorithm in linear MDPs to general settings in Section 4.3 and propose using ensembles for reward uncertainty estimation.
Revised: We also extend our algorithm in linear MDPs to general settings in Section 4.3 and propose using ensembles for reward uncertainty estimation.

Original: We demonstrate the effectiveness of PDS by integrating it with
Revised: We demonstrate the effectiveness of PDS by integrating it with

Original: IQL (Kostrikov et al., 2021), and present Algorithm 3, which is simple and can be easily integrated with other model-free ofﬂine algorithms.
Revised: IQL (Kostrikov et al., 2021), and present Algorithm 3, which is simple and can be easily integrated with other model-free offline algorithms.

Original: 4. P ROVABLE D ATA S HARING
Revised: 4. P ROVABLE D ATA S HARING

Original: To address the issue of potential overestimation of predicted rewards, we ﬁrst analyze the uncertainty in learned reward functions.
Revised: To address the issue of potential overestimation of predicted rewards, we first analyze the uncertainty in learned reward functions.

Original: In the context of linear MDPs, the reward function can be learned via linear regression, and the uncertainty of the parameters is characterized by the elliptical conﬁdence region, as shown in Lemma 4.1.
Revised: In the context of linear MDPs, the reward function can be learned via linear regression, and the uncertainty of the parameters is characterized by the elliptical confidence region, as shown in Lemma 4.1.

Original: This conﬁdence region is important as it allows us to give a more accurate estimation of the reward function while keeping the overall algorithm pessimistic.
Revised: This confidence region is important as it allows us to give a more accurate estimation of the reward function while keeping the overall algorithm pessimistic.

Original: Lemma 4.1 ( Abbasi-Yadkori et al.
Revised: Lemma 4.1 ( Abbasi-Yadkori et al.

Original: Let α = √ ν + r max · (cid:113) 2 log 1 δ + d log (1 + N 0 νd ) , Λ = νI+ (cid:80) [Equation] = [Equation]
Revised: Let α = √ ν + r max · (cid:113) 2 log 1 δ + d log (1 + N 0 νd ) , Λ = νI

Original: [Equation]

Original: τ

Original: [Equation]
Revised: [Equation]

Original: (cid:62) ,
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: (8) where (cid:98) θ is the minimizer in Equation (7) , then we have P ( θ (cid:63) ∈ C ( δ )) ≥ 1 − δ , where θ (cid:63) is the true parameter for the reward function.
Revised: (8) where (cid:98) θ is the minimizer in Equation (7) , then we have P ( θ ⋆ ∈ C ( δ )) ≥ 1 − δ , where θ ⋆ is the true parameter for the reward function.

Original: Proof.
Revised: Proof.

Original: Please refer to Theorem 2 in Abbasi-Yadkori et al.
Revised: Please refer to Theorem 2 in Abbasi-Yadkori et al.

Original: (2011) for detailed proof.
Revised: (2011) for detailed proof.

Original: Lemma 4.1 provides a useful insight: the uncertainty of the learned reward function in linear MDPs only depends on the quality and size of labeled data.
Revised: Lemma 4.1 provides a useful insight: the uncertainty of the learned reward function in linear MDPs only depends on the quality and size of labeled data.

Original: Based on this insight, we propose a two-phase algorithm that guarantees a provable performance bound, as shown in Theorem 4.3.
Revised: Based on this insight, we propose a two-phase algorithm that guarantees a provable performance bound, as shown in Theorem 4.3.

Original: The simple reward prediction method can compromise the pessimistic estimation of the algorithm, while UDS may result in a reward bias that is too large.
Revised: The simple reward prediction method can compromise the pessimistic estimation of the algorithm, while UDS may result in a reward bias that is too large.

Original: Our algorithm consists of two phases: in the ﬁrst phase, we construct a pessimistic reward estimator that ﬁnds the reward function in the conﬁdence set that leads to the lowest optimal value.
Revised: Our algorithm consists of two phases: in the first phase, we construct a pessimistic reward estimator that finds the reward function in the confidence set that leads to the lowest optimal value.

Original: In the second phase, we conduct standard ofﬂine RL with the given pessimistic reward function.
Revised: In the second phase, we conduct standard offline RL with the given pessimistic reward function.

Original: To solve the challenge of ﬁnding the best parameter in the conﬁdence set, which is a bi-level optimization problem, we propose using a simpler method that maintains the pessimistic property of the ofﬂine algorithms.
Revised: To solve the challenge of finding the best parameter in the confidence set, which is a bi-level optimization problem, we propose using a simpler method that maintains the pessimistic property of the offline algorithms.

Original: This method involves using a pessimistic estimation, which allows us to keep the algorithm pessimistic while avoiding the computational challenges of the bi-level optimization problem.
Revised: This method involves using a pessimistic estimation, which allows us to keep the algorithm pessimistic while avoiding the computational challenges of the bi-level optimization problem.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where Λ = νI + (cid:80) N 0 τ =1 φ ( s τ , a τ ) φ ( s τ , a τ ) (cid:62) .
Revised: where Λ = νI + (cid:80) N 0 τ =1 ϕ ( s τ , a τ ) ϕ ( s τ , a τ ) ⊤ .

Original: We adopt the pessimistic estimation in Equation (9) because it provides a lower bound for reward functions in the conﬁdence set C ( δ ) , as guaranteed by the following lemma derived from CauchySchwartz inequalities.
Revised: We adopt the pessimistic estimation in Equation (9) because it provides a lower bound for reward functions in the confidence set C ( δ ) , as guaranteed by the following lemma derived from CauchySchwartz inequalities.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: For any θ ∈ C ( δ ) ,
Revised: For any θ ∈ C ( δ ) ,

Original: [Equation]
Revised: [Equation]

Original: When labeled data is scarce or there is a signiﬁcant distributional shift between the labeled and unlabeled data, Equation (9) degenerates to 0, which is equivalent to the UDS algorithm (Yu et al., 2022). 
Revised: When labeled data is scarce or there is a significant distributional shift between the labeled and unlabeled data, Equation (9) degenerates to 0, which is equivalent to the UDS algorithm (Yu et al., 2022). 1: Require : Labeled dataset

Original: MDP
Revised: , a N 0 τ = , unlabeled dataset D 1 = { [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) V θ is the estimated value function from Algorithm 2 and dataset D 0 ∪ D 1 with reward relabeled with parameter θ . 
Revised: where (cid:98) V θ is the estimated value function from Algorithm 2 and dataset D 0 ∪ D 1 with reward relabeled with parameter θ . 1 with parameter θ from the dataset

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 4. T HEORETICAL A NALYSIS
Revised: 4. T HEORETICAL A NALYSIS

Original: The following subsection analyzes how the provable data-sharing (PDS) algorithm can enhance the performance bound by leveraging unlabeled data.
Revised: The following subsection analyzes how the provable data-sharing (PDS) algorithm can enhance the performance bound by leveraging unlabeled data.

Original: To be speciﬁc, we present the following theorem.
Revised: To be specific, we present the following theorem.

Original: Theorem 4.3 (Performance Bound for PDS) .
Revised: Theorem 4.3 (Performance Bound for PDS) .

Original: Suppose the dataset D 0 , D 1 have positive coverage coefﬁcients C † 0 , C † 1 , and the underlying MDP is a linear MDP.
Revised: Suppose the dataset D 0 , D 1 have positive coverage coefficients C † 0 , C † 1 , and the underlying MDP is a linear MDP.

Original: In Algorithm 1, we set λ
Revised: In Algorithm 1, we set λ

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is an absolute constant and δ ∈ (0 , 1) is the conﬁdence parameter.
Revised: where c > 0 is an absolute constant and δ ∈ (0 , 1) is the confidence parameter.

Original: Then with probability 1 − 2 δ , the policy (cid:98) π generated by PDS satisﬁes for all s ∈ S ,
Revised: Then with probability 1 − 2 δ , the policy (cid:98) π generated by PDS satisfies for all s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Please refer to Appendix B for detailed proof.
Revised: Please refer to Appendix B for detailed proof.

Original: The performance bound of PDS is composed of two terms.
Revised: The performance bound of PDS is composed of two terms.

Original: The ﬁrst is the ofﬂine error, which is inherited from ofﬂine algorithms.
Revised: The first is the offline error, which is inherited from offline algorithms.

Original: This bound is improved when additional unlabeled data with size
Revised: This bound is improved when additional unlabeled data with size

Original: N 1 and coverage C † 1 is available.
Revised: N 1 and coverage C † 1 is available.

Original: The second term is the reward bias, which arises due to uncertainties in the rewards.
Revised: The second term is the reward bias, which arises due to uncertainties in the rewards.

Original: Notably, this term is equivalent to the performance bound of a linear bandit with rewards in the range [0 , r max / (1 − γ )] .
Revised: Notably, this term is equivalent to the performance bound of a linear bandit with rewards in the range [0 , r max / (1 − γ )] .

Original: As the number of unlabeled data approaches inﬁnity, the uncertainty of the dynamics decreases to zero, and the RL problem becomes a linear bandit problem.
Revised: As the number of unlabeled data approaches infinity, the uncertainty of the dynamics decreases to zero, and the RL problem becomes a linear bandit problem.

Original: The theorem demonstrates that PDS outperforms UDS, which suffers from a constant reward bias, and naive reward prediction methods, which lack pessimism and therefore do not offer such guarantees.
Revised: The theorem demonstrates that PDS outperforms UDS, which suffers from a constant reward bias, and naive reward prediction methods, which lack pessimism and therefore do not offer such guarantees.

Original: Moreover, we demonstrate the tightness of the bound by constructing an “adversarial” dataset that matches the bound’s suboptimality (see Appendix F).
Revised: Moreover, we demonstrate the tightness of the bound by constructing an “adversarial” dataset that matches the bound’s suboptimality (see Appendix F).

Original: To better understand the beneﬁts of unlabeled data, we deﬁne the suboptimality bound ratio (SBR) of an ofﬂine algorithm A as the ratio of the suboptimality bound obtained by the policy learned with additional unlabeled data to the suboptimality bound of the policy learned using labeled data alone.
Revised: To better understand the benefits of unlabeled data, we define the suboptimality bound ratio (SBR) of an offline algorithm A as the ratio of the suboptimality bound obtained by the policy learned with additional unlabeled data to the suboptimality bound of the policy learned using labeled data alone.

Original: Mathematically, the SBR of A is given by:
Revised: Mathematically, the SBR of A is given by:

Original: [Equation]
Revised: [Equation]

Original: where SubOpt is the tight upper bound on suboptimality.
Revised: where SubOpt is the tight upper bound on suboptimality.

Original: The SBR provides a measure of the beneﬁt of unlabeled data to the ofﬂine algorithm, with a smaller SBR indicating a greater beneﬁt from the unlabeled data.
Revised: The SBR provides a measure of the benefit of unlabeled data to the offline algorithm, with a smaller SBR indicating a greater benefit from the unlabeled data.

Original: Applying this deﬁnition to PDS, we obtain the following corollary.
Revised: Applying this definition to PDS, we obtain the following corollary.

Original: Corollary 4.4 (Informal) .
Revised: Corollary 4.4 (Informal) .

Original: The SBR of PDS satisﬁes
Revised: The SBR of PDS satisfies

Original: [Equation]
Revised: [Equation]

Original: where c is the constant in Theorem 4.3 and we ignore the logarithmic factors.
Revised: where c is the constant in Theorem 4.3 and we ignore the logarithmic factors.

Original: When does unlabeled data improve the performance of ofﬂine algorithms?
Revised: When does unlabeled data improve the performance of offline algorithms?

Original: Corollary 4.4 allows us to analyze the relative performance of PDS under different conditions.
Revised: Corollary 4.4 allows us to analyze the relative performance of PDS under different conditions.

Original: The ﬁrst term of the bound depends on the qualities and amounts of both labeled and unlabeled datasets.
Revised: The first term of the bound depends on the qualities and amounts of both labeled and unlabeled datasets.

Original: If the unlabeled dataset has a mediocre number of samples or data quality, the ﬁrst term will be sufﬁciently small.
Revised: If the unlabeled dataset has a mediocre number of samples or data quality, the first term will be sufficiently small.

Original: The second term affects the asymptotic performance when the unlabeled data approaches inﬁnity, and it depends on the discount factor and the dimension of the problem.
Revised: The second term affects the asymptotic performance when the unlabeled data approaches infinity, and it depends on the discount factor and the dimension of the problem.

Original: PDS improves over no data-sharing algorithms asymptotically in larger problems or longer horizons.
Revised: PDS improves over no data-sharing algorithms asymptotically in larger problems or longer horizons.

Original: For a more detailed discussion, please refer to Appendix E.
Revised: For a more detailed discussion, please refer to Appendix E.

Original: 4. P RACTICAL I MPLEMENTATION
Revised: 4. P RACTICAL I MPLEMENTATION

Original: This subsection outlines the practical implementation of PDS in general MDPs.
Revised: This subsection outlines the practical implementation of PDS in general MDPs.

Original: We employ L ensembles θ 1 , . .
Revised: We employ L ensembles θ 1 , . .

Original: , θ L to estimate uncertainty, which are learned using Equation (7).
Revised: , θ L to estimate uncertainty, which are learned using Equation (7).

Original: To estimate pessimistic rewards, we use the following pessimistic estimation:
Revised: To estimate pessimistic rewards, we use the following pessimistic estimation:

Original: [Equation]
Revised: [Equation]

Original:  where µ ( s, a ) =
Revised: 15) where µ ( s, a ) =

Original: 1 L (cid:80) Li =
Revised: 1 L (cid:80) Li =

Original: 1 f θ i ( s, a ) , σ ( s, a ) = (cid:113) 1 L (cid:80) Li
Revised: 1 f θ i ( s, a ) , σ ( s, a ) = (cid:113) 1 L (cid:80) Li

Original: =

Revised: =

Original: ( f θ i ( s, a ) − µ ( s, a )) 2 are the mean and standard deviation, respectively.
Revised: ( f θ i ( s, a ) − µ ( s, a )) 2 are the mean and standard deviation, respectively.

Original: Here, k is a hyperparameter used to control the amount of pessimism.
Revised: Here, k is a hyperparameter used to control the amount of pessimism.

Original: We can also use the minimum over L ensembles for the pessimistic estimation, which is linked to Equation (15) following An et al.
Revised: We can also use the minimum over L ensembles for the pessimistic estimation, which is linked to Equation (15) following An et al.

Original: ; Royston (1982) as shown in Equation (16):
Revised: ; Royston (1982) as shown in Equation (16):

Original: [Equation]
Revised: [Equation]

Original: where Φ is the CDF of the standard Gaussian distribution.
Revised: where Φ is the CDF of the standard Gaussian distribution.

Original: The appropriate value of k for each domain can be difﬁcult to determine.
Revised: The appropriate value of k for each domain can be difficult to determine.

Original: To address this issue, we observe that the amount of pessimism required for different domains is proportional to the difference in mean rewards between labeled and unlabeled data.
Revised: To address this issue, we observe that the amount of pessimism required for different domains is proportional to the difference in mean rewards between labeled and unlabeled data.

Original: Leveraging this insight, we propose a simple and efﬁcient automatic mechanism for adjusting the value of the k parameter.
Revised: Leveraging this insight, we propose a simple and efficient automatic mechanism for adjusting the value of the k parameter.

Original: Speciﬁcally, we suggest a method that adjusts k based on the difference in mean rewards, as given by Equation (17):
Revised: Specifically, we suggest a method that adjusts k based on the difference in mean rewards, as given by Equation (17):

Original: [Equation]
Revised: [Equation]

Original: where k = a · max( µ − (cid:98) µ, 0) | µ | + (cid:15) .
Revised: where k = a · max( µ − (cid:98) µ, 0) | µ | + ϵ .

Original: where µ = 1 N 0 (cid:80) N 0 i =1 µ ( s i , a i ) , (cid:98) µ = 1 N 1 (cid:80) N 1 i =1 (cid:98) µ ( s i , a i ) are the mean reward of labeled and (predicted) unlabeled data, respectively.
Revised: where µ = 1 N 0 (cid:80) N 0 i =1 µ ( s i , a i ) , (cid:98) µ = 1 N 1 (cid:80) N 1 i =1 (cid:98) µ ( s i , a i ) are the mean reward of labeled and (predicted) unlabeled data, respectively.

Original: We use a = 25 and L = 10 in all experiments.
Revised: We use a = 25 and L = 10 in all experiments.

Original: Then we can plug in any model-free ofﬂine algorithms.
Revised: Then we can plug in any model-free offline algorithms.

Original: Here we use IQL (Kostrikov et al., 2021) as the backbone ofﬂine algorithm, but we emphasize that it can be easily integrated with other modelfree algorithms.
Revised: Here we use IQL (Kostrikov et al., 2021) as the backbone offline algorithm, but we emphasize that it can be easily integrated with other modelfree algorithms.

Original: The details of our algorithm is summarized in Algorithm 3 in Appendix A.2.[Equation]
Revised: The details of our algorithm is summarized in Algorithm 3 in Appendix A.2.

Original:   XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section [Table] we aim to evaluate the effectiveness of pessimistic reward estimation and answer the following questions: (1) How does PDS perform compared to the naive reward prediction and unlabeled data sharing (UDS) methods in single locomotion and manipulation tasks?
Revised: In this section, we aim to evaluate the effectiveness of pessimistic reward estimation and answer the following questions: (1) How does PDS perform compared to the naive reward prediction and unlabeled data sharing (UDS) methods in single locomotion and manipulation tasks?

Original: (2) How does
Revised: (2) How does

Original: PDS behave in multi-task ofﬂine RL settings compared to baselines?
Revised: PDS behave in multi-task offline RL settings compared to baselines?

Original: (3) What makes PDS effective?
Revised: (3) What makes PDS effective?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Single-task domains and datasets.
Revised: Single-task domains and datasets.

Original: To address Question (1), we empirically evaluate the PDS algorithm on the hopper, walker2d, and kitchen tasks from the D4RL benchmark suite (Fu et al., 2020).
Revised: To address Question (1), we empirically evaluate the PDS algorithm on the hopper, walker2d, and kitchen tasks from the D4RL benchmark suite (Fu et al., 2020).

Original: We use 50 labeled trajectories with varying amounts of unlabeled data of different sizes and qualities.
Revised: We use 50 labeled trajectories with varying amounts of unlabeled data of different sizes and qualities.

Original: This experimental setup is motivated by real-world problems where labeled data is often scarce [Table] and additional unlabeled data may be readily available.
Revised: This experimental setup is motivated by real-world problems where labeled data is often scarce, and additional unlabeled data may be readily available.

Original: [Table]
Revised: [Table]

Original: Multi-task domains and datasets.
Revised: Multi-task domains and datasets.

Original: We investigate Question (2) by evaluating PDS on several multi-task domains.
Revised: We investigate Question (2) by evaluating PDS on several multi-task domains.

Original: The ﬁrst set of domains we consider is Meta-World (Yu et al., 2020a), where we adopt the same setup as in CDS (Yu et al., 2021a) and evaluate PDS on four tasks: door open , door close , drawer open , and drawer close .
Revised: The first set of domains we consider is Meta-World (Yu et al., 2020a), where we adopt the same setup as in CDS (Yu et al., 2021a) and evaluate PDS on four tasks: door open , door close , drawer open , and drawer close .

Original: The second domain is the AntMaze task in D4RL, which consists of mazes of two sizes (medium and large) and includes 3 and 7 tasks, respectively, corresponding to different goal positions.
Revised: The second domain is the AntMaze task in D4RL, which consists of mazes of two sizes (medium and large) and includes 3 and 7 tasks, respectively, corresponding to different goal positions.

Original: For a detailed description of the experimental setting, please refer to Appendix D.
Revised: For a detailed description of the experimental setting, please refer to Appendix D.

Original: Comparisons.
Revised: Comparisons.

Original: To ensure a fair comparison, we combine UDS with IQL (Kostrikov et al., 2021), the same underlying ofﬂine RL method as PDS.
Revised: To ensure a fair comparison, we combine UDS with IQL (Kostrikov et al., 2021), the same underlying offline RL method as PDS.

Original: In addition to UDS, we train a naive reward prediction method and the sharing-all-true-rewards method (Oracle), and adapt them with IQL.
Revised: In addition to UDS, we train a naive reward prediction method and the sharing-all-true-rewards method (Oracle), and adapt them with IQL.

Original: In all experiments, we set the hyperparameters a = 25 and L = 10 for our method.
Revised: In all experiments, we set the hyperparameters a = 25 and L = 10 for our method.

Original: 5. E XPERIMENTAL R ESULTS
Revised: 5. E XPERIMENTAL R ESULTS

Original: Results of Question (1).
Revised: Results of Question (1).

Original: We evaluated each method on the hopper, walker2d, and kitchen domains and found that PDS outperformed the other methods on most tasks and achieved competitive or better performance than the oracle method (Table 3).
Revised: We evaluated each method on the hopper, walker2d, and kitchen domains and found that PDS outperformed the other methods on most tasks and achieved competitive or better performance than the oracle method (Table 3).

Original: Notably, PDS performed well when the labeled and unlabeled datasets had different data qualities, which we attribute to its ability to capture the uncertainties induced by this distribution shift and maintain a pessimistic algorithm.
Revised: Notably, PDS performed well when the labeled and unlabeled datasets had different data qualities, which we attribute to its ability to capture the uncertainties induced by this distribution shift and maintain a pessimistic algorithm.

Original: The prediction method performed well when the unlabeled dataset had high quality, and UDS performed well when the unlabeled data had low quality.
Revised: The prediction method performed well when the unlabeled dataset had high quality, and UDS performed well when the unlabeled data had low quality.

Original: PDS combined the strengths of both methods and achieved superior performance.
Revised: PDS combined the strengths of both methods and achieved superior performance.

Original: Results of Question (2).
Revised: Results of Question (2).

Original: Multi-task settings exhibit greater distributional shifts between labeled and unlabeled data due to the differing task goals.
Revised: Multi-task settings exhibit greater distributional shifts between labeled and unlabeled data due to the differing task goals.

Original: We evaluated PDS and the other methods on the AntMaze and Meta-World domains (Tables 2 and ?? ) and found that PDS’s performance was comparable to the oracle method and outperformed the other methods.
Revised: We evaluated PDS and the other methods on the AntMaze and Meta-World domains (Tables 2 and 1) and found that PDS’s performance was comparable to the oracle method and outperformed the other methods.

Original: UDS performed relatively poorly on the Meta-World dataset, possibly due to the high dataset quality, which made labeling with zeros induce a large reward bias.
Revised: UDS performed relatively poorly on the Meta-World dataset, possibly due to the high dataset quality, which made labeling with zeros induce a large reward bias.

Original: On the multi-task AntMaze domain, PDS outperformed both
Revised: On the multi-task AntMaze domain, PDS outperformed both

Original: UDS and the naive prediction method, especially on the diverse dataset.
Revised: UDS and the naive prediction method, especially on the diverse dataset.

Original: These results aligned with our observation on single-task domains that PDS performs better when the distribution shift between datasets is larger.
Revised: These results aligned with our observation on single-task domains that PDS performs better when the distribution shift between datasets is larger.

Original: Results of Question (3).
Revised: Results of Question (3).

Original: We conducted experiments on the hopper and walker2d tasks with various penalty weights k to investigate the effect of uncertainty weights in PDS (Figure 1).
Revised: We conducted experiments on the hopper and walker2d tasks with various penalty weights k to investigate the effect of uncertainty weights in PDS (Figure 1).

Original: The results shows that PDS can interpolate between UDS and the reward prediction method and offered a better trade-off to balance the conservation and generalization of reward estimators, resulting in better performance.
Revised: The results shows that PDS can interpolate between UDS and the reward prediction method and offered a better trade-off to balance the conservation and generalization of reward estimators, resulting in better performance.

Original: Also, PDS reduces the variance from reward prediction and is close to the oracle method, indicating its ability to reduce the uncertainties from the variance of reward predictors while keeping the reward bias small, as shown in Figure 2.
Revised: Also, PDS reduces the variance from reward prediction and is close to the oracle method, indicating its ability to reduce the uncertainties from the variance of reward predictors while keeping the reward bias small, as shown in Figure 2.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Discussion of PDS, UDS, and Reward Prediction: PDS can be seen as a generalization of both
Revised: Discussion of PDS, UDS, and Reward Prediction: PDS can be seen as a generalization of both

Original: UDS and the reward prediction method.
Revised: UDS and the reward prediction method.

Original: UDS sets the penalty weight k to inﬁnity, while the reward prediction method sets it to zero.
Revised: UDS sets the penalty weight k to infinity, while the reward prediction method sets it to zero.

Original: However, UDS introduces a high reward bias [Figure] and the reward prediction method ruins the pessimism of ofﬂine algorithms.
Revised: However, UDS introduces a high reward bias, and the reward prediction method ruins the pessimism of offline algorithms.

Original: In contrast, PDS offers a trade-off between bias and pessimism by adaptively adjusting k .
Revised: In contrast, PDS offers a trade-off between bias and pessimism by adaptively adjusting k .

Original: To verify that overestimation is the main factor for the suboptimal performance of the reward prediction method, we conduct additional ablation studies, as shown in Appendix G.
Revised: To verify that overestimation is the main factor for the suboptimal performance of the reward prediction method, we conduct additional ablation studies, as shown in Appendix G.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we show that incorporating reward-free data into ofﬂine reinforcement learning can yield signiﬁcant performance improvements.
Revised: In this paper, we show that incorporating reward-free data into offline reinforcement learning can yield significant performance improvements.

Original: Our theoretical analysis reveals that unlabeled data provides additional information about the MDP’s dynamics, reducing the problem to linear bandits in the limit and improving performance bounds therefore.
Revised: Our theoretical analysis reveals that unlabeled data provides additional information about the MDP’s dynamics, reducing the problem to linear bandits in the limit and improving performance bounds therefore.

Original: Building upon these insights, we propose a new algorithm, PDS, that leverages this information by incorporating uncertainty penalties on learned rewards to ensure a conservative approach.
Revised: Building upon these insights, we propose a new algorithm, PDS, that leverages this information by incorporating uncertainty penalties on learned rewards to ensure a conservative approach.

Original: Our method has provable guarantees in theory and achieves superior performance in practice.
Revised: Our method has provable guarantees in theory and achieves superior performance in practice.

Original: In future work, it may be interesting to explore how PDS can be further improved with representation learning methods, and to extend our analysis to more general settings, such as generalized linear MDPs (Wang et al., 2019) and low-rank
Revised: In future work, it may be interesting to explore how PDS can be further improved with representation learning methods, and to extend our analysis to more general settings, such as generalized linear MDPs (Wang et al., 2019) and low-rank

Original: MDPs (Ayoub et al., 2020; Jiang et al., 2017).
Revised: MDPs (Ayoub et al., 2020; Jiang et al., 2017).

Original:  A CKNOWLEDGEMENTS
Revised:  A CKNOWLEDGEMENTS

Original: This work is supported in part by Science and Technology Innovation 2030 - “New Generation
Revised: This work is supported in part by Science and Technology Innovation 2030 - “New Generation

Original: Artiﬁcial Intelligence” Major Project (No. 2018AAA0100904) and the National Natural Science
Revised: Artificial Intelligence” Major Project (No. 2018AAA0100904) and the National Natural Science

Original: Foundation of China (62176135).
Revised: Foundation of China (62176135).

Original: A A LGORITHM D ETAILS A. P ESSIMISTIC V ALUE I TERATION (PEVI,(J IN ET AL ., 2021))
Revised: A A LGORITHM D ETAILS A. P ESSIMISTIC V ALUE I TERATION (PEVI,(J IN ET AL ., 2021))

Original: In this section, we describe the details of PEVI algorithm.
Revised: In this section, we describe the details of PEVI algorithm.

Original: In linear MDPs, we can construct (cid:98) B (cid:98) V and Γ based on D as follows, where (cid:98) B (cid:98) V is the empirical estimation for B (cid:98) V .
Revised: In linear MDPs, we can construct (cid:98) B (cid:98) V and Γ based on D as follows, where (cid:98) B (cid:98) V is the empirical estimation for B (cid:98) V .

Original: For a given dataset D = { ( s τ , a τ , r τ ) } Nτ =1 , we deﬁne the empirical mean squared Bellman error (MSBE) as
Revised: For a given dataset D = { ( s τ , a τ , r τ ) } Nτ =1 , we define the empirical mean squared Bellman error (MSBE) as

Original: [Equation]
Revised: [Equation]

Original: Here λ > 0 is the regularization parameter.
Revised: Here λ > 0 is the regularization parameter.

Original: Note that (cid:98) w has the closed form
Revised: Note that (cid:98) w has the closed form

Original: [Equation]
Revised: [Equation]

Original:  where Λ = λI +
Revised: , where Λ = λI +

Original: N (cid:88) 
Revised: N (cid:88) τ

Original: [Equation] = 
Revised: = ϕ ( s τ , a τ ) ϕ ( s τ , a

Original: [Equation]
Revised: [Equation]

Original: Then we simply let
Revised: Then we simply let

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, we construct Γ based on D as
Revised: Meanwhile, we construct Γ based on D as

Original: [Equation]
Revised: [Equation]

Original: Here β > 0 is the scaling parameter.
Revised: Here β > 0 is the scaling parameter.

Original: The overall PEVI algorithm is summarized in Algorithm 2.
Revised: The overall PEVI algorithm is summarized in Algorithm 2.

Original: Algorithm 2 Pessimistic Value Iteration

Original: 1: Require : Dataset D = { ( s τ , a τ , r τ , s τ +1 ) } Tτ =1 .
Revised: 1: Require : Dataset D = { ( s τ , a τ , r τ , s τ +1 ) } Tτ =1 .

Original: 2: Initialization: Set (cid:98) V ( · ) ← 0 and construct Γ( · , · ) .
Revised: 2: Initialization: Set (cid:98) V ( · ) ← 0 and construct Γ( · , · ) .

Original: 3: while not converged do 4:
Revised: 3: while not converged do 4:

Original: Construct ( (cid:98) B (cid:98) V )( · , · ) 5:
Revised: Construct ( (cid:98) B (cid:98) V )( · , · ) 5:

Original: Set (cid:98) Q ( · , · ) ← ( (cid:98) B (cid:98) V )( · , · ) − Γ( · , · ) .
Revised: Set (cid:98) Q ( · , · ) ← ( (cid:98) B (cid:98) V )( · , · ) − Γ( · , · ) .

Original: Set (cid:98) π ( · | · ) ← argmax π E π (cid:104) (cid:98) Q ( · , · ) (cid:105) .
Revised: Set (cid:98) π ( · | · ) ← argmax π E π (cid:104) (cid:98) Q ( · , · ) (cid:105) .

Original: Set (cid:98) V ( · ) ← E (cid:98) π (cid:104) (cid:98) Q ( · , · ) (cid:105) .
Revised: Set (cid:98) V ( · ) ← E (cid:98) π (cid:104) (cid:98) Q ( · , · ) (cid:105) .

Original: 8: end while 9: Return (cid:98) V , (cid:98) π
Revised: 8: end while 9: Return (cid:98) V , (cid:98) π

Original: A. IQL WITH PROVABLE DATA SHARING
Revised: A. IQL WITH PROVABLE DATA SHARING

Original: In this section, we give a detailed description of our IQL+PDS algorithm.
Revised: In this section, we give a detailed description of our IQL+PDS algorithm.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: B P ROOF OF T HEOREM 4. 
Revised: B P ROOF OF T HEOREM 4.

Original: Proof.
Revised: Proof.

Original: From Equation (11) in Algorithm 1, we have
Revised: From Equation (11) in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) θ is the pessimistic estimation of θ .
Revised: where (cid:101) θ is the pessimistic estimation of θ .

Original: Let E 1 be the event θ (cid:63) ∈ C ( δ ) , then we have P ( E 1 ) ≥ 

Revised: Let E 1 be the event θ ⋆ ∈ C ( δ ) , then we have P ( E 1 ) ≥

Original: −
Revised: −

Original: δ from Lemma 4.1.
Revised: δ from Lemma 4.1.

Original: Let E 2 be the event where the following inequality holds,
Revised: Let E 2 be the event where the following inequality holds,

Original: [Equation]
Revised: [Equation]

Original: then we have P ( E 2 ) ≥ 1 − δ from Lemma C.3.
Revised: then we have P ( E 2 ) ≥ 1 − δ from Lemma C.3.

Original: Condition on E 1 ∩ E 2 , we have.
Revised: Condition on E 1 ∩ E 2 , we have.

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from Lemma C.2.
Revised: where the first inequality follows from Lemma C.2.

Original: The second inequality follows from Equation (21), and the last inequality follows from Lemma C.
Revised: The second inequality follows from Equation (21), and the last inequality follows from Lemma C.5 and C.1.

Original: From the union bound, we have that the above inequality holds with a probability of 1 − 2 δ .
Revised: From the union bound, we have that the above inequality holds with a probability of 1 − 2 δ .

Original: C A DDTIONAL L EMMAS AND M ISSING P ROOFS
Revised: C A DDTIONAL L EMMAS AND M ISSING P ROOFS

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Under the event in Lemma C.3, we have
Revised: Under the event in Lemma C.3, we have

Original: [Equation]
Revised: [Equation]

Original: with probability 1 − δ , for all (cid:107) θ (cid:107) 22 ≤ d .
Revised: with probability 1 − δ , for all ∥ θ ∥ 22 ≤ d .

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From the deﬁnition of (cid:98) Q ( s, a ) and (cid:98) V ( s )
Revised: From the definition of (cid:98) Q ( s [Equation] a ) and (cid:98) V ( s )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Under the condition of Lemma C
Revised: Under the condition of Lemma C.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Revised: Here the first inequality follows from the fact that (cid:98) π ( ·|

Revised: [Equation]

Original: Here the ﬁrst inequality follows from the fact that (cid:98) π ( ·| s ) = argmax π (cid:68) (cid:98) Q ( · , · ) , π ( ·| s ) (cid:69) and the second inequality follows from Equation (25).
Revised: = [Equation] ( ·| s ) (cid:69) and the second inequality follows from Equation (25) [Equation]

Original: By the Cauchy-Schwarz inequality, we have
Revised: By the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: [Equation]

Original: for all s ∈ S .
Revised: for all s ∈ S .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Here { λ j ( s ) } dj =1 are the eigenvalues of Σ π ∗ ,s for all s ∈ S , the second inequality follows from
Revised: Here { λ j ( s ) } dj =1 are the eigenvalues of Σ π ∗ ,s for all s ∈ S , the second inequality follows from

Original: Equation (26).
Revised: Equation (26).

Original: Meanwhile, by Deﬁnition 3.1, we have (cid:107) φ ( s, a ) (cid:107) ≤ 1 for all ( s, a ) ∈ S × A .
Revised: Meanwhile, by Definition 3.1, we have ∥ ϕ ( s, a ) ∥ ≤ 1 for all ( s, a ) ∈ S × A .

Original: By Jensen’s inequality, we have
Revised: By Jensen’s inequality, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all s ∈ S .
Revised: for all s ∈ S .

Original: As Σ π ∗ ,s is positive semideﬁnite, we have λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] .
Revised: As Σ π ∗ [Equation] s is positive semidefinite, we have λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] .

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ S , where the second inequality follows from the fact that λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] , while the third inequality follows from the choice of the scaling parameter β > 0 .
Revised: for all x ∈ S , where the second inequality follows from the fact that λ j ( s ) ∈ [0 , 1] for all s ∈ S and all j ∈ [ d ] , while the third inequality follows from the choice of the scaling parameter β > 0 .

Original: Then we have the conclusion in Lemma C.1.
Revised: Then we have the conclusion in Lemma C.

Revised: [Equation]

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Under the event in Lemma C.3, we have
Revised: Under the event in Lemma C.3, we have

Original: [Equation]
Revised: [Equation]

Original:  with probability 1 − δ , for all (cid:107) θ (cid:107) 22 ≤ d .
Revised: 30) with probability 1 − δ , for all ∥ θ ∥ 22 ≤ d .

Original: Proof.
Revised: Proof.

Original: Similar to the proof of Lemma C.1, let
Revised: Similar to the proof of Lemma C.1, let

Original: [Equation]
Revised: [Equation]

Original: (31) we have
Revised: 31) we have

Original: [Equation]
Revised: [Equation]

Original: Then under the condition of Lemma C.3, it holds that
Revised: Then under the condition of Lemma C.3, it holds that

Original: [Equation]
Revised: [Equation]

Original: Then we have the result immediately.
Revised: Then we have the result immediately.

Original: Lemma C.3 ( ξ -Quantiﬁers) .
Revised: Lemma C.3 ( ξ -Quantifiers) .

Original: Let λ = 1 ,
Revised: Let λ = 1 ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then Γ = β · (cid:0) φ ( s, a ) (cid:62) Λ − 1 φ ( s, a ) (cid:1) 1 / 2 are ξ -quantiﬁers with probability at least 1 − ξ .
Revised: Then Γ = β · (cid:0) ϕ ( s, a ) ⊤ Λ − 1 ϕ ( s, a ) (cid:1) 1 / 2 are ξ -quantifiers with probability at least 1 − ξ .

Original: That is, let
Revised: That is, let

Original: E 2 be the event that the following inequality holds,
Revised: E 2 be the event that the following inequality holds,

Original: [Equation]
Revised: [Equation]

Original: Then we have P ( E 2 ) ≥ 1 − ε .
Revised: Then we have P ( E 2 ) ≥ 1 − ε .

Original: Proof.
Revised: Proof.

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Then we bound (i) and (ii), respectively.
Revised: Then we bound (i) and (ii), respectively.

Original: For (i) [Equation] we have
Revised: For (i), we have

Original: [Equation]
Revised: [Equation]

Original: ) where the ﬁrst inequality follows from Cauchy-Schwartz inequality.
Revised: 36) where the first inequality follows from Cauchy-Schwartz inequality.

Original: The second inequality follows from the fact that || Λ − 1 || op ≤ λ − 1 and Lemma C.4.
Revised: The second inequality follows from the fact that || Λ − 1 || op ≤ λ − 1 and Lemma C.4.

Original: For notation simplicity, let (cid:15) τ = r τ + γ (cid:98) V ( s τ +1 ) − φ (cid:62) τ w , then we have
Revised: For notation simplicity, let ϵ τ = r τ + γ (cid:98) V ( s τ +1 ) − ϕ ⊤ τ w , then we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The term (iii) is depend on the randomness of the data collection process of D .
Revised: The term (iii) is depend on the randomness of the data collection process of D .

Original: To bound this term, we resort to uniform concentration inequalities to upper bound
Revised: To bound this term, we resort to uniform concentration inequalities to upper bound

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where V ( s ; w, β, Σ) = max a { φ ( s, a ) (cid:62) w − β · (cid:112) φ ( s, a ) (cid:62) Σ − 1 φ ( s, a ) } .
Revised: where V ( s ; w, β, Σ) = max a { ϕ ( s, a ) ⊤ w − β · (cid:112) ϕ ( s, a ) ⊤ Σ − 1 ϕ ( s, a ) } .

Original: For all (cid:15) > 0 , let
Revised: For all ϵ > 0 , let

Original: N ( (cid:15) ; R, B, λ ) be the minimal cover if V ( R, B, λ ) .
Revised: N ( ϵ ; R, B, λ ) be the minimal cover if V ( R, B, λ ) .

Original: That is, for any function V ∈ V ( R, B, λ ) , there exists a function V † ∈ N ( (cid:15) ; R, B, λ ) , such that
Revised: That is, for any function V ∈ V ( R, B, λ ) , there exists a function V † ∈ N ( ϵ ; R, B, λ ) , such that

Original: [Equation]
Revised: [Equation]

Original: Let R 

Revised: Let R

Original: =
Revised: =

Original: V max (cid:112) Nd/λ, B 

Revised: V max (cid:112) Nd/λ, B

Original: = 2 β , it is easy to show that at each iteration, (cid:98) V u ∈ V ( R 0 , B 0 , λ ) .
Revised: = 2 β , it is easy to show that at each iteration, (cid:98) V u ∈ V ( R 0 , B 0 , λ ) .

Original: From the deﬁnition of B , we have
Revised: From the definition of B , we have

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Let (cid:15) † τ = r
Revised: Let ϵ † τ = r

Original: ( s τ , a τ ) + γV † ( s τ +1 ) − B V † ( s, a ) , we have(iii [Equation] |
Revised: ( s τ , a τ ) + γV † ( s τ +1 ) − B V † ( s, a ) , we have

Original: [Equation]
Revised: [Equation]

Original: It remains to bound || (cid:80)
Revised: It remains to bound || (cid:80)

Original: Nτ =1 φ τ (cid:15) † τ || 2Λ − 1 .
Revised: Nτ =1 ϕ τ ϵ † τ || 2Λ − 1 .

Original: From the assumption for data collection process, it is easy to show that E
Revised: From the assumption for data collection process, it is easy to show that E

Original: D [ (cid:15) τ | F τ − 1 ] = 0 , where F τ − 1 = σ ( { ( s i , a i ) τi =1 ∪ ( r i , s i +1 )
Revised: D [ ϵ τ | F τ − 1 ] = 0 , where F τ − 1 = σ ( { ( s i , a i ) τi =1 ∪ ( r i , s i +1 )

Original: τi =1 } ) is the σ -algebra generated by the variables from the ﬁrst τ step.
Revised: τi =1 } ) is the σ -algebra generated by the variables from the first τ step.

Original: Moreover, since (cid:15) τ ≤ 2 V max , we have (cid:15) τ are 2 V max -sub-Gaussian conditioning on F τ − 1 .
Revised: Moreover, since ϵ τ ≤ 2 V max , we have ϵ τ are 2 V max -sub-Gaussian conditioning on F τ − 1 .

Original: Then we invoke Lemma C.7 with M 

Revised: Then we invoke Lemma C.7 with M

Original: = λ · I and
Revised: = λ · I and

Original: M k = λ · I + (cid:80) kτ =1 φ ( s τ , a τ ) φ ( s τ , a τ ) (cid:62) .
Revised: M k = λ · I + (cid:80) kτ =1 ϕ ( s τ , a τ ) ϕ ( s τ , a τ ) ⊤ .

Original: For the ﬁxed function V : S → [0 , V max ] , we have
Revised: For the fixed function V : S → [0 , V max ] , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for all δ ∈ (0 , 1) .
Revised: for all δ ∈ (0 , 1) .

Original: Note that (cid:107) φ ( s, a ) (cid:107) ≤ 1 for all ( s, a ) ∈ S
Revised: Note that ∥ ϕ ( s, a ) ∥ ≤ 1 for all ( s, a ) ∈ S

Original: × A by Deﬁnition 3.1.
Revised: × A by Definition 3.1.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) op denotes the matrix operator norm.
Revised: where ∥ · ∥ op denotes the matrix operator norm

Original: Hence, it holds that det(Λ) ≤ ( λ + N ) d and det( λ · I ) =
Revised: Hence, it holds that det(Λ) ≤ ( λ + N ) d and det( λ · I ) =

Original: λ d , which implies
Revised: λ d , which implies

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we conclude the proof of Lemma C
Revised: we conclude the proof of Lemma C

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma C.3 and the union bound
Revised: Applying Lemma C.3 and the union bound, we have

Original: [Equation]
Revised: [Equation]

Original: we have P D
Revised: |

Original: [Equation]
Revised: [Equation]

Original: Recall that(cid:98) V ∈ V
Revised: Recall that

Original: ( R 0 , B 0 , λ ) , where R 

Revised: [Equation] where R

Original: = V max (cid:112) Nd/λ, B {+
+}
Revised: = V max (cid:112) Nd/λ, B {+
+}

Revised: = 2 β, λ = 1 , β = c · dV max (cid:112)

Original: [Equation]
Revised: [Equation]

Original: Here c > 0 is an absolute constant, ξ ∈ (0 , 1) is the conﬁdence parameter, and ζ = log(2 dV max /ξ ) is speciﬁed in Algorithm 2.
Revised: Here c > 0 is an absolute constant, ξ ∈ (0 , 1) is the confidence parameter [Equation] and ζ = log(2 dV max /ξ ) is specified in Algorithm 2.

Original: Applying Lemma C.6 with ε = dV max /N , we have
Revised: Applying Lemma C [Equation] 6 with ε = dV max /N , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By setting δ = ξ/ |N ( ε ) | , we have that with probability at least 1 − ξ ,
Revised: By setting δ = ξ/ |N ( ε ) | , we have that with probability at least 1 − ξ ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here the last inequality follows from simple algebraic inequalities.
Revised: Here the last inequality follows from simple algebraic inequalities.

Original: We set c ≥ 1 to be sufﬁciently large [Equation] which ensures that 36 + 8 · log(64 c 2 ) ≤ c 2 / 4 on the right-hand side of Equation (46).
Revised: We set c ≥ 1 to be sufficiently large, which ensures that 36 + 8 · log(64 c 2 ) ≤ c 2 / 4 on the right-hand side of Equation (46).

Original: By
Revised: By

Original: Equations (37) and (46) [Equation] it holds that
Revised: Equations (37) and (46), it holds that

Original: [Equation]
Revised: [Equation]

Original: By Equations (20), (35) [Equation]
Revised: By Equations (20), (35), (36)

Original: [Equation]
Revised: [Equation]

Original: and (47) [Equation] for all ( s, a ) ∈ S × A , it holds that
Revised: and (47), for all ( s, a ) ∈ S × A , it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − ξ .
Revised: with probability at least 1 − ξ .

Original: Therefore, we conclude the proof of Lemma C.3.
Revised: Therefore, we conclude the proof of Lemma C.3.

Original: Lemma C.4 (Bounded weight of value function) .
Revised: Lemma C.4 (Bounded weight of value function) .

Original: Let V max =
Revised: Let V max =

Original: r max / (1 − γ ) .
Revised: r max / (1 − γ ) .

Original: For any function
Revised: For any function

Original: V : S → [0 , V max ] , we have
Revised: V : S → [0 , V max ] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since 
Revised: Since w ⊤

Original: [Equation]
Revised: [Equation]

Revised: s, a

Revised: [Equation]

Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For (cid:98) w , we have
Revised: For (cid:98) w , we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: V

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: =1 || Λ − 1 ϕ

Original: [Equation]
Revised: [Equation]

Original: ≤ V
Revised: ( r

Original: [Equation]
Revised: [Equation]

Original: (cid:114) N λ
Revised: s

Original: [Equation]
Revised: [Equation]

Original:  Λ − 
Revised: +1 )) || ≤ V max N (cid:88) τ =1 || Λ − 1 ϕ τ || ≤ V max

Original: [Equation]
Revised: N (cid:88) τ = (cid:113) ϕ ⊤ τ Λ − 1 / 2 Λ − 1 Λ − 1 / 2 ϕ τ ≤

Original: = φ
Revised: V max √ λ

Original: [Equation]
Revised: N (cid:88) τ = (cid:113) ϕ ⊤ τ Λ − 1 ϕ τ ≤ V max (cid:114) N λ (cid:118) (cid:117) (cid:117) (cid:116) Tr(Λ −

Original: (cid:62) τ
Revised: T (cid:88) τ

Original: [Equation] ≤ V max (cid:114) Nd λ .
Revised: = ϕ τ ϕ ⊤ τ ) ≤ V max (cid:114) Nd λ .

Original: Lemma C.5.
Revised: Lemma C.5.

Original: For policy π ∗ and any reward function parameter θ ∈ C ( δ ) , we have
Revised: For policy π ∗ and any reward function parameter θ ∈ C ( δ ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the deﬁnition
Revised: From the definition

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Where the ﬁrst inequality follows from the Cauchy-Schwartz inequality and the second inequality uses the fact that θ ∈ C ( δ ) .
Revised: Where the first inequality follows from the Cauchy-Schwartz inequality and the second inequality uses the fact that θ ∈ C ( δ ) .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the second to last inequality follows similarly as Equation (26) (27) and the last inequality follows from the fact that α = 2 r max √ dζ 2 .
Revised: where the second to last inequality follows similarly as Equation (26) (27) and the last inequality follows from the fact that α = 2 r max √ dζ 2 .

Original: Note that such a choice of α is sufﬁcient for Lemma 4.1 to hold since
Revised: Note that such a choice of α is sufficient for Lemma 4.1 to hold since

Original: [Equation]
Revised: [Equation]

Original: where the inequalities holds for sufﬁciently small δ > 0 and d ≥ 2 .
Revised: where the inequalities holds for sufficiently small δ > 0 and d ≥ 2 .

Original: C. T ECHNICAL L EMMAS
Revised: C. T ECHNICAL L EMMAS

Original: Lemma C.6 ( ε -Covering Number (Jin et al., 2020)) .
Revised: Lemma C.6 ( ε -Covering Number (Jin et al., 2020)) .

Original: For all h ∈ [ H ] and all ε > 0 , we have
Revised: For all h ∈ [ H ] and all ε > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma C.6.
Revised: Proof of Lemma C.6.

Original: See Lemma D.6 in Jin et al.
Revised: See Lemma D.6 in Jin et al.

Original: (2020) for detailed proof.
Revised: (2020) for detailed proof.

Original: Lemma C.7 (Concentration of Self-Normalized Processes (Abbasi-Yadkori et al., 2011)) .
Revised: Lemma C.7 (Concentration of Self-Normalized Processes (Abbasi-Yadkori et al., 2011)) .

Original: Let {F t } ∞ t
Revised: Let {F t } ∞ t

Original: =
Revised: =

Original: be a ﬁltration and { (cid:15) t } ∞ t =1 be an R -valued stochastic process such that (cid:15) t is F t -measurable for all t ≥ 1 .
Revised: be a filtration and { ϵ t } ∞ t =1 be an R -valued stochastic process such that ϵ t is F t -measurable for all t ≥ 1 .

Original: Moreover, suppose that conditioning on F t − 1 , (cid:15) t is a zero-mean and σ -sub-Gaussian random variable for all t ≥ 1 , that is,
Revised: Moreover, suppose that conditioning on F t − 1 , ϵ t is a zero-mean and σ -sub-Gaussian random variable for all t ≥ 1 , that is,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, let { φ t } ∞ t =1 be an R d -valued stochastic process such that φ t is F t − 1 -measurable for all t ≥ 1 .
Revised: Meanwhile, let { ϕ t } ∞ t =1 be an R d -valued stochastic process such that ϕ t is F t − 1 -measurable for all t ≥ 1 .

Original: Also, let
Revised: Also, let

Original: M 0 ∈ R
Revised: M 0 ∈ R

Original: d × d be a deterministic positive-deﬁnite matrix and
Revised: d × d be a deterministic positive-definite matrix and

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 1 .
Revised: for all t ≥ 1 .

Original: For all δ > 0 , it holds that
Revised: For all δ > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: for all t ≥ 1 with probability at least 1 − δ .
Revised: for all t ≥ 1 with probability at least 1 − δ .

Original: Proof.
Revised: Proof.

Original: See Theorem 1 of Abbasi-Yadkori et al.
Revised: See Theorem 1 of Abbasi-Yadkori et al.

Original: (2011) for a detailed proof.
Revised: (2011) for a detailed proof.

Original: D E XPERIMENTAL S ETTINGS D. M ULTI - TASK A NTMAZE
Revised: D E XPERIMENTAL S ETTINGS D. M ULTI - TASK A NTMAZE

Original: We ﬁrst divide the source dataset (e.g., antmaze-medium-diverse-v2) into the directed dataset or the undirected dataset.
Revised: We first divide the source dataset (e.g., antmaze-medium-diverse-v2) into the directed dataset or the undirected dataset.

Original: The trajectories in the undirected dataset are randomly and uniformly divided into different tasks.
Revised: The trajectories in the undirected dataset are randomly and uniformly divided into different tasks.

Original: In contrast, the directed dataset is associated with the goal closest to the ﬁnal state of the trajectory.
Revised: In contrast, the directed dataset is associated with the goal closest to the final state of the trajectory.

Original: Then, for each subtask (e.g., goal=(0.0, 16.0)), we relabel the corresponding undirected or undirected dataset according to the goal.
Revised: Then, for each subtask (e.g., goal=(0.0, 16.0)), we relabel the corresponding undirected or undirected dataset according to the goal.

Original: Finally, we keep the rewards in the target task dataset (labeled dataset) while setting the rewards in the other task dataset to 0 (unlabeled dataset).
Revised: Finally, we keep the rewards in the target task dataset (labeled dataset) while setting the rewards in the other task dataset to 0 (unlabeled dataset).

Original: We visualize the directed or undirected dataset in Figure 3.
Revised: We visualize the directed or undirected dataset in Figure 3.

Original: We can ﬁnd that the distribution shift issue in the directed dataset is more severe than the undirected dataset, which further challenges the unlabeled data sharing algorithms.
Revised: We can find that the distribution shift issue in the directed dataset is more severe than the undirected dataset, which further challenges the unlabeled data sharing algorithms.

Original: D. M ULTI - TASK M ETA -W ORLD
Revised: D. M ULTI - TASK M ETA -W ORLD

Original: We consider the same setup as in CDS (Yu et al., 2021a) and four tasks: door open , door close , drawer open and drawer close , which is shown in Figure 4.
Revised: We consider the same setup as in CDS (Yu et al., 2021a) and four tasks: door open , door close , drawer open and drawer close , which is shown in Figure 4.

Original: We use medium-replay datasets with 152K transitions for task door open and drawer close and use 10 expert trajectories for task door close and drawer open.
Revised: We use medium-replay datasets with 152K transitions for task door open and drawer close and use 10 expert trajectories for task door close and drawer open.

Original: [Figure]
Revised: [Figure]

Original: E M ORE DISCUSSION ON THE RELATIVE PERFORMANCE OF UDS
Revised: E M ORE DISCUSSION ON THE RELATIVE PERFORMANCE OF UDS

Original: Based on the discussion in the main text, we can summarize the factors that affect the relative performance of PDS algorithms as follows scenarios large number of unlabeled data high quality of unlabeled data long horizon large dimension ﬁnite-sample term
Revised: Based on the discussion in the main text, we can summarize the factors that affect the relative performance of PDS algorithms as follows. scenarios large number of unlabeled data high quality of unlabeled data long horizon large dimension finite-sample term

Original: [Table]
Revised: [Table]

Original: However, it is worth noting that the asymptotic term is dependent on the backbone ofﬂine algorithm we choose.
Revised: However, it is worth noting that the asymptotic term is dependent on the backbone offline algorithm we choose.

Original: For model-based algorithms like Uehara & Sun (2021), the performance bound scale as O ( d ) and thus the problem’s dimension does not affect the asymptotic term.
Revised: For model-based algorithms like Uehara & Sun (2021), the performance bound scale as O ( d ) and thus the problem’s dimension does not affect the asymptotic term.

Original: The dependence over the discount factor is also dependent on the choice of the backbone algorithm.
Revised: The dependence over the discount factor is also dependent on the choice of the backbone algorithm.

Original: However, it is known that the lower bound of ofﬂine RL algorithms in linear MDPs scales as O ((1 − γ ) 1 . 5 ) so this asymptotic term must scales at least as O ((1 − γ ) 0 . 5 ) and thus negatively depends on the discount factor.
Revised: However, it is known that the lower bound of offline RL algorithms in linear MDPs scales as O ((1 − γ ) 1 . 5 ) so this asymptotic term must scales at least as O ((1 − γ ) 0 . 5 ) and thus negatively depends on the discount factor.

Original: That is, the larger the discount factor, the better the relative performance of PDS.
Revised: That is, the larger the discount factor, the better the relative performance of PDS.

Original: F D ISCUSSION ON THE P ERFORMANCE B OUND F. C ONSTRUCTION OF A DVERSARIAL E XAMPLES
Revised: F D ISCUSSION ON THE P ERFORMANCE B OUND F. C ONSTRUCTION OF A DVERSARIAL E XAMPLES

Original: In this section, we show that there is an MDP and an “unfortunate” dataset such that the suboptimality of Algorithm 3 matches the performance bound in Theorem 4.3.
Revised: In this section, we show that there is an MDP and an “unfortunate” dataset such that the suboptimality of Algorithm 3 matches the performance bound in Theorem 4.3.

Original: We ﬁrst focus on the case without data sharing.
Revised: We first focus on the case without data sharing.

Original: The same techniques can be used to match the reward-learning bound.
Revised: The same techniques can be used to match the reward-learning bound.

Original: We only need to show that all inequalities in Theorem 4.3 can become equality.
Revised: We only need to show that all inequalities in Theorem 4.3 can become equality.

Original: Suppose we have an
Revised: Suppose we have an

Original: MDP with one state and N > d actions.
Revised: MDP with one state and N > d actions.

Original: d of them are optimal actions, with feature mapφ ( s, a i ) =
Revised: d of them are optimal actions, with feature map

Original: (cid:124) (cid:123)(cid:122) (cid:125)
Revised: [Equation]

Original: i − 1 zeros , √ d, 0 , . . . ) and let the optimal policy be uniform over d actions.
Revised: and let the optimal policy be uniform over d actions.

Original: Such construction makes Σ π ∗ ,s =
Revised: Such construction makes Σ π [Equation] s

Original:  I so that inequalities in Equation (25) ∼ (27) become equalities.
Revised: [Equation] I so that inequalities in Equation (25) ∼ (27) become equalities.

Original: Then we let samples from other actions match the conﬁdence upper bound while samples from the optimal action match the conﬁdence lower bound, and we also need all the samples to be symmetric over different dimensions (this is required by the Cauchy-Schwitz inequality used in the proof), such that the conﬁdence bound inequalities in
Revised: Then we let samples from other actions match the confidence upper bound while samples from the optimal action match the confidence lower bound, and we also need all the samples to be symmetric over different dimensions (this is required by the Cauchy-Schwitz inequality used in the proof), such that the confidence bound inequalities in

Original: Lemma C.3 also become equalities.
Revised: Lemma C.3 also become equalities.

Original: Then, in this case, the suboptimality bound is matched and the bound in Theorem 4.3 is tight.
Revised: Then, in this case, the suboptimality bound is matched and the bound in Theorem 4.3 is tight.

Original: F. R EWARD B IAS OF UDS
Revised: F. R EWARD B IAS OF UDS

Original: In this section, we show that UDS suffers from a constant reward bias.
Revised: In this section, we show that UDS suffers from a constant reward bias.

Original: Lemma F.1.
Revised: Lemma F.1.

Original: By labeling all rewards in the unlabeled dataset to zeros, we have
Revised: By labeling all rewards in the unlabeled dataset to zeros, we have

Original: [Equation]
Revised: [Equation]

Original: That is, the reward bias does not vanish as long as the ratio of labeled data size and unlabeled data size keeps constant.
Revised: That is, the reward bias does not vanish as long as the ratio of labeled data size and unlabeled data size keeps constant.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: G H OW BAD IS THE REWARD PREDICTION BASELINE ACTUALLY ?
Revised: G H OW BAD IS THE REWARD PREDICTION BASELINE ACTUALLY ?

Original: We conduct ablation studies for the reward prediction method from three aspects, including model size, ensemble number, and early stopping, which are shown in Table 5, Table 6 and Table 7.
Revised: We conduct ablation studies for the reward prediction method from three aspects, including model size, ensemble number, and early stopping, which are shown in Table 5, Table 6 and Table 7.

Original: For the model size, 256*3 denotes the 256 hidden neurons with three hidden layers.
Revised: For the model size, 256*3 denotes the 256 hidden neurons with three hidden layers.

Original: As for the early stopping, Epoch Number = 3 denotes traversing the entire training dataset 3 times.
Revised: As for the early stopping, Epoch Number = 3 denotes traversing the entire training dataset 3 times.

Original: (We ﬁnd the
Revised: (We find the

Original: Epoch Number=3 is enough to reduce the prediction error to a small range, and increasing epochs leads to overﬁtting.)
Revised: Epoch Number=3 is enough to reduce the prediction error to a small range, and increasing epochs leads to overfitting.)

Original: We conduct experiments in a setting where the quality of the reward labeled and reward-free data differs signiﬁcantly.
Revised: We conduct experiments in a setting where the quality of the reward labeled and reward-free data differs significantly.

Original: For example, walker2d-expert(50K)-random(0.1M) denotes 50K reward-labeled data from expert datasets and 0.1M reward-free data from random datasets.
Revised: For example, walker2d-expert(50K)-random(0.1M) denotes 50K reward-labeled data from expert datasets and 0.1M reward-free data from random datasets.

Original: We set the PDS as the default parameter in all comparisons, including the model size 256*2, the training epoch 3, and the ensemble number 10.
Revised: We set the PDS as the default parameter in all comparisons, including the model size 256*2, the training epoch 3, and the ensemble number 10.

Original: All experimental results adopt the normalized score metric averaged over ﬁve seeds with standard deviation.
Revised: All experimental results adopt the normalized score metric averaged over five seeds with standard deviation.

Original: The experimental results show that ﬁne-tuning reward prediction can improve its performance in some cases, Nevertheless, a well-tuned reward prediction method still performs poorly compared to PDS in general.
Revised: The experimental results show that fine-tuning reward prediction can improve its performance in some cases, Nevertheless, a well-tuned reward prediction method still performs poorly compared to PDS in general.

Original: We hypothesize that this is because the ”test” (reward-free) dataset may have a large distributional shift from the ”training” (reward-labeled) dataset, violating the i.i.d.
Revised: We hypothesize that this is because the ”test” (reward-free) dataset may have a large distributional shift from the ”training” (reward-labeled) dataset, violating the i.i.d.

Original: assumption in supervised learning.
Revised: assumption in supervised learning.

Original: Model Size (Reward Prediction)

Original: [Equation]
Revised: [Equation]

Original: PDS

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: parameter epoch number

Original: [Table]
Revised: [Table]