Original: O PTIMAL C ONSERVATIVE O FFLINE RL WITH G ENERAL F UNCTION A PPROXIMATION VIA A UGMENTED L AGRANGIAN
Revised: O PTIMAL C ONSERVATIVE O FFLINE RL WITH G ENERAL F UNCTION A PPROXIMATION VIA A UGMENTED L AGRANGIAN

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Offline reinforcement learning (RL), which refers to decision-making from a previously-collected dataset of interactions, has received significant attention over the past years.
Revised: Offline reinforcement learning (RL), which refers to decision-making from a previously-collected dataset of interactions, has received significant attention over the past years.

Original: Much effort has focused on improving offline RL practicality by addressing the prevalent issue of partial data coverage through various forms of conservative policy learning.
Revised: Much effort has focused on improving offline RL practicality by addressing the prevalent issue of partial data coverage through various forms of conservative policy learning.

Original: While the majority of algorithms do not have finitesample guarantees, several provable conservative offline RL algorithms are designed and analyzed within the single-policy concentrability framework that handles partial coverage.
Revised: While the majority of algorithms do not have finitesample guarantees, several provable conservative offline RL algorithms are designed and analyzed within the single-policy concentrability framework that handles partial coverage.

Original: Yet, in the nonlinear function approximation setting where confidence intervals are difficult to obtain, existing provable algorithms suffer from computational intractability, prohibitively strong assumptions, and suboptimal statistical rates.
Revised: Yet, in the nonlinear function approximation setting where confidence intervals are difficult to obtain, existing provable algorithms suffer from computational intractability, prohibitively strong assumptions, and suboptimal statistical rates.

Original: In this paper, we leverage the marginalized importance sampling (MIS) formulation of RL and present the first set of offline RL algorithms that are statistically optimal and practical under general function approximation and single-policy concentrability, bypassing the need for uncertainty quantification.
Revised: In this paper, we leverage the marginalized importance sampling (MIS) formulation of RL and present the first set of offline RL algorithms that are statistically optimal and practical under general function approximation and single-policy concentrability, bypassing the need for uncertainty quantification.

Original: We identify that the key to successfully solving the sample-based approximation of the MIS problem is ensuring that certain occupancy validity constraints are nearly satisfied.
Revised: We identify that the key to successfully solving the sample-based approximation of the MIS problem is ensuring that certain occupancy validity constraints are nearly satisfied.

Original: We enforce these constraints by a novel application of the augmented Lagrangian method and prove the following result: with the MIS formulation, augmented Lagrangian is enough for statistically optimal offline RL.
Revised: We enforce these constraints by a novel application of the augmented Lagrangian method and prove the following result: with the MIS formulation, augmented Lagrangian is enough for statistically optimal offline RL.

Original: In stark contrast to prior algorithms that induce additional conservatism through methods such as behavior regularization, our approach provably eliminates this need and reinterprets regularizers as “enforcers of occupancy validity” than “promoters of conservatism.”
Revised: In stark contrast to prior algorithms that induce additional conservatism through methods such as behavior regularization, our approach provably eliminates this need and reinterprets regularizers as “enforcers of occupancy validity” than “promoters of conservatism.”

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The goal of offline RL is to design agents that learn to achieve competence in a task using only a previously-collected dataset of interactions (Lange et al., 2012).
Revised: The goal of offline RL is to design agents that learn to achieve competence in a task using only a previously-collected dataset of interactions (Lange et al., 2012).

Original: Offline RL is a promising tool for many critical applications, from healthcare to autonomous driving to scientific discovery, where the online mode of learning by interacting with the environment is dangerous, impractical, costly, or even impossible (Levine et al., 2020).
Revised: Offline RL is a promising tool for many critical applications, from healthcare to autonomous driving to scientific discovery, where the online mode of learning by interacting with the environment is dangerous, impractical, costly, or even impossible (Levine et al., 2020).

Original: Despite this, offline RL has not yet been truly successful in practice (Fujimoto et al., 2019) and impressive RL performance has been limited to settings with known environments (Silver et al., 2017; Moravˇcík et al., 2017), access to accurate simulators (Mnih et al., 2015; Degrave et al., 2022; Fawzi et al., 2022), or expert demonstrations (Vinyals et al., 2017).
Revised: Despite this, offline RL has not yet been truly successful in practice (Fujimoto et al., 2019) and impressive RL performance has been limited to settings with known environments (Silver et al., 2017; Moravˇcík et al., 2017), access to accurate simulators (Mnih et al., 2015; Degrave et al., 2022; Fawzi et al., 2022), or expert demonstrations (Vinyals et al., 2017).

Original: One of the central challenges in offline RL is the lack of uniform coverage in real datasets and the distribution shift between the occupancy of candidate policies and offline data distribution, which pose difficulties in accurately evaluating the candidate policies.
Revised: One of the central challenges in offline RL is the lack of uniform coverage in real datasets and the distribution shift between the occupancy of candidate policies and offline data distribution, which pose difficulties in accurately evaluating the candidate policies.

Original: Over the past years, a body of literature has focused on addressing this challenge through developing conservative algorithms, which aim at picking a policy among those well-covered in the data.
Revised: Over the past years, a body of literature has focused on addressing this challenge through developing conservative algorithms, which aim at picking a policy among those well-covered in the data.

Original: On the practical front, various forms of conservatism are proposed such as behavior regularization through policy constraints (Kumar et al., 2019; Fujimoto et al., 2019; Nachum & Dai, 2020), learning conservative values (Kumar et al., 2020; Liu et al., 2020; Agarwal et al., 2020), or learning pessimistic models (Kidambi et al., 2020; Yu et al., 2020; 2021); see Appendix B for further discussion on related work.
Revised: On the practical front, various forms of conservatism are proposed such as behavior regularization through policy constraints (Kumar et al., 2019; Fujimoto et al., 2019; Nachum & Dai, 2020), learning conservative values (Kumar et al., 2020; Liu et al., 2020; Agarwal et al., 2020), or learning pessimistic models (Kidambi et al., 2020; Yu et al., 2020; 2021); see Appendix B for further discussion on related work.

Original: From a theoretical standpoint, partial data coverage has recently been studied within variants of the single-policy concentrability framework (Rashidinejad et al., 2021; Xie et al., 2021; Uehara & Sun, 2021), which characterizes the distribution shift between offline data and occupancy of a target (often optimal) policy, in contrast to all-policy concentrability commonly used in earlier works (Scherrer, 2014; Chen & Jiang, 2019;
Revised: From a theoretical standpoint, partial data coverage has recently been studied within variants of the single-policy concentrability framework (Rashidinejad et al., 2021; Xie et al., 2021; Uehara & Sun, 2021), which characterizes the distribution shift between offline data and occupancy of a target (often optimal) policy, in contrast to all-policy concentrability commonly used in earlier works (Scherrer, 2014; Chen & Jiang, 2019;

Original: Liao et al., 2020; Zhang et al., 2020a; Xie & Jiang, 2021).
Revised: Liao et al., 2020; Zhang et al., 2020a; Xie & Jiang, 2021).

Original: Within this framework and in the tabular and linear function approximation settings, pessimistic algorithms that leverage uncertainty quantifiers to construct lower confidence bounds (Jin et al., 2021;
Revised: Within this framework and in the tabular and linear function approximation settings, pessimistic algorithms that leverage uncertainty quantifiers to construct lower confidence bounds (Jin et al., 2021;

Original: Rashidinejad et al., 2021;
Revised: Rashidinejad et al., 2021;

Original: Yin et al., 2021; Shi et al., 2022; Li et al., 2022) enjoy optimal statistical rate.
Revised: Yin et al., 2021; Shi et al., 2022; Li et al., 2022) enjoy optimal statistical rate.

Original: In the general function approximation setting, pessimistic algorithms largely assume oracle access to uncertainty quantification, either for constructing penalties that are subtracted from rewards (Jin et al., 2021; Jiang & Huang, 2020) or selecting the most pessimistic option among those that fall within the confidence region implied by the offline data (Uehara & Sun, 2021; Xie et al., 2021;
Revised: In the general function approximation setting, pessimistic algorithms largely assume oracle access to uncertainty quantification, either for constructing penalties that are subtracted from rewards (Jin et al., 2021; Jiang & Huang, 2020) or selecting the most pessimistic option among those that fall within the confidence region implied by the offline data (Uehara & Sun, 2021; Xie et al., 2021;

Original: Chen & Jiang, 2022).
Revised: Chen & Jiang, 2022).

Original: However, uncertainty quantifiers are difficult to obtain in non-linear function approximation and existing heuristics are empirically observed to be unreliable (Rashid et al., 2019;
Revised: However, uncertainty quantifiers are difficult to obtain in non-linear function approximation and existing heuristics are empirically observed to be unreliable (Rashid et al., 2019;

Original: Tennenholtz et al., 2021; Yu et al., 2021).
Revised: Tennenholtz et al., 2021; Yu et al., 2021).

Original: Recent works by Cheng et al.
Revised: Recent works by Cheng et al.

Original: (2022) and Zhan et al.
Revised: (2022) and Zhan et al.

Original: (2022) propose provable alternatives to uncertainty-based methods, but leave achieving optimal statistical rate of 1 / √ N , where N is the dataset size, as an open problem.
Revised: (2022) propose provable alternatives to uncertainty-based methods, but leave achieving optimal statistical rate of 1 / √ N , where N is the dataset size, as an open problem.

Original: Among all, the marginal importance sampling (MIS) methods, which aim at learning weights w that estimate the distribution shift between induced policy occupancy d w and data distribution µ , lend themselves well to the single-policy concentrability framework.
Revised: Among all, the marginal importance sampling (MIS) methods, which aim at learning weights w that estimate the distribution shift between induced policy occupancy d w and data distribution µ , lend themselves well to the single-policy concentrability framework.

Original: Though more popular in off-policy evaluation (Liu et al., 2018; Xie et al., 2019; Uehara et al., 2020; Zhang et al., 2020b), MIS has also been used for conservative offline RL such as AlgaeDICE (Nachum et al., 2019b) and OptiDICE (Lee et al., 2021), both of which incorporate behavior regularization.
Revised: Though more popular in off-policy evaluation (Liu et al., 2018; Xie et al., 2019; Uehara et al., 2020; Zhang et al., 2020b), MIS has also been used for conservative offline RL such as AlgaeDICE (Nachum et al., 2019b) and OptiDICE (Lee et al., 2021), both of which incorporate behavior regularization.

Original: Recently, Zhan et al.
Revised: Recently, Zhan et al.

Original: (2022) theoretically studied a variant of OptiDICE, showing that MIS with behavior regularization enjoys finite-sample guarantees (though achieving a suboptimal 1 /N 1 / 6 rate) and circumvents certain fundamental difficulties observed in value-based offline RL with function approximation (Du et al., 2019; Wang et al., 2020; 2021; Weisz et al., 2021; Zanette, 2021; Foster et al., 2021).
Revised: (2022) theoretically studied a variant of OptiDICE, showing that MIS with behavior regularization enjoys finite-sample guarantees (though achieving a suboptimal 1 /N 1 / 6 rate) and circumvents certain fundamental difficulties observed in value-based offline RL with function approximation (Du et al., 2019; Wang et al., 2020; 2021; Weisz et al., 2021; Zanette, 2021; Foster et al., 2021).

Original: 1. C ONTRIBUTIONS AND RESULTS
Revised: 1. C ONTRIBUTIONS AND RESULTS

Original: Motivated by the benefits offered by MIS, we study designing statistically optimal offline learning algorithms under the MIS formulation with general function approximation and single-policy concentrability.
Revised: Motivated by the benefits offered by MIS, we study designing statistically optimal offline learning algorithms under the MIS formulation with general function approximation and single-policy concentrability.

Original: We conduct theoretical investigations and design algorithms starting from multi-armed bandits (MABs), going forward to contextual bandits (CBs), and finally Markov decision processes (MDPs).
Revised: We conduct theoretical investigations and design algorithms starting from multi-armed bandits (MABs), going forward to contextual bandits (CBs), and finally Markov decision processes (MDPs).

Original: In the rest of this section, we present a preview of our contributions and results.
Revised: In the rest of this section, we present a preview of our contributions and results.

Original: Multi-armed bandits.
Revised: Multi-armed bandits.

Original: Empirical MIS algorithms often incorporate behavior regularization, whose role is justified as promoting conservatism by keeping the occupancies of learned and behavior policies close (Nachum et al., 2019b; Lee et al., 2021).
Revised: Empirical MIS algorithms often incorporate behavior regularization, whose role is justified as promoting conservatism by keeping the occupancies of learned and behavior policies close (Nachum et al., 2019b; Lee et al., 2021).

Original: Yet, whether and why these regularizers are necessary from a theoretical perspective remain unclear.
Revised: Yet, whether and why these regularizers are necessary from a theoretical perspective remain unclear.

Original: Zhan et al.
Revised: Zhan et al.

Original: (2022) motivates behavior regularization as a way of introducing curvature in an otherwise linear optimization problem.
Revised: (2022) motivates behavior regularization as a way of introducing curvature in an otherwise linear optimization problem.

Original: We extensively investigate the effect of regularization, starting from the simplest setting of MABs with function approximation, as existing algorithms when specialized to offline MABs, are either intractable, have suboptimal finite-sample guarantees, or require access to uncertainty quantifiers.
Revised: We extensively investigate the effect of regularization, starting from the simplest setting of MABs with function approximation, as existing algorithms when specialized to offline MABs, are either intractable, have suboptimal finite-sample guarantees, or require access to uncertainty quantifiers.

Original: We state our results on offline MABs with general function approximation and single-policy concentrability in the informal theorem below.
Revised: We state our results on offline MABs with general function approximation and single-policy concentrability in the informal theorem below.

Original: Theorem (informal) (I)
Revised: Theorem (informal) (I)

Original: There exists an offline MAB instance where the unregularized MIS fails to achieve a suboptimality that decays with N .
Revised: There exists an offline MAB instance where the unregularized MIS fails to achieve a suboptimality that decays with N .

Original: (II) MIS with behavior regularization (PRO-MAB
Revised: (II) MIS with behavior regularization (PRO-MAB

Original: Algorithm 1) achieves O (1 / √ N ) suboptimality.
Revised: Algorithm 1) achieves O (1 / √ N ) suboptimality.

Original: (III)
Revised: (III)

Original: If one searches only over the space of weights that induce valid occupancies ( d w = 1) , then unregularized MIS achieves O (1 / √ N ) suboptimality.
Revised: If one searches only over the space of weights that induce valid occupancies ( d w = 1) , then unregularized MIS achieves O (1 / √ N ) suboptimality.

Original: Here, we prove that unregularized MIS fails even in bandits and provide a tight analysis of PROMAB, a special case of PRO-RL algorithm, improving over the original 1 /N 1 / 6 rate shown by Zhan et al.
Revised: Here, we prove that unregularized MIS fails even in bandits and provide a tight analysis of PROMAB, a special case of PRO-RL algorithm, improving over the original 1 /N 1 / 6 rate shown by Zhan et al.

Original: In our analysis, we find that the key to the success of PRO-MAB is near-validity of the learned occupancy d w .
Revised: In our analysis, we find that the key to the success of PRO-MAB is near-validity of the learned occupancy d w .

Original: In MABs, the validity constraint simply requires the learned occupancy to be a probability distribution: d w = (cid:80) a w ( a ) µ ( a ) = 1 , where a is an arm.
Revised: In MABs, the validity constraint simply requires the learned occupancy to be a probability distribution: d w = (cid:80) a w ( a ) µ ( a ) = 1 , where a is an arm.

Original: With a proper choice of hyperparameter, we show that behavior regularization enforces learned occupancy to be nearly valid: d w = Ω(1) .
Revised: With a proper choice of hyperparameter, we show that behavior regularization enforces learned occupancy to be nearly valid: d w = Ω(1) .

Original: We further prove that regularization is not required if validity is otherwise satisfied.
Revised: We further prove that regularization is not required if validity is otherwise satisfied.

Original: Given that occupancy validity is the constraint of the optimization problem solved by MIS (see e.g.
Revised: Given that occupancy validity is the constraint of the optimization problem solved by MIS (see e.g.

Original: (1)), we ask whether there are any methods for solving empirical optimization problems that find more constraint-adhering solutions compared to those yielded by Lagrange multipliers adopted in prior works (Lee et al., 2021; Zhan et al., 2022).
Revised: (1)), we ask whether there are any methods for solving empirical optimization problems that find more constraint-adhering solutions compared to those yielded by Lagrange multipliers adopted in prior works (Lee et al., 2021; Zhan et al., 2022).

Original: The augmented Lagrangian method (ALM), which adds a quadratic loss on the constraints ( d w − 1) 2 , is a natural choice for our purpose.
Revised: The augmented Lagrangian method (ALM), which adds a quadratic loss on the constraints ( d w − 1) 2 , is a natural choice for our purpose.

Original: The ALM term can be easily estimated from offline data and forms Algorithm 1.
Revised: The ALM term can be easily estimated from offline data and forms Algorithm 1.

Original: We show that ALM results in d w = Ω(1) , ensuring near-validity of learned occupancy and leading to the following guarantee.
Revised: We show that ALM results in d w = Ω(1) , ensuring near-validity of learned occupancy and leading to the following guarantee.

Original: Theorem (informal)
Revised: Theorem (informal)

Original: The policy returned by an algorithm that combines ALM with MIS (ALMIS) for offline MABs (Algorithm 1) achieves O (1 / √ N ) suboptimality.
Revised: The policy returned by an algorithm that combines ALM with MIS (ALMIS) for offline MABs (Algorithm 1) achieves O (1 / √ N ) suboptimality.

Original: ALMIS offers several benefits over PRO-MAB such as eliminating the need for picking the regularizer and only requiring single-policy concentrability instead of the two-policy requirement of
Revised: ALMIS offers several benefits over PRO-MAB such as eliminating the need for picking the regularizer and only requiring single-policy concentrability instead of the two-policy requirement of

Original: PRO-MAB, which can be strong (Section 5.3).
Revised: PRO-MAB, which can be strong (Section 5.3).

Original: Additionally, behavior regularization introduces bias in the solution even with infinite data (Chen & Jiang, 2022) and the bias-variance tradeoff must be carefully handled.
Revised: Additionally, behavior regularization introduces bias in the solution even with infinite data (Chen & Jiang, 2022) and the bias-variance tradeoff must be carefully handled.

Original: However, ALM merely enforces the optimization constraints and leads to provably unbiased solutions (Lemma 14).
Revised: However, ALM merely enforces the optimization constraints and leads to provably unbiased solutions (Lemma 14).

Original: More importantly, as we see shortly, going beyond the single-state MABs, behavior regularization becomes suboptimal while ALMIS maintains optimality.
Revised: More importantly, as we see shortly, going beyond the single-state MABs, behavior regularization becomes suboptimal while ALMIS maintains optimality.

Original: Contextual bandits.
Revised: Contextual bandits.

Original: In offline CBs, we analyze two approaches: MIS with behavior regularization, and an extension of ALMIS.
Revised: In offline CBs, we analyze two approaches: MIS with behavior regularization, and an extension of ALMIS.

Original: We state our results in the following informal theorem.
Revised: We state our results in the following informal theorem.

Original: Theorem (informal) (I)
Revised: Theorem (informal) (I)

Original: There exists a CB instance where MIS with behavior regularization (PROCB Algorithm 6) suffers from suboptimality Ω( N β ) with β > − 1 / 2 .
Revised: There exists a CB instance where MIS with behavior regularization (PROCB Algorithm 6) suffers from suboptimality Ω( N β ) with β > − 1 / 2 .

Original: (II) Policy returned by ALMIS for offline CBs (Algorithm 2) achieves suboptimality of O (1 / √ N ) .
Revised: (II) Policy returned by ALMIS for offline CBs (Algorithm 2) achieves suboptimality of O (1 / √ N ) .

Original: Informally, the failure of PRO-CB to achieve the optimal rate is because the regularization parameter has to be small to control bias, but such small regularization is not strong enough to ensure the validity of learned occupancy in most states.
Revised: Informally, the failure of PRO-CB to achieve the optimal rate is because the regularization parameter has to be small to control bias, but such small regularization is not strong enough to ensure the validity of learned occupancy in most states.

Original: Therefore, one must choose larger regularization, leading to an overall suboptimal rate.
Revised: Therefore, one must choose larger regularization, leading to an overall suboptimal rate.

Original: Prior works Chen & Jiang (2022); Cheng et al.
Revised: Prior works Chen & Jiang (2022); Cheng et al.

Original: (2022) also allude to this phenomenon, explaining that regularizers appear to be the culprit behind suboptimal rates.
Revised: (2022) also allude to this phenomenon, explaining that regularizers appear to be the culprit behind suboptimal rates.

Original: In CBs, the occupancy validity constraints require conditional occupancy to be a valid probability distribution in every state.
Revised: In CBs, the occupancy validity constraints require conditional occupancy to be a valid probability distribution in every state.

Original: In Algorithm 2, we incorporate ALM in offline CBs by adding a weighted sum of quadratic losses describing the validity constraint in each state, where the weights are set to the state occupancies to capture their relative importance.
Revised: In Algorithm 2, we incorporate ALM in offline CBs by adding a weighted sum of quadratic losses describing the validity constraint in each state, where the weights are set to the state occupancies to capture their relative importance.

Original: Enforcement of the constraints by ALM without introducing any bias is the key to the optimality of our algorithm.
Revised: Enforcement of the constraints by ALM without introducing any bias is the key to the optimality of our algorithm.

Original: MDPs.
Revised: MDPs.

Original: Validity constraints in MDPs ensure that the learned state occupancy d w ( s ) = (cid:80) a w ( s, a ) µ ( s, a ) is close to the actual state occupancy d π w ( s ) , where π w is the policy computed from weights w .
Revised: Validity constraints in MDPs ensure that the learned state occupancy d w ( s ) = (cid:80) a w ( s, a ) µ ( s, a ) is close to the actual state occupancy d π w ( s ) , where π w is the policy computed from weights w .

Original: 1 Directly enforcing this constraint results in an ALM term that cannot easily be estimated from offline data.
Revised: 1 Directly enforcing this constraint results in an ALM term that cannot easily be estimated from offline data.

Original: We address this difficulty by expressing the ALM term in the variational form.
Revised: We address this difficulty by expressing the ALM term in the variational form.

Original: From there, we derive two variants, one model-based and one model-free, of the ALMIS algorithm for offline RL, that enjoy the following guarantee.
Revised: From there, we derive two variants, one model-based and one model-free, of the ALMIS algorithm for offline RL, that enjoy the following guarantee.

Original: Theorem (informal) Both variants of ALMIS for offline RL achieve O (1 / √ N ) suboptimality.
Revised: Theorem (informal) Both variants of ALMIS for offline RL achieve O (1 / √ N ) suboptimality.

Original: This marks ALMIS as the first practical and statistically optimal offline RL algorithm that operates in the general function approximation and partial data coverage setting, while avoiding uncertainty quantification and additional regularizers.
Revised: This marks ALMIS as the first practical and statistically optimal offline RL algorithm that operates in the general function approximation and partial data coverage setting, while avoiding uncertainty quantification and additional regularizers.

Original: Conservatism of ALMIS is baked into the MIS formulation and supported by the ALM: bounded MIS weights prevent learned occupancy to deviate significantly from data distribution, and ALM ensures closeness of the learned and actual occupancies.
Revised: Conservatism of ALMIS is baked into the MIS formulation and supported by the ALM: bounded MIS weights prevent learned occupancy to deviate significantly from data distribution, and ALM ensures closeness of the learned and actual occupancies.

Original: When combined, ALMIS learns a policy whose actual occupancy is close to the data distribution.
Revised: When combined, ALMIS learns a policy whose actual occupancy is close to the data distribution.

Original: We thus proved that ALM improves sample complexity compared to alternatives such as behavior regularization.
Revised: We thus proved that ALM improves sample complexity compared to alternatives such as behavior regularization.

Original: This is in addition to the benefits on optimization stability that are likely to be offered by the ALM, as the ALM improves over the ill-posed Lagrange multiplier objective (BenTal & Nemirovski, 2022).
Revised: This is in addition to the benefits on optimization stability that are likely to be offered by the ALM, as the ALM improves over the ill-posed Lagrange multiplier objective (BenTal & Nemirovski, 2022).

Original: Our theoretical findings can explain the empirical observations of Yang et al.
Revised: Our theoretical findings can explain the empirical observations of Yang et al.

Original: (2020), who find MIS with behavior regularization to be unstable and propose regularizers in “the spirit of ALM” that gain superior performance and attribute performance gain to improved optimization.
Revised: (2020), who find MIS with behavior regularization to be unstable and propose regularizers in “the spirit of ALM” that gain superior performance and attribute performance gain to improved optimization.

Original: In this work, we present a theoretically-founded way of introducing ALM in offline
Revised: In this work, we present a theoretically-founded way of introducing ALM in offline

Original: RL and our analysis shows that ALM also leads to optimal sample complexity.
Revised: RL and our analysis shows that ALM also leads to optimal sample complexity.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Markov decision process.
Revised: Markov decision process.

Original: An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P, R, ρ, γ ) , where S is the state space, A is the action space, P : S ×A (cid:55)→ ∆( S ) is the transition kernel, R : S ×A (cid:55)→ ∆([0 , 1]) encodes a family of reward distributions with r : S ×A (cid:55)→ [0 , 1] as the expected reward function, ρ : S (cid:55)→ ∆( S ) is the initial state distribution, and γ ∈ [0 , 1) is the discount factor.
Revised: An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P, R, ρ, γ ) , where S is the state space, A is the action space, P : S ×A (cid:55)→ ∆( S ) is the transition kernel, R : S ×A (cid:55)→ ∆([0 , 1]) encodes a family of reward distributions with r : S ×A (cid:55)→ [0 , 1] as the expected reward function, ρ : S (cid:55)→ ∆( S ) is the initial state distribution, and γ ∈ [0 , 1) is the discount factor.

Original: We assume S and A are finite however, our results do not depend on their cardinalities and can be naturally extended to infinite sets.
Revised: We assume S and A are finite however, our results do not depend on their cardinalities and can be naturally extended to infinite sets.

Original: A stationary (stochastic) policy π : S (cid:55)→ ∆( A ) specifies a distribution over actions in each state.
Revised: A stationary (stochastic) policy π : S (cid:55)→ ∆( A ) specifies a distribution over actions in each state.

Original: Each policy π induces an occupancy density over state-action pairs d π : S ×A (cid:55)→ [0 , 1] defined as d π ( s, a ) :
Revised: Each policy π induces an occupancy density over state-action pairs d π : S ×A (cid:55)→ [0 , 1] defined as d π ( s, a ) :

Original: = (1 − γ ) (cid:80) ∞
Revised: = (1 − γ ) (cid:80) ∞

Original: t =0 γ t P t ( s t = s, a t = a ; π ) , where P t ( s t = s, a t = a ; π ) denotes ( s, a ) visitation probability at step t , starting at s 0 ∼ ρ ( · ) and following π .
Revised: t =0 γ t P t ( s t = s, a t = a ; π ) , where P t ( s t = s, a t = a ; π ) denotes ( s, a ) visitation probability at step t , starting at s 0 ∼ ρ ( · ) and following π .

Original: We abuse notation and also write d π ( s )
Revised: We abuse notation and also write d π ( s )

Original: = (cid:80) a ∈A d π ( s, a ) to denote the discounted state occupancy.
Revised: = (cid:80) a ∈A d π ( s, a ) to denote the discounted state occupancy.

Original: Additionally, operator P π is applied to any function u : S ×
Revised: Additionally, operator P π is applied to any function u : S ×

Original: A → R and is defined as ( P π u )( s, a ) :
Revised: A → R and is defined as ( P π u )( s, a ) :

Original: = (cid:80) s ′ ,a ′ P ( s ′ | s, a ) π ( a ′ | s ′ )
Revised: = (cid:80) s ′ ,a ′ P ( s ′ | s, a ) π ( a ′ | s ′ )

Original: u ( s ′ , a ′ ) .
Revised: u ( s ′ , a ′ ) .

Original: An important quantity is the value a policy π , which is the discounted sum of rewards V π ( s ) :
Revised: An important quantity is the value a policy π , which is the discounted sum of rewards V π ( s ) :

Original: =
Revised: =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: = s, a t ∼ π ( · | s t ) , ∀ t ≥ 0] starting at s ∈ S .
Revised: = s, a t ∼ π ( · | s t ) , ∀ t ≥ 0] starting at s ∈ S .

Original: Q-function Q π ( s, a ) of a policy is similarly defined.
Revised: Q-function Q π ( s, a ) of a policy is similarly defined.

Original: We write J ( π ) :
Revised: We write J ( π ) :

Original: = (1 − γ ) E s ∼ ρ
Revised: = (1 − γ ) E s ∼ ρ

Original: [ V π ( s )] =
Revised: [ V π ( s )] =

Original: E s,a ∼ d π [ r ( s, a )] to represent a scalar summary of the performance of a policy π .
Revised: E s,a ∼ d π [ r ( s, a )] to represent a scalar summary of the performance of a policy π .

Original: We denote by π ⋆ the optimal policy that maximizes the above objective and use the shorthand V ⋆ :=
Revised: We denote by π ⋆ the optimal policy that maximizes the above objective and use the shorthand V ⋆ :=

Original: V π ⋆ to denote the optimal value function.
Revised: V π ⋆ to denote the optimal value function.

Original: Offline reinforcement learning.
Revised: Offline reinforcement learning.

Original: We focus on the offline RL, where the agent is only provided with a previously-collected offline dataset D = { ( s i , a i , r i , s ′ i ) }
Revised: We focus on the offline RL, where the agent is only provided with a previously-collected offline dataset D = { ( s i , a i , r i , s ′ i ) }

Original: Ni =1 .
Revised: Ni =1 .

Original: Here, r i ∼ R ( s i , a i ) , s ′ i ∼ P ( · | s i , a i ) , and we assume s i , a i pairs are generated i.i.d.
Revised: Here, r i ∼ R ( s i , a i ) , s ′ i ∼ P ( · | s i , a i ) , and we assume s i , a i pairs are generated i.i.d.

Original: according to a data distribution µ ∈ ∆( S ×A ) .
Revised: according to a data distribution µ ∈ ∆( S ×A ) .

Original: To streamline the analysis, we assume that the conditional distribution µ ( a | s ) is known.
Revised: To streamline the analysis, we assume that the conditional distribution µ ( a | s ) is known.

Original: The goal of offline RL is to learn a policy ˆ π based on the offline dataset so as to minimize the sub-optimality with respect to the optimal policy π ⋆ , i.e. J ( π ⋆ ) − J (ˆ π ) with high probability.
Revised: The goal of offline RL is to learn a policy ˆ π based on the offline dataset so as to minimize the sub-optimality with respect to the optimal policy π ⋆ , i.e. J ( π ⋆ ) − J (ˆ π ) with high probability.

Original: Marginalized importance sampling.
Revised: Marginalized importance sampling.

Original: In this paper, we consider marginalized importance sampling (MIS) formulation that aims at learning weights w ( s, a ) to represent policy occupancy when multiplied by data distribution: d w ( s, a ) =
Revised: In this paper, we consider marginalized importance sampling (MIS) formulation that aims at learning weights w ( s, a ) to represent policy occupancy when multiplied by data distribution: d w ( s, a ) =

Original: w ( s, a ) µ ( s, a ) .
Revised: w ( s, a ) µ ( s, a ) .

Original: Also denote d w ( s ) =
Revised: Also denote d w ( s ) =

Original: (cid:80) a ∈A d w ( s, a ) .
Revised: (cid:80) a ∈A d w ( s, a ) .

Original: We define the policy induced by w as π w ( a | s ) =
Revised: We define the policy induced by w as π w ( a | s ) =

Original: d w ( s, a ) /d
Revised: d w ( s, a ) /d

Original: w ( s ) for d w ( s ) > 0 and π w ( a | s ) = 1 / |A| for d w ( s ) = 0 .
Revised: w ( s ) for d w ( s ) > 0 and π w ( a | s ) = 1 / |A| for d w ( s ) = 0 .

Original: Offline data coverage assumption.
Revised: Offline data coverage assumption.

Original: We design and analyze our algorithms within the single-policy concentrability framework (Rashidinejad et al., 2021), stated below.
Revised: We design and analyze our algorithms within the single-policy concentrability framework (Rashidinejad et al., 2021), stated below.

Original: Definition 1 (Single-policy concentrability)
Revised: Definition 1 (Single-policy concentrability)

Original: Given a policy π , define C π to be the smallest constant that satisfies d π ( s,a ) µ ( s,a ) ≤ C π for all s ∈ S and a ∈ A .
Revised: Given a policy π , define C π to be the smallest constant that satisfies d π ( s,a ) µ ( s,a ) ≤ C π for all s ∈ S and a ∈ A .

Original: C π ⋆ = C ⋆ captures coverage of π ⋆ in the offline data and is much weaker than the widely used all-policy concentrability that assumes bounded max π C π ; see Appendix B for further discussion.
Revised: C π ⋆ = C ⋆ captures coverage of π ⋆ in the offline data and is much weaker than the widely used all-policy concentrability that assumes bounded max π C π ; see Appendix B for further discussion.

Original: Notation.
Revised: Notation.

Original: Given a set S , we write |S| to represent its cardinality and ∆( S ) to denote the probability simplex over S .
Revised: Given a set S , we write |S| to represent its cardinality and ∆( S ) to denote the probability simplex over S .

Original: For a function class W , we write |W| to denote its cardinality (discrete) or covering number (continuous).
Revised: For a function class W , we write |W| to denote its cardinality (discrete) or covering number (continuous).

Original: We use the notation x ≲ y when there exists constant c > 0 such that x ≤ cy and x ≍ y if constants c 1 , c 2 > 0 exist such that c 1 | x |≤ | y |≤ c 2 | x | .
Revised: We use the notation x ≲ y when there exists constant c > 0 such that x ≤ cy and x ≍ y if constants c 1 , c 2 > 0 exist such that c 1 | x |≤ | y |≤ c 2 | x | .

Original: We write f ( x ) =
Revised: We write f ( x ) =

Original: O ( g ( x ))
Revised: O ( g ( x ))

Original: if
Revised: if

Original: M > 0 , x 0 exist such that | f ( x ) |≤ Mg ( x ) for all x ≥ x 0 and use (cid:101) O ( · ) to be the big-
Revised: M > 0 , x 0 exist such that | f ( x ) |≤ Mg ( x ) for all x ≥ x 0 and use (cid:101) O ( · ) to be the big-

Original: O notation ignoring logarithmic factors.
Revised: O notation ignoring logarithmic factors.

Original: Define clip( x, a, b ) ≜ max { a, min { x, b }} for x, a, b ∈ R .
Revised: Define clip( x, a, b ) ≜ max { a, min { x, b }} for x, a, b ∈ R .

Original:  M ULTI - ARMED BANDITS
Revised:  M ULTI - ARMED BANDITS

Original: We start by considering the offline learning problem in the multi-armed bandit (MAB) setting, which is a special case of MDP with γ = 0 , |S| = 1 , and D = { ( a i , r i ) } N i =1 , where a i ∼ µ ( · ) , r i ∼ R ( a i ) .
Revised: We start by considering the offline learning problem in the multi-armed bandit (MAB) setting, which is a special case of MDP with γ = 0 , |S| = 1 , and D = { ( a i , r i ) } N i =1 , where a i ∼ µ ( · ) , r i ∼ R ( a i ) .

Original: The goal of offline learning in MABs can be described as the following constrained optimization problem, where d represents occupancy over actions (arms)
Revised: The goal of offline learning in MABs can be described as the following constrained optimization problem, where d represents occupancy over actions (arms)

Original: [Equation]
Revised: [Equation]

Original: 3. P RIMAL - DUAL REGULARIZED OFFLINE BANDITS
Revised: 3. P RIMAL - DUAL REGULARIZED OFFLINE BANDITS

Original: To solve (1), the MIS approach with behavior regularization defines importance weights w ( a ) =
Revised: To solve (1), the MIS approach with behavior regularization defines importance weights w ( a ) =

Original: d ( a ) /µ
Revised: d ( a ) /µ

Original: ( a ) and converts the problem (1) to its dual form by introducing the Lagrange multiplier v : max
Revised: ( a ) and converts the problem (1) to its dual form by introducing the Lagrange multiplier v : max

Original: [Equation]
Revised: [Equation]

Original: The last term in (2) is the behavior regularizer that characterizes the f -divergence between the learned occupancy d and data distribution µ .
Revised: The last term in (2) is the behavior regularizer that characterizes the f -divergence between the learned occupancy d and data distribution µ .

Original: This term was originally proposed to induce conservatism by keeping the learned policy close to behavior policy (Nachum et al., 2019b; Lee et al., 2021).
Revised: This term was originally proposed to induce conservatism by keeping the learned policy close to behavior policy (Nachum et al., 2019b; Lee et al., 2021).

Original: Problem (2) satisfies strong duality and we denote optimal primal and dual variables by w ⋆α and v ⋆α (Appendix C.2).
Revised: Problem (2) satisfies strong duality and we denote optimal primal and dual variables by w ⋆α and v ⋆α (Appendix C.2).

Original: When α = 0 , weights w ⋆ := w ⋆ 0 induce optimal policy and v ⋆ :
Revised: When α = 0 , weights w ⋆ := w ⋆ 0 induce optimal policy and v ⋆ :

Original: = v ⋆ 0 is the optimal reward.
Revised: = v ⋆ 0 is the optimal reward.

Original: Approximating w and v to belong to classes W ⊆ R |A| and V ∈ R and solving the empirical version of (2) yields primal-dual regularized offline MAB (PRO-MAB Algorithm 5), a special case of PRO-RL algorithm of Zhan et al.
Revised: Approximating w and v to belong to classes W ⊆ R |A| and V ∈ R and solving the empirical version of (2) yields primal-dual regularized offline MAB (PRO-MAB Algorithm 5), a special case of PRO-RL algorithm of Zhan et al.

Original: One might wonder whether the unregularized algorithm ( α = 0 ) is sufficient for solving the offline learning problem in MABs, particularly under the natural and common assumption that elements of the function class W are bounded: w ( a ) =
Revised: One might wonder whether the unregularized algorithm ( α = 0 ) is sufficient for solving the offline learning problem in MABs, particularly under the natural and common assumption that elements of the function class W are bounded: w ( a ) =

Original: d ( a ) /µ
Revised: d ( a ) /µ

Original: ( a ) ≤ B w .
Revised: ( a ) ≤ B w .

Original: In the following proposition, we show that the answer is negative and there exist a MAB instance in which the unregularized algorithm finds a policy that suffers from a constant suboptimality.
Revised: In the following proposition, we show that the answer is negative and there exist a MAB instance in which the unregularized algorithm finds a policy that suffers from a constant suboptimality.

Original: The proof is provided in Appendix C.3.
Revised: The proof is provided in Appendix C.3.

Original: Proposition 1 (Unregularized MIS fails in MABs) Assume 0 ≤ w ( a ) ≤ B w for w ∈ W and | v |≤ B v for v ∈ V .
Revised: Proposition 1 (Unregularized MIS fails in MABs) Assume 0 ≤ w ( a ) ≤ B w for w ∈ W and | v |≤ B v for v ∈ V .

Original: Suppose realizability of w ⋆ ∈ W and v ⋆ ∈ V and concentrability of π ⋆ :=
Revised: Suppose realizability of w ⋆ ∈ W and v ⋆ ∈ V and concentrability of π ⋆ :=

Original: π w ⋆ .
Revised: π w ⋆ .

Original: For any N ≥ 2 , there exists a MAB instance where ˆ π returned by Algorithm 5 with α = 0 satisfies
Revised: For any N ≥ 2 , there exists a MAB instance where ˆ π returned by Algorithm 5 with α = 0 satisfies

Original: J ( π ⋆ ) − J (ˆ π ) = 1 / 6 with a constant probability.
Revised: J ( π ⋆ ) − J (ˆ π ) = 1 / 6 with a constant probability.

Original: We note that Zhan et al.
Revised: We note that Zhan et al.

Original: (2022) also argues the failure of the unregularized algorithm by giving a counterexample in the MDP setting.
Revised: (2022) also argues the failure of the unregularized algorithm by giving a counterexample in the MDP setting.

Original: We discuss this example in detail in Section 5.3.
Revised: We discuss this example in detail in Section 5.3.

Original: Propositionreveals additional insights: the objective (16) with α = 0 fails not just in MDPs but also in bandits, even when the optimal policy is unique and data are collected by running a behavior policy.
Revised: Propositionreveals additional insights: the objective (16) with α = 0 fails not just in MDPs but also in bandits, even when the optimal policy is unique and data are collected by running a behavior policy.

Original: Given the failure of the unregularized algorithm, we conduct a tight analysis of PRO-MAB with α > 0 .
Revised: Given the failure of the unregularized algorithm, we conduct a tight analysis of PRO-MAB with α > 0 .

Original: In the next theorem, we prove that under similar assumptions as Zhan et al.
Revised: In the next theorem, we prove that under similar assumptions as Zhan et al.

Original: (2022) and
Revised: (2022) and

Original: with a proper choice of α , PRO-MAB returns a policy that enjoys optimal sample complexity.
Revised: with a proper choice of α , PRO-MAB returns a policy that enjoys optimal sample complexity.

Original: Theorem 1 (Suboptimality of PRO-MAB) Let f be M f -strongly convex and non-negative with bounded value | f ( x ) |≤ B f and derivative |
Revised: Theorem 1 (Suboptimality of PRO-MAB) Let f be M f -strongly convex and non-negative with bounded value | f ( x ) |≤ B f and derivative |

Original: f ′ ( x ) |≤ B f ′ .
Revised: f ′ ( x ) |≤ B f ′ .

Original: Assume 0 ≤ w ( a ) ≤ B w for w ∈ W and | v |≤ B v for v ∈ V .
Revised: Assume 0 ≤ w ( a ) ≤ B w for w ∈ W and | v |≤ B v for v ∈ V .

Original: Fix δ ≥ 0 and set α ≍ B w ( B v + 1) + B f ) /M
Revised: Fix δ ≥ 0 and set α ≍ B w ( B v + 1) + B f ) /M

Original: f (cid:112) log( |V||W| /δ ) /N.
Revised: f (cid:112) log( |V||W| /δ ) /N.

Original: Suppose realizability of w ⋆α ∈ W and v ⋆α ∈ V and concentrability of π ⋆ :=
Revised: Suppose realizability of w ⋆α ∈ W and v ⋆α ∈ V and concentrability of π ⋆ :=

Original: π w ⋆ and π ⋆α :=
Revised: π w ⋆ and π ⋆α :=

Original: π w ⋆α .
Revised: π w ⋆α .

Original: Then, with probability at least 1 − δ , policy ˆ π returned by Algorithm 5 achieves
Revised: Then, with probability at least 1 − δ , policy ˆ π returned by Algorithm 5 achieves

Original: [Equation]
Revised: [Equation]

Original: To our knowledge, this is the first statistically optimal guarantee for a practical offline MAB algorithm with function approximation and partial coverage and improves over the 1 /N 1 / 6 guarantee given by Zhan et al.
Revised: To our knowledge, this is the first statistically optimal guarantee for a practical offline MAB algorithm with function approximation and partial coverage and improves over the 1 /N 1 / 6 guarantee given by Zhan et al.

Original: We now briefly explain the differences between the analysis methods; a complete proof is deferred to Appendix C.4.
Revised: We now briefly explain the differences between the analysis methods; a complete proof is deferred to Appendix C.4.

Original: Zhan et al.
Revised: Zhan et al.

Original: (2022) bounds policy suboptimality by α +1 /
Revised: (2022) bounds policy suboptimality by α +1 /

Original: ( α 1 / 2 N 1 / 4 ) , where the first term stems from the bias caused by the regularizer and the second term emerges from bounding the difference of ˆ w and w ⋆α via strong convexity of L α .
Revised: ( α 1 / 2 N 1 / 4 ) , where the first term stems from the bias caused by the regularizer and the second term emerges from bounding the difference of ˆ w and w ⋆α via strong convexity of L α .

Original: Optimizing the bound over α gives the final 1 /N 1 / 6 guarantee.
Revised: Optimizing the bound over α gives the final 1 /N 1 / 6 guarantee.

Original: In contrast, our analysis connects suboptimality to occupancy validity .
Revised: In contrast, our analysis connects suboptimality to occupancy validity .

Original: We prove that suboptimality is bounded by α
Revised: We prove that suboptimality is bounded by α

Original: + 1 / ( d ˆ w √ N ) , where d ˆ w = (cid:80) a ˆ w ( a ) µ ( a ) .
Revised: + 1 / ( d ˆ w √ N ) , where d ˆ w = (cid:80) a ˆ w ( a ) µ ( a ) .

Original: We then show that setting α ≍ 1 / √ N is sufficient to ensure near-validity of occupancy d ˆ w = Ω(1) , yielding the optimal rate.
Revised: We then show that setting α ≍ 1 / √ N is sufficient to ensure near-validity of occupancy d ˆ w = Ω(1) , yielding the optimal rate.

Original: We observe a similar phenomenon in Proposition 1 that small d w for certain w ∈ W can cause the unregularized MIS to fail.
Revised: We observe a similar phenomenon in Proposition 1 that small d w for certain w ∈ W can cause the unregularized MIS to fail.

Original: In the following section, we investigate this phenomenon further, leading to a new offline learning algorithm.
Revised: In the following section, we investigate this phenomenon further, leading to a new offline learning algorithm.

Original: 3. A UGMENTED L AGRANGIAN REPLACES BEHAVIOR REGULARIZATION
Revised: 3. A UGMENTED L AGRANGIAN REPLACES BEHAVIOR REGULARIZATION

Original: The next proposition further affirms the importance of policy validity and shows that if the occupancy is valid, such as by searching only over the weights that induce valid occupancies, then the unregularized algorithm enjoys an optimal rate.
Revised: The next proposition further affirms the importance of policy validity and shows that if the occupancy is valid, such as by searching only over the weights that induce valid occupancies, then the unregularized algorithm enjoys an optimal rate.

Original: Proof of this result can be found in Appendix C.5.
Revised: Proof of this result can be found in Appendix C.5.

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 (Constraint satisfaction is sufficient in MAB) Assume as in Theorem 1.
Revised: Proposition 2 (Constraint satisfaction is sufficient in MAB) Assume as in Theorem 1.

Original: Let ˆ π be the output of Algorithm 5 with α = 0 and assume that (cid:80) a µ ( a ) ˆ w ( a ) = 1 .
Revised: Let ˆ π be the output of Algorithm 5 with α = 0 and assume that (cid:80) a µ ( a ) ˆ w ( a ) = 1 .

Original: Then, for any δ > 0 , the following holds with probability of at least 1 − δ
Revised: Then, for any δ > 0 , the following holds with probability of at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Motivated by the discussion above, we take a step back and ask: are there any other methods for solving constrained optimization problems that find more constraint-satisfying solutions when applied to the empirical approximation of the original problem?
Revised: Motivated by the discussion above, we take a step back and ask: are there any other methods for solving constrained optimization problems that find more constraint-satisfying solutions when applied to the empirical approximation of the original problem?

Original: A promising candidate is the augmented
Revised: A promising candidate is the augmented

Original: Lagrangian method (ALM) which adds a quadratic loss on the constraints to the objective.
Revised: Lagrangian method (ALM) which adds a quadratic loss on the constraints to the objective.

Original: Applied to (1), ALM forms the following objective, whose empirical version leads to Algorithm 1.
Revised: Applied to (1), ALM forms the following objective, whose empirical version leads to Algorithm 1.

Original: [Equation]
Revised: [Equation]

Original: The following theorem establishes an upper bound on the suboptimality of the policy returned by
Revised: The following theorem establishes an upper bound on the suboptimality of the policy returned by

Original: Algorithm 1.
Revised: Algorithm 1.

Original: This theorem is a special case of Theorem 3, whose proof is given in Appendix D.3.
Revised: This theorem is a special case of Theorem 3, whose proof is given in Appendix D.3.

Original: Theorem 2 (Suboptimality of Algorithm 1) Assume that 0 ≤ w ( a ) ≤ B w for any w ∈ W and | v |≤ B v for any v ∈ V .
Revised: Theorem 2 (Suboptimality of Algorithm 1) Assume that 0 ≤ w ( a ) ≤ B w for any w ∈ W and | v |≤ B v for any v ∈ V .

Original: Further suppose realizability of w ⋆ ∈ W and v ⋆ ∈ V and concentrability of π ⋆ = π w ⋆ .
Revised: Further suppose realizability of w ⋆ ∈ W and v ⋆ ∈ V and concentrability of π ⋆ = π w ⋆ .

Original: For any fixed δ > 0 , policy ˆ π returned by Algorithm 1 achieves the following bound with probability of at least 1 − δ
Revised: For any fixed δ > 0 , policy ˆ π returned by Algorithm 1 achieves the following bound with probability of at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: In the proof, we show that ALM results in near-validity of ˆ w by ensuring that d ˆ w = Ω(1) , leading to the optimal rate.
Revised: In the proof, we show that ALM results in near-validity of ˆ w by ensuring that d ˆ w = Ω(1) , leading to the optimal rate.

Original: Note that Algorithm 1 does not include any explicit form of conservatism through regularizers or uncertainty quantifiers.
Revised: Note that Algorithm 1 does not include any explicit form of conservatism through regularizers or uncertainty quantifiers.

Original: Colloquially, the MIS formulation and boundedness of W elements ensure that d ˆ w ( a ) /µ
Revised: Colloquially, the MIS formulation and boundedness of W elements ensure that d ˆ w ( a ) /µ

Original: ( a ) =
Revised: ( a ) =

Original: ˆ w ( a ) ≤ B w and ALM ensures that d ˆ w is close to the actual occupancy.
Revised: ˆ w ( a ) ≤ B w and ALM ensures that d ˆ w is close to the actual occupancy.

Original: Thus, Algorithm 1 seeks a policy whose actual occupancy is within data distribution.
Revised: Thus, Algorithm 1 seeks a policy whose actual occupancy is within data distribution.

Original: Algorithm 1 offers several benefits compared to PRO-MAB: it only requires π ⋆ -concentrability instead of the π ⋆ , π ⋆α -concentrability requirement of PRO-MAB, removes the need to design regularization function f and adjust α , and does not introduce bias in the objective.
Revised: Algorithm 1 offers several benefits compared to PRO-MAB: it only requires π ⋆ -concentrability instead of the π ⋆ , π ⋆α -concentrability requirement of PRO-MAB, removes the need to design regularization function f and adjust α , and does not introduce bias in the objective.

Original: The main advantage of ALM, however, becomes more evident as we move beyond bandits, where the behavior regularization provably fails to achieve the optimal statistical rate while ALM maintains optimality.
Revised: The main advantage of ALM, however, becomes more evident as we move beyond bandits, where the behavior regularization provably fails to achieve the optimal statistical rate while ALM maintains optimality.

Original:  C ONTEXTUAL BANDITS
Revised:  C ONTEXTUAL BANDITS

Original: The problem offline contextual bandits (CB) is a special case of offline RL with γ = 0 and offline dataset D
Revised: The problem offline contextual bandits (CB) is a special case of offline RL with γ = 0 and offline dataset D

Original: = { ( s i , a i , r i ) } Ni =1 , where s i ∼ µ ( · ) =
Revised: = { ( s i , a i , r i ) } Ni =1 , where s i ∼ µ ( · ) =

Original: ρ ( · ) , a i ∼ µ ( · | a i ) , and r i ∼ R ( s i , a i ) .
Revised: ρ ( · ) , a i ∼ µ ( · | a i ) , and r i ∼ R ( s i , a i ) .

Original: The linear programming constrained optimization problem for CB is given by
Revised: The linear programming constrained optimization problem for CB is given by

Original: [Equation]
Revised: [Equation]

Original: 4. A NALYSIS OF PRIMAL - DUAL REGULARIZED OFFLINE CONTEXTUAL BANDITS
Revised: 4. A NALYSIS OF PRIMAL - DUAL REGULARIZED OFFLINE CONTEXTUAL BANDITS

Original: In the following proposition, we prove a performance lower bound on the primal-dual regularized offline CB (PRO-CB) presented in Algorithm 6, which is MIS with behavior regularization.
Revised: In the following proposition, we prove a performance lower bound on the primal-dual regularized offline CB (PRO-CB) presented in Algorithm 6, which is MIS with behavior regularization.

Original: Proposition 3 (PRO-CB is suboptimal)
Revised: Proposition 3 (PRO-CB is suboptimal)

Original: Let f be M f -strongly convex, differentiable, and nonnegative with bounded values | f ( x ) |≤ B f and derivative | f ′ ( x ) |≤ B f ′ .
Revised: Let f be M f -strongly convex, differentiable, and nonnegative with bounded values | f ( x ) |≤ B f and derivative | f ′ ( x ) |≤ B f ′ .

Original: Assume 0 ≤ w ( s, a ) ≤ B w for w ∈ W , | v ( s ) |≤ B v , realizability w ⋆ , w ⋆α ∈ W , v ⋆ , v ⋆α ∈ V , and concentrability of π ⋆ , π ⋆α .
Revised: Assume 0 ≤ w ( s, a ) ≤ B w for w ∈ W , | v ( s ) |≤ B v , realizability w ⋆ , w ⋆α ∈ W , v ⋆ , v ⋆α ∈ V , and concentrability of π ⋆ , π ⋆α .

Original: Let ˆ π be the output of Algorithm 6.
Revised: Let ˆ π be the output of Algorithm 6.

Original: Then, for N ≥ poly ( δ, B w , B v , B f , B f ′ ) and any α ≥ 0 there exists a CB instance such that J ( π ⋆ ) − J (ˆ π ) ≳
Revised: Then, for N ≥ poly ( δ, B w , B v , B f , B f ′ ) and any α ≥ 0 there exists a CB instance such that J ( π ⋆ ) − J (ˆ π ) ≳

Original: N β with a constant probability, where β > − 1 / 2 .
Revised: N β with a constant probability, where β > − 1 / 2 .

Original: Algorithm
Revised: Algorithm

Original: : Inputs
Revised: : Inputs

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , function classes W
Revised: , function classes W

Original: [Equation]
Revised: [Equation]

Original: V 2: Find a solution ˆ w, ˆ v to the following problem
Revised: V 2: Find a solution ˆ w, ˆ v to the following problem

Original: [Equation]
Revised: [Equation]

Original: Proposition 3 shows that behavior regularization is statistically suboptimal regardless of α .
Revised: Proposition 3 shows that behavior regularization is statistically suboptimal regardless of α .

Original: The proof is presented in Appendix D.2.
Revised: The proof is presented in Appendix D.2.

Original: The main takeaway is that ensuring occupancy validity (cid:80) a ˆ w ( s, a ) µ ( a | s ) = Ω(1) for nearly all states appears to be critical in achieving the optimal rate.
Revised: The main takeaway is that ensuring occupancy validity (cid:80) a ˆ w ( s, a ) µ ( a | s ) = Ω(1) for nearly all states appears to be critical in achieving the optimal rate.

Original: Yet, without introducing a large bias, behavior regularization is insufficient for such a guarantee.
Revised: Yet, without introducing a large bias, behavior regularization is insufficient for such a guarantee.

Original: 4. O FFLINE CONTEXTUAL BANDITS WITH AUGMENTED L AGRANGIAN
Revised: 4. O FFLINE CONTEXTUAL BANDITS WITH AUGMENTED L AGRANGIAN

Original: To encourage occupancy validity, we extend ALM to CBs and propose the following objective:
Revised: To encourage occupancy validity, we extend ALM to CBs and propose the following objective:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When |S| = 1 , (7) simplifies to the ALM objective (2) for MABs.
Revised: When |S| = 1 , (7) simplifies to the ALM objective (2) for MABs.

Original: The ALM term can be understood as follows.
Revised: The ALM term can be understood as follows.

Original: Each element encourages the validity of occupancy in each state (cid:80) a w ( s, a ) µ ( s, a ) ≈and the elements are weighted according to the true state distribution: validity is more important in states that are actually more likely to be visited.
Revised: Each element encourages the validity of occupancy in each state (cid:80) a w ( s, a ) µ ( s, a ) ≈and the elements are weighted according to the true state distribution: validity is more important in states that are actually more likely to be visited.

Original: Denote by w ⋆ the optimal solution to (7), which is equal to the optimal solution of (6), and define v ⋆ ( s ) :
Revised: Denote by w ⋆ the optimal solution to (7), which is equal to the optimal solution of (6), and define v ⋆ ( s ) :

Original: = V ⋆ ( s ) .
Revised: = V ⋆ ( s ) .

Original: The following theorem states that
Revised: The following theorem states that

Original: ALMIS achieves optimal rate in offline CBs, whose proof is in Appendix D.3.
Revised: ALMIS achieves optimal rate in offline CBs, whose proof is in Appendix D.3.

Original: Theorem 3 (Suboptimality of Algorithm 2) Assume 0 ≤ w ( s, a ) ≤ B w for w ∈ W and v ( s ) ≤
Revised: Theorem 3 (Suboptimality of Algorithm 2) Assume 0 ≤ w ( s, a ) ≤ B w for w ∈ W and v ( s ) ≤

Original: B v for v ∈ V .
Revised: B v for v ∈ V .

Original: Moreover, suppose realizability of w ⋆ ∈ W and v ⋆ ∈ V and concentrability of π ⋆ = π w ⋆ .
Revised: Moreover, suppose realizability of w ⋆ ∈ W and v ⋆ ∈ V and concentrability of π ⋆ = π w ⋆ .

Original: For any fixed δ ≥ 0 , policy ˆ π returned by Algorithm 2 achieves the following suboptimality bound with probability of at least 1 − δ
Revised: For any fixed δ ≥ 0 , policy ˆ π returned by Algorithm 2 achieves the following suboptimality bound with probability of at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original:  M ARKOV DECISION PROCESSES
Revised:  M ARKOV DECISION PROCESSES

Original: We now turn to offline RL.
Revised: We now turn to offline RL.

Original: In addition to the offline dataset, we assume access to a dataset D 

Revised: In addition to the offline dataset, we assume access to a dataset D

Original: = { s i } Ni =1 with i.i.d.
Revised: = { s i } Ni =1 with i.i.d.

Original: samples from the initial distribution ρ , similar to prior works (Lee et al., 2021;
Revised: samples from the initial distribution ρ , similar to prior works (Lee et al., 2021;

Original: Zhan et al., 2022).
Revised: Zhan et al., 2022).

Original: The linear programming formulation of RL (Puterman, 2014) solves
Revised: The linear programming formulation of RL (Puterman, 2014) solves

Original: [Equation]
Revised: [Equation]

Original: The constraints are known as Bellman flow equations and restrict the search to the space of valid occupancy distributions d π that can be induced in the MDP by running a policy π .
Revised: The constraints are known as Bellman flow equations and restrict the search to the space of valid occupancy distributions d π that can be induced in the MDP by running a policy π .

Original: 5. C ONSERVATIVE OFFLINE RL WITH AUGMENTED L AGRANGIAN
Revised: 5. C ONSERVATIVE OFFLINE RL WITH AUGMENTED L AGRANGIAN

Original: Motivated by the success of ALM in bandits, we propose the following extension to offline RL:
Revised: Motivated by the success of ALM in bandits, we propose the following extension to offline RL:

Original: [Equation]
Revised: [Equation]

Original: where e v ( s, a ) :
Revised: where e v ( s, a ) :

Original: = r ( s, a ) + γ (cid:80) s
Revised: = r ( s, a ) + γ (cid:80) s

Original: ′ P ( s ′ | s, a ) v ( s ′ )
Revised: ′ P ( s ′ | s, a ) v ( s ′ )

Original: − v ( s ) .
Revised: − v ( s ) .

Original: One can check that the first two terms are the Lagrange dual of (9) and the last term is a generalization of the ALM terms in bandits.
Revised: One can check that the first two terms are the Lagrange dual of (9) and the last term is a generalization of the ALM terms in bandits.

Original: The
Revised: The

Original: ALM elements encourage occupancy d w ( s ) to be close in ratio to the actual occupancy d π w ( s ) in each state and as before, the ALM elements are weighted according to their actual visitation d π w ( s ) .
Revised: ALM elements encourage occupancy d w ( s ) to be close in ratio to the actual occupancy d π w ( s ) in each state and as before, the ALM elements are weighted according to their actual visitation d π w ( s ) .

Original: Our particular ALM construction can be intuitively understood as follows.
Revised: Our particular ALM construction can be intuitively understood as follows.

Original: The MIS formulation learns bounded weights ˆ w ( s, a ) =
Revised: The MIS formulation learns bounded weights ˆ w ( s, a ) =

Original: d ˆ w ( s, a ) /µ
Revised: d ˆ w ( s, a ) /µ

Original: ( s, a ) ≤ B w .
Revised: ( s, a ) ≤ B w .

Original: The ALM term ensures that the ratio d ˆ w ( s ) /d π ˆ w ( s ) = Ω(1) which roughly translates to d π ˆ w ( s, a ) /µ
Revised: The ALM term ensures that the ratio d ˆ w ( s ) /d π ˆ w ( s ) = Ω(1) which roughly translates to d π ˆ w ( s, a ) /µ

Original: ( s, a )
Revised: ( s, a )

Original: ≲ B w .
Revised: ≲ B w .

Original: The ALM term in (10) is difficult to estimate as it involves the expectation over unknown occupancy d π w and the computation of the ratio d w ( s ) /d
Revised: The ALM term in (10) is difficult to estimate as it involves the expectation over unknown occupancy d π w and the computation of the ratio d w ( s ) /d

Original: π w ( s ) .
Revised: π w ( s ) .

Original: We resolve this difficulty in the next sections.
Revised: We resolve this difficulty in the next sections.

Original: 1: Inputs: Datasets D , D 0 , D m , function classes W , V , U , P , f − 1 ∗ ( x ) = 2 √ x + 1 − 2 .
Revised: 1: Inputs: Datasets D , D 0 , D m , function classes W , V , U , P , f − 1 ∗ ( x ) = 2 √ x + 1 − 2 .

Original: 2: Estimate transitions via maximum likelihood: ˆ P = argmax P ∈P (cid:80) N m i =
Revised: 2: Estimate transitions via maximum likelihood: ˆ P = argmax P ∈P (cid:80) N m i =

Original: 1 ln P ( s ′ i
Revised: 1 ln P ( s ′ i

Original: | s i , a i ) .
Revised: | s i , a i ) .

Original: 3: Find a solution ˆ w, ˆ v, ˆ u to the following problem
Revised: 3: Find a solution ˆ w, ˆ v, ˆ u to the following problem

Original: [Equation]
Revised: [Equation]

Original: Return: ˆ π = π ˆ w .
Revised: Return: ˆ π = π ˆ w .

Original: 5. E STIMATING THE ALM TERM AND ALMIS ALGORITHMS FOR OFFLINE RL
Revised: 5. E STIMATING THE ALM TERM AND ALMIS ALGORITHMS FOR OFFLINE RL

Original: We view the ALM term as the negative f -divergence between d w and d π w with f ( x ) :=
Revised: We view the ALM term as the negative f -divergence between d w and d π w with f ( x ) :=

Original: ( x − 1)and express it in the variational form (Nguyen et al., 2010):
Revised: ( x − 1)and express it in the variational form (Nguyen et al., 2010):

Original: [Equation]
Revised: [Equation]

Original: Here, f ∗ is the convex conjugate of f and we used the fact
Revised: Here, f ∗ is the convex conjugate of f and we used the fact

Original: that d w ( s, a ) /d
Revised: that d w ( s, a ) /d

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: s ) /
Revised: s ) /

Original: [Equation]
Revised: [Equation]

Original: Notice that E d πw [ f ∗ ( x ( s, a ))] is the value of π w in the same MDP but with rewards f ∗ ( x ( s, a )) .
Revised: Notice that E d πw [ f ∗ ( x ( s, a ))] is the value of π w in the same MDP but with rewards f ∗ ( x ( s, a )) .

Original: Define u as the fixed point of the following Bellman equation
Revised: Define u as the fixed point of the following Bellman equation

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since u ( s, a ) is the Q-function of π w with rewards f ∗ ( x ( s, a )) , we can rewrite (11) as
Revised: Since u ( s, a ) is the Q-function of π w with rewards f ∗ ( x ( s, a )) , we can rewrite (11) as

Original: [Equation]
Revised: [Equation]

Original: Equation (13) involves expectations over ρ and µ , which can be estimated empirically.
Revised: Equation (13) involves expectations over ρ and µ , which can be estimated empirically.

Original: Below, we discuss model-free and model-based methods for estimating the term involving the transition operator P π w .
Revised: Below, we discuss model-free and model-based methods for estimating the term involving the transition operator P π w .

Original: We include some details on practical implementations in Appendix E.1.
Revised: We include some details on practical implementations in Appendix E.1.

Original: Model-based ALMIS.
Revised: Model-based ALMIS.

Original: For the model-based route, we assume access to a class P that contains the true transitions and an additional dataset D m =
Revised: For the model-based route, we assume access to a class P that contains the true transitions and an additional dataset D m =

Original: ( s i , a i , s ′ i ) } N m i =1 , where s i , a i ∼ µ and s ′ i ∼
Revised: ( s i , a i , s ′ i ) } N m i =1 , where s i , a i ∼ µ and s ′ i ∼

Original: P ( .
Revised: P ( .

Original: | s i , a i ) .
Revised: | s i , a i ) .

Original: Given D m , we obtain a maximum likelihood estimate of transitions and approximate the expectations using D 0 and D , which leads to model-based ALMIS for offline RL (Algorithm 3).
Revised: Given D m , we obtain a maximum likelihood estimate of transitions and approximate the expectations using D 0 and D , which leads to model-based ALMIS for offline RL (Algorithm 3).

Original: Model-free ALMIS.
Revised: Model-free ALMIS.

Original: As an alternative, we consider developing a model-free that uses a singlesample estimate of f − 1 ∗ ( u ( s, a ) − γ ( P π w u )( s, a )) .
Revised: As an alternative, we consider developing a model-free that uses a singlesample estimate of f − 1 ∗ ( u ( s, a ) − γ ( P π w u )( s, a )) .

Original: This, however, roughly leads to the infamous double sampling problem (Baird, 1995).
Revised: This, however, roughly leads to the infamous double sampling problem (Baird, 1995).

Original: To circumvent this difficulty, in Appendix E.2 we use the dual embedding trick of Nachum et al. (2019a), to derive model-free ALMIS (Algorithm 4).
Revised: To circumvent this difficulty, in Appendix E.2 we use the dual embedding trick of Nachum et al. (2019a), to derive model-free ALMIS (Algorithm 4).

Original: Theorem 4 shows that ALMIS for offline RL enjoys optimal rates; see Appendix E.3 for the proof.
Revised: Theorem 4 shows that ALMIS for offline RL enjoys optimal rates; see Appendix E.3 for the proof.

Original: Theorem 4 (Suboptimality of ALMIS for offline RL) Assume 0 ≤ w ( s, a ) ≤ B w for w ∈ W , | v ( s ) |≤ B v for v ∈ V , and | u ( s, a ) |≤ B u .
Revised: Theorem 4 (Suboptimality of ALMIS for offline RL) Assume 0 ≤ w ( s, a ) ≤ B w for w ∈ W , | v ( s ) |≤ B v for v ∈ V , and | u ( s, a ) |≤ B u .

Original: Suppose realizability of w ⋆ ∈ W and v ⋆ ( s )
Revised: Suppose realizability of w ⋆ ∈ W and v ⋆ ( s )

Original: =
Revised: =

Original: V ⋆ ( s ) ∈
Revised: V ⋆ ( s ) ∈

Original: V and concentrability of π ⋆ = π w ⋆ .
Revised: V and concentrability of π ⋆ = π w ⋆ .

Original: Let ˜ x w ( s, a ) =
Revised: Let ˜ x w ( s, a ) =

Original: clip( x ⋆w ( s, a ) , − B x , B x ) , where x ⋆w is a solution to (11) and B x = (1 − γ ) / 4 , and define u ⋆w as the fixed-point solution to (12) when x = ˜ x w .
Revised: clip( x ⋆w ( s, a ) , − B x , B x ) , where x ⋆w is a solution to (11) and B x = (1 − γ ) / 4 , and define u ⋆w as the fixed-point solution to (12) when x = ˜ x w .

Original: Assume u ⋆w ∈ U for any w ∈ W .
Revised: Assume u ⋆w ∈ U for any w ∈ W .

Original: Then, B u satisfies (1 − γ ) − 1 ( B 2 x / 4 + B x ) ≤ B u ≤ 12 .
Revised: Then, B u satisfies (1 − γ ) − 1 ( B 2 x / 4 + B x ) ≤ B u ≤ 12 .

Original: Moreover, for any fixed δ ≥ 0 , the following statements hold:
Revised: Moreover, for any fixed δ ≥ 0 , the following statements hold:

Original: (I) Assume N = N 0 = N m for simplicity.
Revised: (I) Assume N = N 0 = N m for simplicity.

Original: If P ⋆ ∈ P , then ˆ π returned by Algorithm 3 achieves
Revised: If P ⋆ ∈ P , then ˆ π returned by Algorithm 3 achieves

Original: [Equation]
Revised: [Equation]

Original: (II) Assume N = N 0 for simplicity.
Revised: (II) Assume N = N 0 for simplicity.

Original: Let ζ ⋆ w,u = argmax ζ< 

Revised: Let ζ ⋆ w,u = argmax ζ<

Original: L model-free AL ( w, v, u, ζ ) defined in (54) .
Revised: L model-free AL ( w, v, u, ζ ) defined in (54) .

Original: Assume ζ
Revised: Assume ζ

Original: ⋆w
Revised: ⋆w

Original: ⋆ ,u ∈ Z for u ∈ U and B ζ,L ≤ | ζ ( s, a ) |≤ B ζ,U for ζ ∈ Z , where B ζ,L ∈ (0 , 2 / (2 + B x )) and B ζ,U ≥ 2 / (2 − B x ) .
Revised: ⋆ ,u ∈ Z for u ∈ U and B ζ,L ≤ | ζ ( s, a ) |≤ B ζ,U for ζ ∈ Z , where B ζ,L ∈ (0 , 2 / (2 + B x )) and B ζ,U ≥ 2 / (2 − B x ) .

Original: Let B ζ = max { B ζ,U , B − 1 ζ,L } .
Revised: Let B ζ = max { B ζ,U , B − 1 ζ,L } .

Original: Then, ˆ π returned by Algorithm 4 achieves
Revised: Then, ˆ π returned by Algorithm 4 achieves

Original: [Equation]
Revised: [Equation]

Original: N .
Revised: N .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In Theorem 4, we make realizability assumptions on u ⋆w for w ∈ W and ζ ⋆w ⋆ ,u for u ∈ U .
Revised: In Theorem 4, we make realizability assumptions on u ⋆w for w ∈ W and ζ ⋆w ⋆ ,u for u ∈ U .

Original: Such assumptions are common in theory of RL with function approximation (Munos & Szepesvári, 2008;
Revised: Such assumptions are common in theory of RL with function approximation (Munos & Szepesvári, 2008;

Original: Xie et al., 2021; Jiang & Huang, 2020) and removing them can be difficult or impossible (Foster et al., 2021).
Revised: Xie et al., 2021; Jiang & Huang, 2020) and removing them can be difficult or impossible (Foster et al., 2021).

Original: Recently, Zhan et al.
Revised: Recently, Zhan et al.

Original: ; Chen & Jiang (2022) propose algorithms that only require optimal solution realizability, however, these algorithms are either intractable or suboptimal.
Revised: ; Chen & Jiang (2022) propose algorithms that only require optimal solution realizability, however, these algorithms are either intractable or suboptimal.

Original: 5. E XAMPLE : B EHAVIOR REGULARIZATION VS . AUGMENTED L AGRANGIAN
Revised: 5. E XAMPLE : B EHAVIOR REGULARIZATION VS . AUGMENTED L AGRANGIAN

Original: [Equation]
Revised: [Equation]

Original: R L
Revised: R L

Original: L
Revised: L

Original: R the agent collects +1 reward.
Revised: R the agent collects +1 reward.

Original: Action R leads to C , from where only one action leads to a +1 reward.
Revised: Action R leads to C , from where only one action leads to a +1 reward.

Original: Nature decides which MDP is presented to the learner.
Revised: Nature decides which MDP is presented to the learner.

Original: Data distribution is µ ( A, L ) = 1 / 4 , µ ( A, R ) = 1 / 2 , µ ( B ) = 1 / 4 , µ ( C ) = 0 , which satisfies π w 1 -concentrability.
Revised: Data distribution is µ ( A, L ) = 1 / 4 , µ ( A, R ) = 1 / 2 , µ ( B ) = 1 / 4 , µ ( C ) = 0 , which satisfies π w 1 -concentrability.

Original: We examine the MDP example in Figure 1 presented by Zhan et al.
Revised: We examine the MDP example in Figure 1 presented by Zhan et al.

Original: Assume V = { v ⋆ } and W = { w 1 , w 2 } , where w 1 always selects L from A and w 2 always selects R from A .
Revised: Assume V = { v ⋆ } and W = { w 1 , w 2 } , where w 1 always selects L from A and w 2 always selects R from A .

Original: One can check w 1 ( A, L ) = 2 , w 1 ( A, R )
Revised: One can check w 1 ( A, L ) = 2 , w 1 ( A, R )

Original: = 0 and w 2 ( A, L ) = 0 , w 2 ( A, R )
Revised: = 0 and w 2 ( A, L ) = 0 , w 2 ( A, R )

Original: = 1 .
Revised: = 1 .

Original: Unregularized algorithm.
Revised: Unregularized algorithm.

Original: As Zhan et al.
Revised: As Zhan et al.

Original: (2022) state, the unregularized algorithm fails to distinguish between w 1 and w 2 even with infinite data as the objectives at w 1 and w 2 are exactly equal.
Revised: (2022) state, the unregularized algorithm fails to distinguish between w 1 and w 2 even with infinite data as the objectives at w 1 and w 2 are exactly equal.

Original: Behavior regularization.
Revised: Behavior regularization.

Original: Consider an instantiation of PRORL with regularizer − α E µ [ w 2 ( s, a )] .
Revised: Consider an instantiation of PRORL with regularizer − α E µ [ w 2 ( s, a )] .

Original: Since in this example E µ [ w 21 ( s, a )] > E µ [ w 22 ( s, a )] , PRO-RL picks the wrong weight w 2 , suffering constant suboptimality.
Revised: Since in this example E µ [ w 21 ( s, a )] > E µ [ w 22 ( s, a )] , PRO-RL picks the wrong weight w 2 , suffering constant suboptimality.

Original: Note, however, that PRORL guarantees assume π ⋆α -concentrability.
Revised: Note, however, that PRORL guarantees assume π ⋆α -concentrability.

Original: Intuitively, behavior regularization causes π ⋆α to be more stochastic and thus requiring µ ( s, a ) > 0 for more states and actions.
Revised: Intuitively, behavior regularization causes π ⋆α to be more stochastic and thus requiring µ ( s, a ) > 0 for more states and actions.

Original: Here, since µ covers both ( A, L ) and ( A, R ) , behavior regularization causes π ⋆α ( R | A ) >and thus d π ⋆α ( C ) > 0 .
Revised: Here, since µ covers both ( A, L ) and ( A, R ) , behavior regularization causes π ⋆α ( R | A ) >and thus d π ⋆α ( C ) > 0 .

Original: To handle the MDP in Figure 1, PRO-RL additionally requires µ ( C ) > 0 to satisfy π ⋆α -concentrability.
Revised: To handle the MDP in Figure 1, PRO-RL additionally requires µ ( C ) > 0 to satisfy π ⋆α -concentrability.

Original: ALM.
Revised: ALM.

Original: In this example, ALM successfully picks the optimal w 1 , as it avoids a mismatch between the actual and learned occupancies.
Revised: In this example, ALM successfully picks the optimal w 1 , as it avoids a mismatch between the actual and learned occupancies.

Original: This is because in (10) the ALM term is zero at w 1 due to realizability whereas at w 2 , it has a lower bound E s ∼ d π 2 ( d w 2 ( C ) /d
Revised: This is because in (10) the ALM term is zero at w 1 due to realizability whereas at w 2 , it has a lower bound E s ∼ d π 2 ( d w 2 ( C ) /d

Original: π 2 ( C ) − 1) 2 ≥
Revised: π 2 ( C ) − 1) 2 ≥

Original: d
Revised: d

Original: π 2 ( C ) > 0 .
Revised: π 2 ( C ) > 0 .

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We present a set of practical and statistically optimal algorithms for offline MAB, CB, and RL, under general function approximation and single-policy concentrability.
Revised: We present a set of practical and statistically optimal algorithms for offline MAB, CB, and RL, under general function approximation and single-policy concentrability.

Original: Our algorithms are designed within the MIS formulation combined with a novel application of augmented Lagrangian method.
Revised: Our algorithms are designed within the MIS formulation combined with a novel application of augmented Lagrangian method.

Original: Importantly, our optimality guarantees hold under MIS combined with ALM alone, without any additional form of conservatism such as via regularization or uncertainty quantification.
Revised: Importantly, our optimality guarantees hold under MIS combined with ALM alone, without any additional form of conservatism such as via regularization or uncertainty quantification.

Original: Furthermore, we investigate the role of regularizers in MIS algorithms.
Revised: Furthermore, we investigate the role of regularizers in MIS algorithms.

Original: Although the empirical benefits of such regularizers are often attributed to conservatism, our analysis suggests that conservatism stems from the MIS formulation while the role of regularizers is to ensure the validity of learned occupancy.
Revised: Although the empirical benefits of such regularizers are often attributed to conservatism, our analysis suggests that conservatism stems from the MIS formulation while the role of regularizers is to ensure the validity of learned occupancy.

Original: Interesting future directions include conducting empirical evaluations of ALM, examining the possibility of removing strong realizability assumptions, and investigating practical and optimal offline
Revised: Interesting future directions include conducting empirical evaluations of ALM, examining the possibility of removing strong realizability assumptions, and investigating practical and optimal offline

Original: RL algorithms whose guarantees hold under milder variants of single-policy concentrability more suited to function approximation.
Revised: RL algorithms whose guarantees hold under milder variants of single-policy concentrability more suited to function approximation.

Original: A R ESULTS SUMMARY
Revised: A R ESULTS SUMMARY

Original: [Table]
Revised: [Table]

Original: B R ELATED WORK
Revised: B R ELATED WORK

Original: We covered a number of related works in the introduction and throughout the paper.
Revised: We covered a number of related works in the introduction and throughout the paper.

Original: In this section, we review more related literature
Revised: In this section, we review more related literature

Original: [Table]
Revised: [Table]

Original: B. C ONCENTRABILITY ASSUMPTIONS
Revised: B. C ONCENTRABILITY ASSUMPTIONS

Original: The lack of sufficient coverage in the offline dataset is one of the main challenges in offline RL.
Revised: The lack of sufficient coverage in the offline dataset is one of the main challenges in offline RL.

Original: In RL theory, dataset coverage has often been characterized by concentrability definitions (Munos, 2007;
Revised: In RL theory, dataset coverage has often been characterized by concentrability definitions (Munos, 2007;

Original: Scherrer, 2014).
Revised: Scherrer, 2014).

Original: Earlier works on offline RL impose all-policy concentrability on the density ratio for all states and actions (Scherrer, 2014; Liu et al., 2019a;
Revised: Earlier works on offline RL impose all-policy concentrability on the density ratio for all states and actions (Scherrer, 2014; Liu et al., 2019a;

Original: Chen & Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a), with some requiring this ratio to be bounded for every time step (Szepesvári & Munos, 2005; Munos, 2007; Antos et al., 2008; Farahmand et al., 2010; Antos et al., 2007).
Revised: Chen & Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a), with some requiring this ratio to be bounded for every time step (Szepesvári & Munos, 2005; Munos, 2007; Antos et al., 2008; Farahmand et al., 2010; Antos et al., 2007).

Original: The works Xie & Jiang (2021); Feng et al. (2019); Uehara et al.
Revised: The works Xie & Jiang (2021); Feng et al. (2019); Uehara et al.

Original: (2020) use slightly milder definitions, such as requiring a bound on a weighted norm of density ratios.
Revised: (2020) use slightly milder definitions, such as requiring a bound on a weighted norm of density ratios.

Original: The work Xie & Jiang (2021) makes even stronger assumptions such as lower bounded conditionals µ ( a | s ) and exploratoriness of state marginals to circumvent the Bellman completeness requirement.
Revised: The work Xie & Jiang (2021) makes even stronger assumptions such as lower bounded conditionals µ ( a | s ) and exploratoriness of state marginals to circumvent the Bellman completeness requirement.

Original: To handle partial coverage, recent algorithms are analyzed based on variants of single-policy concentrability (Rashidinejad et al., 2021).
Revised: To handle partial coverage, recent algorithms are analyzed based on variants of single-policy concentrability (Rashidinejad et al., 2021).

Original: Some variants such as the ones presented in works Uehara & Sun (2021) (model-based) or Xie et al. (2021); Song et al.
Revised: Some variants such as the ones presented in works Uehara & Sun (2021) (model-based) or Xie et al. (2021); Song et al.

Original: (2022) (model-free) are more suited to function approximation as they avoid bounded ratio assumption for all states and actions.
Revised: (2022) (model-free) are more suited to function approximation as they avoid bounded ratio assumption for all states and actions.

Original: However, existing offline RL algorithms based on these weaker definitions are either computationally intractable (Uehara & Sun, 2021; Xie et al., 2021) or their statistical rate is suboptimal (Cheng et al., 2022).
Revised: However, existing offline RL algorithms based on these weaker definitions are either computationally intractable (Uehara & Sun, 2021; Xie et al., 2021) or their statistical rate is suboptimal (Cheng et al., 2022).

Original: The most related works are Zhan et al.
Revised: The most related works are Zhan et al.

Original: (2022), which requires two-policy concentrability, and Chen & Jiang (2022), which requires single-policy concentrability on density ratio for all states and actions.
Revised: (2022), which requires two-policy concentrability, and Chen & Jiang (2022), which requires single-policy concentrability on density ratio for all states and actions.

Original: B. C ONSERVATIVE OFFLINE RL
Revised: B. C ONSERVATIVE OFFLINE RL

Original: A series of recent works on offline RL have focused on addressing partial coverage of offline dataset through conservative algorithm design.
Revised: A series of recent works on offline RL have focused on addressing partial coverage of offline dataset through conservative algorithm design.

Original: Broadly speaking, these methods can be broken down into several categories.
Revised: Broadly speaking, these methods can be broken down into several categories.

Original: The first category of methods applies policy constraints, enforcing the learned policy to be close to the behavior policy.
Revised: The first category of methods applies policy constraints, enforcing the learned policy to be close to the behavior policy.

Original: Such constraints are applied either explicitly (Fujimoto et al., 2019;
Revised: Such constraints are applied either explicitly (Fujimoto et al., 2019;

Original: Ghasemipour et al., 2020; Jaques et al., 2019; Siegel et al., 2020; Kumar et al., 2019;
Revised: Ghasemipour et al., 2020; Jaques et al., 2019; Siegel et al., 2020; Kumar et al., 2019;

Original: Wu et al., 2019; Fujimoto & Gu, 2021), implicitly (Peng et al., 2019; Nair et al., 2020), or through importance sampling (Liu et al., 2019b; Swaminathan & Joachims, 2015; Nachum et al., 2019b;
Revised: Wu et al., 2019; Fujimoto & Gu, 2021), implicitly (Peng et al., 2019; Nair et al., 2020), or through importance sampling (Liu et al., 2019b; Swaminathan & Joachims, 2015; Nachum et al., 2019b;

Original: Lee et al., 2021; Zhang et al., 2020c;b).
Revised: Lee et al., 2021; Zhang et al., 2020c;b).

Original: Another category involves learning conservative values such as conservative Q-learning (Kumar et al., 2020), fitted Q-iteration with conservative update (Liu et al., 2020), subtracting penalties (Rezaeifar et al., 2022), and critic regularization (Kostrikov et al., 2021).
Revised: Another category involves learning conservative values such as conservative Q-learning (Kumar et al., 2020), fitted Q-iteration with conservative update (Liu et al., 2020), subtracting penalties (Rezaeifar et al., 2022), and critic regularization (Kostrikov et al., 2021).

Original: The last category includes model-based methods such as learning pessimistic models (Kidambi et al., 2020; Guo et al., 2022), adversarial model learning (Rigter et al., 2022), forming penalties using model ensembles (Yu et al., 2020), or incorporating a combination of model and values (Yu et al., 2021).
Revised: The last category includes model-based methods such as learning pessimistic models (Kidambi et al., 2020; Guo et al., 2022), adversarial model learning (Rigter et al., 2022), forming penalties using model ensembles (Yu et al., 2020), or incorporating a combination of model and values (Yu et al., 2021).

Original: On the theoretical side, as discussed in the introduction, the majority of works design pessimistic offline RL algorithms that rely on some form of uncertainty quantification (Yin & Wang, 2021; Uehara et al., 2021;
Revised: On the theoretical side, as discussed in the introduction, the majority of works design pessimistic offline RL algorithms that rely on some form of uncertainty quantification (Yin & Wang, 2021; Uehara et al., 2021;

Original: Zhang et al., 2022; Yan et al., 2022; Yin et al., 2022; Kumar et al., 2021; Shi & Chi, 2022; Wang et al., 2022).
Revised: Zhang et al., 2022; Yan et al., 2022; Yin et al., 2022; Kumar et al., 2021; Shi & Chi, 2022; Wang et al., 2022).

Original: One exception is the work of Zanette et al.
Revised: One exception is the work of Zanette et al.

Original: (2021) that uses value-function perturbation with actor-critic in linear function approximation setting.
Revised: (2021) that uses value-function perturbation with actor-critic in linear function approximation setting.

Original: Other examples include the recent theoretical works on MIS (Zhan et al., 2022; Chen & Jiang, 2022) and adversarially trained actor-critic (Cheng et al., 2022).
Revised: Other examples include the recent theoretical works on MIS (Zhan et al., 2022; Chen & Jiang, 2022) and adversarially trained actor-critic (Cheng et al., 2022).

Original: Most related to our work are methods that focus on provable conservative offline RL under general function approximation and partial coverage, summarized under the pessimistic algorithms segment of Table B.2.
Revised: Most related to our work are methods that focus on provable conservative offline RL under general function approximation and partial coverage, summarized under the pessimistic algorithms segment of Table B.2.

Original: Uehara & Sun (2021) propose a pessimistic model-based algorithm that under a generalization of single-policy concentrability to bounded TV distance ratio, enjoys a 1 / √ N rate but is computationally intractable.
Revised: Uehara & Sun (2021) propose a pessimistic model-based algorithm that under a generalization of single-policy concentrability to bounded TV distance ratio, enjoys a 1 / √ N rate but is computationally intractable.

Original: The work of Xie et al.
Revised: The work of Xie et al.

Original: (2021) presents a pessimistic modelfree algorithm under a variant of single-policy concentrability framework that requires a bounded ratio of average Bellman error and Bellman completeness.
Revised: (2021) presents a pessimistic modelfree algorithm under a variant of single-policy concentrability framework that requires a bounded ratio of average Bellman error and Bellman completeness.

Original: While the original version of the algorithm achieves the optimal 1 / √ N rate, it is computationally intractable.
Revised: While the original version of the algorithm achieves the optimal 1 / √ N rate, it is computationally intractable.

Original: A practical version of the algorithm is presented and has a suboptimal 1 /N 1 / 5 guarantee.
Revised: A practical version of the algorithm is presented and has a suboptimal 1 /N 1 / 5 guarantee.

Original: Another related work by Chen &
Revised: Another related work by Chen &

Original: Jiang (2022) studies MIS combined with value function approximation under π ⋆ -concentrability and proves a 1 / (cid:112) gap ( Q ⋆ ) N rate, yet the guarantee degrades with Q ⋆ gap and the algorithm is computationally intractable.
Revised: Jiang (2022) studies MIS combined with value function approximation under π ⋆ -concentrability and proves a 1 / (cid:112) gap ( Q ⋆ ) N rate, yet the guarantee degrades with Q ⋆ gap and the algorithm is computationally intractable.

Original: Cheng et al.
Revised: Cheng et al.

Original: (2022) propose an adversarially trained actor-critic method that enjoys provable 1 /N 1 / 3 rate under the single-policy concentrability definition of Xie et al.
Revised: (2022) propose an adversarially trained actor-critic method that enjoys provable 1 /N 1 / 3 rate under the single-policy concentrability definition of Xie et al.

Original: (2021) and Bellman completeness and performs well in offline RL benchmarks when combined with deep neural networks
Revised: (2021) and Bellman completeness and performs well in offline RL benchmarks when combined with deep neural networks

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (this work)
Revised: (this work)

Original: [Table]
Revised: [Table]

Original: B. O THER TOPICS
Revised: B. O THER TOPICS

Original: Apart from RL, our work on bandits is related to the selection problem (Hong et al., 2021), though the majority of works in this area are in the online setting.
Revised: Apart from RL, our work on bandits is related to the selection problem (Hong et al., 2021), though the majority of works in this area are in the online setting.

Original: Additionally, in our analysis, we solve a subset of stochastic optimization problems with possibly large or infinite stochastic constraints involving conditional expectations.
Revised: Additionally, in our analysis, we solve a subset of stochastic optimization problems with possibly large or infinite stochastic constraints involving conditional expectations.

Original: To our knowledge, finite-sample properties of such stochastic optimization problems have not been addressed (Shapiro et al., 2021) and our work may open up avenues for further research in this area.
Revised: To our knowledge, finite-sample properties of such stochastic optimization problems have not been addressed (Shapiro et al., 2021) and our work may open up avenues for further research in this area.

Original: C S UPPLEMENTARY MATERIALS FOR MULTI - ARMED BANDITS
Revised: C S UPPLEMENTARY MATERIALS FOR MULTI - ARMED BANDITS

Original: We start by presenting a pseudocode for the behavior regularized MIS algorithm in Appendix C.1.
Revised: We start by presenting a pseudocode for the behavior regularized MIS algorithm in Appendix C.1.

Original: In
Revised: In

Original: Appendix C.2, we characterize the bias caused by adding the behavior regularization in the primaldual objective (2).
Revised: Appendix C.2, we characterize the bias caused by adding the behavior regularization in the primaldual objective (2).

Original: In Appendix C.3, we prove Proposition 1 that demonstrates the failure of unregularized MIS for solving offline MABs, even when the optimal solutions are realizable and the optimal policy is covered in the offline data.
Revised: In Appendix C.3, we prove Proposition 1 that demonstrates the failure of unregularized MIS for solving offline MABs, even when the optimal solutions are realizable and the optimal policy is covered in the offline data.

Original: Appendix C.4 is devoted to the proof of Theorem 1, which gives a tight performance upper bound of the PRO-MAB algorithm.
Revised: Appendix C.4 is devoted to the proof of Theorem 1, which gives a tight performance upper bound of the PRO-MAB algorithm.

Original: Finally in Appendix
Revised: Finally in Appendix

Original: C.5, we prove Proposition 2, showing that constraint satisfaction is sufficient for the success of unregularized MIS.
Revised: C.5, we prove Proposition 2, showing that constraint satisfaction is sufficient for the success of unregularized MIS.

Original: C. P RIMAL - DUAL REGULARIZED OFFLINE MULTI - ARMED BANDITS (PRO-MAB)
Revised: C. P RIMAL - DUAL REGULARIZED OFFLINE MULTI - ARMED BANDITS (PRO-MAB)

Original: Algorithm
Revised: Algorithm

Original: Inputs
Revised: Inputs

Original: Dataset
Revised: Dataset

Original: [Equation]
Revised: [Equation]

Original: ( w
Revised: ( w

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: w ( a
Revised: w ( a

Original: [Equation]
Revised: [Equation]

Original: − αf ( w ( a i )) − v ( w ( a i ) − 1) .
Revised: − αf ( w ( a i )) − v ( w ( a i ) − 1) .

Original: C. S OLUTIONS TO THE PRIMAL - DUAL REGULARIZED OBJECTIVE
Revised: C. S OLUTIONS TO THE PRIMAL - DUAL REGULARIZED OBJECTIVE

Original: In the following lemma, we characterize the optimal solution ( w ⋆α , v ⋆α ) to the behavior-regularized population objective (2) as well as the suboptimality of the policy induced by w ⋆α .
Revised: In the following lemma, we characterize the optimal solution ( w ⋆α , v ⋆α ) to the behavior-regularized population objective (2) as well as the suboptimality of the policy induced by w ⋆α .

Original: Lemma 1 (Regularized primal-dual solutions, MAB) Let f be differentiable, strictly convex, nonnegative, and bounded by B f .
Revised: Lemma 1 (Regularized primal-dual solutions, MAB) Let f be differentiable, strictly convex, nonnegative, and bounded by B f .

Original: Denote r ⋆ := max a ∈A r ( a ) .
Revised: Denote r ⋆ := max a ∈A r ( a ) .

Original: Then, the following statements hold:
Revised: Then, the following statements hold:

Original: (I) w ⋆ 0 = w ⋆ , where w ⋆ is the importance weight corresponding to an optimal policy; (II) v ⋆α =
Revised: (I) w ⋆ 0 = w ⋆ , where w ⋆ is the importance weight corresponding to an optimal policy; (II) v ⋆α =

Original: r ⋆ − cα , where 0 ≤ c ≤ f ′ ( C ⋆ ) ; (III) policy π ⋆α :=
Revised: r ⋆ − cα , where 0 ≤ c ≤ f ′ ( C ⋆ ) ; (III) policy π ⋆α :=

Original: π w ⋆α satisfies J ( π ⋆ ) − J ( π ⋆α ) ≤ αB f .
Revised: π w ⋆α satisfies J ( π ⋆ ) − J ( π ⋆α ) ≤ αB f .

Original: Proof.
Revised: Proof.

Original: Part (I) follows directly by strong duality.
Revised: Part (I) follows directly by strong duality.

Original: For part (II), notice that KKT conditions imply the following relation between w ⋆α ( a ) and v ⋆α :
Revised: For part (II), notice that KKT conditions imply the following relation between w ⋆α ( a ) and v ⋆α :

Original: [Equation]
Revised: [Equation]

Original: Since f is strictly convex, f ′ is a monotonically increasing function.
Revised: Since f is strictly convex, f ′ is a monotonically increasing function.

Original: Therefore, the optimal arm a ⋆ has the largest w ⋆α ( a ) , which should be nonzero due to realizability of w ⋆α .
Revised: Therefore, the optimal arm a ⋆ has the largest w ⋆α ( a ) , which should be nonzero due to realizability of w ⋆α .

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: We now proceed to find a bound on f ′ ( w ⋆α ( a ⋆ )) .
Revised: We now proceed to find a bound on f ′ ( w ⋆α ( a ⋆ )) .

Original: Since w ⋆α is the optimal solution to (2), it must satisfy the constraint
Revised: Since w ⋆α is the optimal solution to (2), it must satisfy the constraint

Original: [Equation]
Revised: [Equation]

Original: where the last inequality stems from the single-policy concentrability assumption of π ∗ .
Revised: where the last inequality stems from the single-policy concentrability assumption of π ∗ .

Original: Since f ′ is an increasing function, we have f ′ ( w ⋆α ( a ⋆ ))
Revised: Since f ′ is an increasing function, we have f ′ ( w ⋆α ( a ⋆ ))

Original: ≤ f ′ ( C ⋆ ) , which combined with (17) yields the following lower bound on v ⋆α
Revised: ≤ f ′ ( C ⋆ ) , which combined with (17) yields the following lower bound on v ⋆α

Original: [Equation]
Revised: [Equation]

Original: Moreover, the convexity of f immediately gives the upper bound on v ⋆α ≤ r ⋆ , which completes the proof of part (II).
Revised: Moreover, the convexity of f immediately gives the upper bound on v ⋆α ≤ r ⋆ , which completes the proof of part (II).

Original: We now prove the last part.
Revised: We now prove the last part.

Original: Since w ⋆α is the optimal solution to the regularized population objective (2), by strong duality, we have
Revised: Since w ⋆α is the optimal solution to the regularized population objective (2), by strong duality, we have

Original: [Equation]
Revised: [Equation]

Original: where d w ⋆α ( a ) = µ ( a ) w ⋆α ( a ) by definition ( ?? ) and we used the fact that E a ∼ µ [ w ⋆α ( a )] − 1 =
Revised: where d w ⋆α ( a ) = µ ( a ) w ⋆α ( a ) by definition given in Section 2 and we used the fact that E a ∼ µ [ w ⋆α ( a )] − 1 =

Original: E a ∼ µ [ w ⋆ ( a )] − 1 = 0 .
Revised: E a ∼ µ [ w ⋆ ( a )] − 1 = 0 .

Original: Therefore, the suboptimality of π ⋆α can be bounded as follows
Revised: Therefore, the suboptimality of π ⋆α can be bounded as follows

Original: [Equation]
Revised: [Equation]

Original: where in the second to last inequality we used the non-negativity of f and in the last equality, we used the boundedness of f .
Revised: where in the second to last inequality we used the non-negativity of f and in the last equality, we used the boundedness of f .

Original: □
Revised: □

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: Consider a 2-armed bandit instance with the following reward distributions, data distribution, and function classes.
Revised: Consider a 2-armed bandit instance with the following reward distributions, data distribution, and function classes.

Original: • Reward distributions: The first arm is optimal with deterministic reward and the second arm has a Bernoulli distribution:
Revised: • Reward distributions: The first arm is optimal with deterministic reward and the second arm has a Bernoulli distribution:

Original: [Equation]
Revised: [Equation]

Original: • Data distribution: We consider a scenario where most data are concentrated on the optimal arm:
Revised: • Data distribution: We consider a scenario where most data are concentrated on the optimal arm:

Original: [Equation]
Revised: [Equation]

Original: Here, the single-policy concentrability coefficient is C ⋆ = 1 /µ (1) and is finite for N > 2 .
Revised: Here, the single-policy concentrability coefficient is C ⋆ = 1 /µ (1) and is finite for N > 2 .

Original: Let N ( a ) denotethe
Revised: Let N ( a ) denote

Original:  number of samples on arm a .
Revised: the number of samples on arm a .

Original: To obtain upper and lower bounds on
Revised: To obtain upper and lower bounds on

Original: N ( a ) , we resort to the following lemma, which is a direct consequence of the Chernoff bound for binomial variables.
Revised: N ( a ) , we resort to the following lemma, which is a direct consequence of the Chernoff bound for binomial variables.

Original: Lemma 2 (Chernoff bounds, binomial)
Revised: Lemma 2 (Chernoff bounds, binomial)

Original: (I) With probability at least 1 − exp( − Nµ ( a ) δ 2 u / (2 + δ u )) , one has N ( a ) ≤ (1 + δ u ) Nµ ( a ) for any δ u > 0 ; (II) With probability at least 1 − exp( − Nµ ( a )
Revised: (I) With probability at least 1 − exp( − Nµ ( a ) δ 2 u / (2 + δ u )) , one has N ( a ) ≤ (1 + δ u ) Nµ ( a ) for any δ u > 0 ; (II) With probability at least 1 − exp( − Nµ ( a )

Original: δ 2 l / 2) , one has (1 − δ l ) Nµ ( a ) ≤ N ( a ) for any 0 < δ l < 1 .
Revised: δ 2 l / 2) , one has (1 − δ l ) Nµ ( a ) ≤ N ( a ) for any 0 < δ l < 1 .

Original: We condition on the event that the number of samples on the second arm is between 1 andwhich occurs with probability larger than 1 − exp (cid:16) − 2 · 0 .
Revised: We condition on the event that the number of samples on the second arm is between 1 andwhich occurs with probability larger than 1 − exp (cid:16) − 2 · 0 .

Original: 9 2 2 (cid:17) − exp (cid:16) − 2 · 1 .
Revised: 9 2 2 (cid:17) − exp (cid:16) − 2 · 1 .

Original: 95+2 (cid:17) ≥ 0 .due
Revised: 95+2 (cid:17) ≥ 0 .due

Original: to Lemma 2 when setting δ l = 0 .
Revised: to Lemma 2 when setting δ l = 0 .

Original: 9 and δ u = 1 .
Revised: 9 and δ u = 1 .

Original: (1 − 0 . 9) · Nµ (2) ≤ N (2) ≤ (1 + 1 . 95) · Nµ (2) [Equation]
Revised: (1 − 0 . 9) · Nµ (2) ≤ N (2) ≤ (1 + 1 . 95) · Nµ (2) ⇒ 1 ≤ N (2) ≤ 5 .

Original: • Function classes: Assume that W = { w 

Revised: • Function classes: Assume that W = { w

Original: =
Revised: =

Original: ( C ⋆ , 0) , w 2 = (0 , B w ) } and V = { 1 / 2 } .
Revised: ( C ⋆ , 0) , w 2 = (0 , B w ) } and V = { 1 / 2 } .

Original: By Lemma 1, we have v ⋆ 0 =
Revised: By Lemma 1, we have v ⋆ 0 =

Original: r ⋆ = 1 / 2 .
Revised: r ⋆ = 1 / 2 .

Original: Therefore, the problem is realizable as v ⋆ 0 ∈ V and w ⋆ 0 = w ⋆ =
Revised: Therefore, the problem is realizable as v ⋆ 0 ∈ V and w ⋆ 0 = w ⋆ =

Original: ( C ⋆ , 0) ∈ W .
Revised: ( C ⋆ , 0) ∈ W .

Original: Furthermore, notice that for the second candidate w 2 = (0 , B w ) ∈ W , the normalization factor is small for a constant B w as d w 2 = (cid:80) a w 2 ( a ) µ ( a ) =
Revised: Furthermore, notice that for the second candidate w 2 = (0 , B w ) ∈ W , the normalization factor is small for a constant B w as d [Equation] a [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B w /N .
Revised: /N .

Original: Consider the case where all N ( a ) samples on the second arm observe a reward of 1, which happens with a probability of at least 1 3 5 as we conditioned on the event that 1 ≤ N (2) ≤ 5 .
Revised: Consider the case where all N ( a ) samples on the second arm observe a reward of 1, which happens with a probability of at least 1 3 5 as we conditioned on the event that 1 ≤ N (2) ≤ 5 .

Original: We now compute
Revised: We now compute

Original: ˆ w by solving the empirical objective (16) with α = 0 .
Revised: ˆ w by solving the empirical objective (16) with α = 0 .

Original: Note that since |V| = 1 , it suffices to compute ˆ w = arg max w ∈W ˆ L MAB( w, v = 1 / 2) .
Revised: Note that since |V| = 1 , it suffices to compute ˆ w = arg max w ∈W ˆ L MAB( w, v = 1 / 2) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Since we conditioned on the event with N (2) ≥ 1 , solving the optimization problem
Revised: Since we conditioned on the event with N (2) ≥ 1 , solving the optimization problem

Original: [Equation]
Revised: [Equation]

Original: ( w, v = 1 / 2) finds ˆ w = (0 , B w ) , leading to a policy that picks the second arm with probability one.
Revised: ( w, v = 1 / 2) finds ˆ w = (0 , B w ) , leading to a policy that picks the second arm with probability one.

Original: Therefore, with constant probability of 0 .
Revised: Therefore, with constant probability of 0 .

Original: 4 × 1 / 3 5 > 0 .
Revised: 4 × 1 / 3 5 > 0 .

Original: 001 , we have
Revised: 001 , we have

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: Before embarking on the main proof, we present two lemmas related to the primal-dual regularized approach.
Revised: Before embarking on the main proof, we present two lemmas related to the primal-dual regularized approach.

Original: The first lemma shows the closeness of population objective (2) to its empirical approximation used in Algorithm 1, which is a direct consequence of Hoeffding’s inequality.
Revised: The first lemma shows the closeness of population objective (2) to its empirical approximation used in Algorithm 1, which is a direct consequence of Hoeffding’s inequality.

Original: We also show that closeness of objectives results in the closeness of w ⋆α and ˆ w , which are respectively the optimums to (2) and (16).
Revised: We also show that closeness of objectives results in the closeness of w ⋆α and ˆ w , which are respectively the optimums to (2) and (16).

Original: The proof of this lemma is deferred to the end of this subsection.
Revised: The proof of this lemma is deferred to the end of this subsection.

Original: Lemma 3 (Empirical and population closeness, PRO-MAB) Fix δ > 0 and define
Revised: Lemma 3 (Empirical and population closeness, PRO-MAB) Fix δ > 0 and define

Original: [Equation]
Revised: [Equation]

Original: For any w ∈ W and v ∈ V , the following bounds hold with probability at least 1 − δ
Revised: For any w ∈ W and v ∈ V , the following bounds hold with probability at least 1 − δ

Original: (I) | L MAB α ( w, v ) − ˆ L MAB α ( w, v ) |≤ ϵ MABstat ,α ; (II) L MAB α ( w ⋆α , v ) − L MAB α ( ˆ w, v ) ≤ 2 ϵ MABstat ,α .
Revised: (I) | L MAB α ( w, v ) − ˆ L MAB α ( w, v ) |≤ ϵ MABstat ,α ; (II) L MAB α ( w ⋆α , v ) − L MAB α ( ˆ w, v ) ≤ 2 ϵ MABstat ,α .

Original: The second lemma finds a lower bound on the occupancy normalization factor d ˆ w = (cid:80) a ˆ w ( a ) µ ( a ) enforced by the behavior regularization.
Revised: The second lemma finds a lower bound on the occupancy normalization factor d ˆ w = (cid:80) a ˆ w ( a ) µ ( a ) enforced by the behavior regularization.

Original: Lemma 4 (Occupancy validity enforced by behavior regularization) Let f be an M f -stronglyconvex function and fix δ > 0 .
Revised: Lemma 4 (Occupancy validity enforced by behavior regularization) Let f be an M f -stronglyconvex function and fix δ > 0 .

Original: Then, with probability at least 1 − δ , one has
Revised: Then, with probability at least 1 − δ , one has

Original: [Equation]
Revised: [Equation]

Original: where ϵ MABstat ,α is defined in (18) .
Revised: where ϵ MABstat ,α is defined in (18) .

Original: For the rest of this proof, we condition on the high probability events of Lemmas 3 and 4.
Revised: For the rest of this proof, we condition on the high probability events of Lemmas 3 and 4.

Original: Define
Revised: Define

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By part (II) of Lemma 3, we have L MAB α
Revised: By part (II) of Lemma 3, we have L MAB α

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Recall from Lemma 1 that we have v ⋆α =
Revised: Recall from Lemma 1 that we have v ⋆α =

Original: r ⋆ − αc , where c ≤ f ′ ( C ⋆ ) .
Revised: r ⋆ − αc , where c ≤ f ′ ( C ⋆ ) .

Original: Thus, combined with (20), we write
Revised: Thus, combined with (20), we write

Original: [Equation]
Revised: [Equation]

Original: 21) where in the second line we used the bounds | f ( x ) |≤ B f and d ˆ w ≤ B w .
Revised: 21) where in the second line we used the bounds | f ( x ) |≤ B f and d ˆ w ≤ B w .

Original: Note that setting α = 16 ϵ MAB stat , 1 /M f , Lemma 4 asserts that d ˆ w ≥ 1 / 2 .
Revised: Note that setting α = 16 ϵ MAB stat , 1 /M f , Lemma 4 asserts that d ˆ w ≥ 1 / 2 .

Original: Since d ˆ w ≥ 1 / 2 , the learned policy is written as
Revised: Since d ˆ w ≥ 1 / 2 , the learned policy is written as

Original: ˆ π = ˆ w ( a ) µ ( a ) /d
Revised: ˆ π = ˆ w ( a ) µ ( a ) /d

Original: ˆ w .
Revised: ˆ w .

Original: With simple algebraic manipulations, we find the following expression for the suboptimality of ˆ π with respect to π ⋆α :
Revised: With simple algebraic manipulations, we find the following expression for the suboptimality of ˆ π with respect to π ⋆α :

Original: [Equation]
Revised: [Equation]

Original: Let ϵ reg =
Revised: Let ϵ reg =

Original: J ( π ⋆ ) − J ( π ⋆α ) =
Revised: J ( π ⋆ ) − J ( π ⋆α ) =

Original: r ⋆ − J ( π ⋆α ) denote the suboptimality suffered due to behavior regularization.
Revised: r ⋆ − J ( π ⋆α ) denote the suboptimality suffered due to behavior regularization.

Original: Suboptimality J ( π ⋆α ) − J (ˆ π ) can be expressed as
Revised: Suboptimality J ( π ⋆α ) − J (ˆ π ) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: We use the above inequality to bound the suboptimality with respect to the optimal policy:
Revised: We use the above inequality to bound the suboptimality with respect to the optimal policy:

Original: [Equation]
Revised: [Equation]

Original: Recall that we have 1 /d ˆ w ≤ 2 and that ϵ reg is bounded by αB f by Lemma 1.
Revised: Recall that we have 1 /d ˆ w ≤ 2 and that ϵ reg is bounded by αB f by Lemma 1.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where the penultimate inequality relies on the bound derived in (21).
Revised: where the penultimate inequality relies on the bound derived in (21).

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: ˆ L MAB α ( w, v ) is an empirical average over independent and bounded random variables, where the bound on individual variables is computed as
Revised: ˆ L MAB α ( w, v ) is an empirical average over independent and bounded random variables, where the bound on individual variables is computed as

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that E D [ˆ L MAB α ( w, v )]
Revised: It is easy to see that E D [ˆ L MAB α ( w, v )]

Original: =
Revised: =

Original: L MAB α ( w, v ) , where the expectation is taken with respect to the randomness in dataset D .
Revised: L MAB α ( w, v ) , where the expectation is taken with respect to the randomness in dataset D .

Original: Part (I) of this lemma is proved by applying Hoeffding’s inequality along with a union bound on w and v .
Revised: Part (I) of this lemma is proved by applying Hoeffding’s inequality along with a union bound on w and v .

Original: The proof of part (II) is similar to Lemma 7 of Zhan et al.
Revised: The proof of part (II) is similar to Lemma 7 of Zhan et al.

Original: and relies on decomposing the objective difference and using the fact that ( ˆ w, ˆ v ) correspond to the saddle points of L MAB α and ˆ L MAB α .
Revised: and relies on decomposing the objective difference and using the fact that ( ˆ w, ˆ v ) correspond to the saddle points of L MAB α and ˆ L MAB α .

Original: For any w ∈ W , define
Revised: For any w ∈ W , define

Original: [Equation]
Revised: [Equation]

Original: We write
Revised: We write

Original: L MAB α
Revised: L MAB α

Original: [Equation]
Revised: [Equation]

Original: Each term is bounded as follows:
Revised: Each term is bounded as follows:

Original: • T 1 = 0 because w ⋆α satisfies the constraint (cid:80) a w ⋆α ( a ) µ ( a ) = 1 and for
Revised: • T 1 = 0 because w ⋆α satisfies the constraint (cid:80) a w ⋆α ( a ) µ ( a ) = 1 and for

Original: any
Revised: any

Original: v 1 , v 2 we have L MAB α ( w ⋆α , v 1 ) =
Revised: v 1 , v 2 we have L MAB α ( w ⋆α , v 1 ) =

Original: L MAB α ( w ⋆α , v 2 ) .
Revised: L MAB α ( w ⋆α , v 2 ) .

Original: • T 2 ≤ ϵ stat due to Lemma 3.
Revised: • T 2 ≤ ϵ stat due to Lemma 3.

Original: • T 3 ≤ 0 because ˆ w = arg max w ∈W ˆ L α (ˆ v w , w ) .
Revised: • T 3 ≤ 0 because ˆ w = arg max w ∈W ˆ L α (ˆ v w , w ) .

Original: • T 4 ≤ 0 because ˆ v = arg min v ∈V ˆ L MAB α
Revised: • T 4 ≤ 0 because ˆ v = arg min v ∈V ˆ L MAB α

Original: ( v, ˆ w ) .
Revised: ( v, ˆ w ) .

Original: • T 5 ≤ ϵ stat due to Lemma 3.
Revised: • T 5 ≤ ϵ stat due to Lemma 3.

Original: Summing up the bounds on each term yields the desired bound.
Revised: Summing up the bounds on each term yields the desired bound.

Original: □
Revised: □

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: This lemma is a direct consequence of Lemma 8 in Zhan et al.
Revised: This lemma is a direct consequence of Lemma 8 in Zhan et al.

Original: For completeness, we present a simplified proof for the multi-armed bandit setting.
Revised: For completeness, we present a simplified proof for the multi-armed bandit setting.

Original: First, observe that since f is M f -strongly-convex, the function L MAB α ( v ⋆α , w ) is αM f -stronglyconcave with respect to w and norm ∥·∥ 2 ,µ .
Revised: First, observe that since f is M f -strongly-convex, the function L MAB α ( v ⋆α , w ) is αM f -stronglyconcave with respect to w and norm ∥·∥ 2 ,µ .

Original: Furthermore, since w ⋆α = arg max w L MAB α ( v ⋆ , w ) , we have
Revised: Furthermore, since w ⋆α = arg max w L MAB α ( v ⋆ , w ) , we have

Original: [Equation]
Revised: [Equation]

Original: The above bound along with the bound on L MAB α
Revised: The above bound along with the bound on L MAB α

Original: [Equation]
Revised: [Equation]

Original: L MAB α ( ˆ w, v ⋆α ) ≤ 2 ϵ MABstat ,α showed in
Revised: L MAB α ( ˆ w, v ⋆α ) ≤ 2 ϵ MABstat ,α showed in

Original: Lemma 1, give the following bound on | d ˆ w − 1 |
Revised: Lemma 1, give the following bound on | d ˆ w − 1 |

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: □
Revised: □

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: Consider the difference between population objective with α = 0 at w ⋆ 0 = w ⋆ and ˆ w , which is bounded by Lemma 3:
Revised: Consider the difference between population objective with α = 0 at w ⋆ 0 = w ⋆ and ˆ w , which is bounded by Lemma 3:

Original: [Equation]
Revised: [Equation]

Original: We have E a ∼ µ [ w ⋆ ( a )]
Revised: We have E a ∼ µ [ w ⋆ ( a )]

Original: = 1 due to realizability and E a ∼ µ [ ˆ w ( a )]
Revised: = 1 due to realizability and E a ∼ µ [ ˆ w ( a )]

Original: = 1 is our assumption.
Revised: = 1 is our assumption.

Original: Thus the second term in (23) is zero.
Revised: Thus the second term in (23) is zero.

Original: Moreover, note that ˆ π ( a ) =
Revised: Moreover, note that ˆ π ( a ) =

Original: ˆ w ( a ) µ ( a ) / E a ∼ µ [ w ( a )]
Revised: ˆ w ( a ) µ ( a ) / E a ∼ µ [ w ( a )]

Original: = ˆ w ( a ) .
Revised: = ˆ w ( a ) .

Original: Substituting the expression for ϵ MABstat ,α from (18) with α = 0 , we obtain
Revised: Substituting the expression for ϵ MABstat ,α from (18) with α = 0 , we obtain

Original: J
Revised: J

Original: [Equation]
Revised: [Equation]

Original: − J (
Revised: − J (

Original: [Equation]
Revised: [Equation]

Original: = E a
Revised: = E a

Original: [Equation]
Revised: [Equation]

Original: µ [ r
Revised: µ [ r

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: w ⋆
Revised: w ⋆

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: + 1)
Revised: + 1)

Original: [Equation]
Revised: [Equation]

Original: N , where we used the fact that B w ≍ B w + 1 since B w ≥ 1 due to realizability of w ⋆ .
Revised: N , where we used the fact that B w ≍ B w + 1 since B w ≥ 1 due to realizability of w ⋆ .

Original: D P ROOFS FOR CONTEXTUAL BANDITS
Revised: D P ROOFS FOR CONTEXTUAL BANDITS

Original: This section of the appendix is organized as follows.
Revised: This section of the appendix is organized as follows.

Original: In Appendix D.1, we present details of the
Revised: In Appendix D.1, we present details of the

Original: PRO-CB algorithm.
Revised: PRO-CB algorithm.

Original: Appendix D.2 is devoted to the proof of Proposition 1, which shows that
Revised: Appendix D.2 is devoted to the proof of Proposition 1, which shows that

Original: the
Revised: the

Original: PRO-CB algorithm fails to achieve the statistically optimal rate of 1 / √ N .
Revised: PRO-CB algorithm fails to achieve the statistically optimal rate of 1 / √ N .

Original: The proof of suboptimality upper bound for the conservative offline CB algorithm with ALM is presented in Theorem 3.
Revised: The proof of suboptimality upper bound for the conservative offline CB algorithm with ALM is presented in Theorem 3.

Original: D. P RIMAL - DUAL REGULARIZED OFFLINE CONTEXTUAL BANDITS (PRO-CB)
Revised: D. P RIMAL - DUAL REGULARIZED OFFLINE CONTEXTUAL BANDITS (PRO-CB)

Original: Define importance weights w ( s, a ) =
Revised: Define importance weights w ( s, a ) =

Original: d ( s, a ) /µ
Revised: d ( s, a ) /µ

Original: ( s, a ) to denote the ratio of occupancy and data distribution.
Revised: ( s, a ) to denote the ratio of occupancy and data distribution.

Original: The primal-dual regularized approach (Zhan et al., 2022) solves the following population objective
Revised: The primal-dual regularized approach (Zhan et al., 2022) solves the following population objective

Original: [Equation]
Revised: [Equation]

Original: The above optimization problem satisfies strong duality.
Revised: The above optimization problem satisfies strong duality.

Original: We define w ⋆α , v ⋆α to respectively denote the optimal solutions to the primal and dual variables.
Revised: We define w ⋆α , v ⋆α to respectively denote the optimal solutions to the primal and dual variables.

Original: Approximating w, v to belong to function classes
Revised: Approximating w, v to belong to function classes

Original: W , V and solving the empirical version of objective (24) leads to the PRO-CB given in Algorithm 6.
Revised: W , V and solving the empirical version of objective (24) leads to the PRO-CB given in Algorithm 6.

Original: 1: Inputs: Dataset D = { ( s i , a
Revised: 1: Inputs: Dataset D = { ( s i , a

Original: , r i ) } Ni =1 , function classes W , V , function f ( · ) , parameter
Revised: , r i ) } Ni =1 , function classes W , V , function f ( · ) , parameter

Original: 2: Find a solution ˆ w, ˆ to the following problem
Revised: 2: Find a solution ˆ w, ˆ to the following problem

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3: Return: ˆ π = π ˆ w .
Revised: 3: Return: ˆ π = π ˆ w .

Original: D. P ROOF OF P ROPOSITION
Revised: D. P ROOF OF P ROPOSITION

Original: We separate the proof into two cases: α ≥ N β for β > − 1 / 2 and α ≤ (cid:101) O
Revised: We separate the proof into two cases: α ≥ N β for β > − 1 / 2 and α ≤ (cid:101) O

Original: ( N − 1 / 2 ) .
Revised: ( N − 1 / 2 ) .

Original: When α is large, we show that the large bias caused by regularization results in suboptimality of α even in MABs.
Revised: When α is large, we show that the large bias caused by regularization results in suboptimality of α even in MABs.

Original: When α is small, we construct a two-state CB instance (as
Revised: When α is small, we construct a two-state CB instance (as

Original: the single-state case is indeed successful due to
Revised: the single-state case is indeed successful due to

Original: Theorem 1)
Revised: Theorem 1)

Original: , showing
Revised: , showing

Original: that
Revised: that

Original: such small α does not sufficiently enforce occupancy validity in states with a relatively small but still significant state distribution ρ ( s )
Revised: such small α does not sufficiently enforce occupancy validity in states with a relatively small but still significant state distribution ρ ( s )

Original: D.2. P ROOF FOR LARGE α
Revised: D.2. P ROOF FOR LARGE α

Original: If there exists − 12 < β such that α ≥ N β , then we consider a simple single-state two-arm contextual bandit (equivalently multi-armed bandit) instance:
Revised: If there exists − 12 < β such that α ≥ N β , then we consider a simple single-state two-arm contextual bandit (equivalently multi-armed bandit) instance:

Original: • Reward distribution : Both arms have deterministic rewards and the suboptimal arm has a value gap of α :
Revised: • Reward distribution : Both arms have deterministic rewards and the suboptimal arm has a value gap of α :

Original: [Equation]
Revised: [Equation]

Original: • Data distribution : We construct the data distribution such that both arms have constant probability density, which implies a constant concentrability ratio C ⋆ .
Revised: • Data distribution : We construct the data distribution such that both arms have constant probability density, which implies a constant concentrability ratio C ⋆ .

Original: Here we assume M f < 100 for convenience, but if M f is larger we can use the same construction with an even larger constant as the denominator.
Revised: Here we assume M f < 100 for convenience, but if M f is larger we can use the same construction with an even larger constant as the denominator.

Original: [Equation]
Revised: [Equation]

Original: • Function classes :
Revised: • Function classes :

Original: We assume both W and V contain only the optimal regularized solutions ( w ⋆α , v ⋆α ) and the optimal unregularized solutions ( w ⋆ , v ⋆ ) , which satisfy the realizability requirements of PRO-CB:
Revised: We assume both W and V contain only the optimal regularized solutions ( w ⋆α , v ⋆α ) and the optimal unregularized solutions ( w ⋆ , v ⋆ ) , which satisfy the realizability requirements of PRO-CB:

Original: [Equation]
Revised: [Equation]

Original: Our argument is broken down in two steps.
Revised: Our argument is broken down in two steps.

Original: In the first step, we show that the suboptimality of the optimal regularized policy, which is the policy induced by the regularized optimal weights π ⋆α :=
Revised: In the first step, we show that the suboptimality of the optimal regularized policy, which is the policy induced by the regularized optimal weights π ⋆α :=

Original: π w ⋆α , is at least of order min { 1 , α } .
Revised: π w ⋆α , is at least of order min { 1 , α } .

Original: Then, in the second step, we prove that w ⋆α is chosen with a constant probability.
Revised: Then, in the second step, we prove that w ⋆α is chosen with a constant probability.

Original: Step 1: Suboptimality of π ⋆α .
Revised: Step 1: Suboptimality of π ⋆α .

Original: In the particular offline bandit instance above, we show the following lower bound on suboptimality of π ⋆α
Revised: In the particular offline bandit instance above, we show the following lower bound on suboptimality of π ⋆α

Original: [Equation]
Revised: [Equation]

Original: To establish (26), we show that w ⋆α (2) >
Revised: To establish (26), we show that w ⋆α (2) >

Original: c for a fixed constant c = 12 .
Revised: c for a fixed constant c = 12 .

Original: We prove this by contradiction.
Revised: We prove this by contradiction.

Original: Suppose
Revised: Suppose

Original: [Equation]
Revised: [Equation]

Original: By KKT conditions we have
Revised: By KKT conditions we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, using the fact that f ′ is strictly increasing since f is strictly convex, we lower bound v ⋆α according to
Revised: Therefore, using the fact that f ′ is strictly increasing since f is strictly convex, we lower bound v ⋆α according to

Original: [Equation]
Revised: [Equation]

Original: Combining the above bound on v ⋆α with the KKT condition on w ⋆α (1) , we then obtain
Revised: Combining the above bound on v ⋆α with the KKT condition on w ⋆α (1) , we then obtain

Original: [Equation]
Revised: [Equation]

Original: Here, we used the fact that v ⋆α ≥ r ⋆ = r (1) and that f (0) = 

Revised: Here, we used the fact that v ⋆α ≥ r ⋆ = r (1) and that f (0) =

Original: so ( f ′ ) − 1 (( r (1) − v ⋆α ) /α ) ≥ 0 .
Revised: so ( f ′ ) − 1 (( r (1) − v ⋆α ) /α ) ≥ 0 .

Original: Moreover, since the regularization function f is M f -strongly convex, we write
Revised: Moreover, since the regularization function f is M f -strongly convex, we write

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can continue to upper bound the RHS of (28):
Revised: Therefore, we can continue to upper bound the RHS of (28):

Original: [Equation]
Revised: [Equation]

Original: which further implies that
Revised: which further implies that

Original: [Equation]
Revised: [Equation]

Original: Note that (30) contradicts with the fact that ( w ⋆α , v ⋆α ) is the optimal min-max solution of L MAB α because it violates the constraint E µ [ w ( a )] = 1 .
Revised: Note that (30) contradicts with the fact that ( w ⋆α , v ⋆α ) is the optimal min-max solution of L MAB α because it violates the constraint E µ [ w ( a )] = 1 .

Original: Therefore, (27) should not hold in the first place, and we must have
Revised: Therefore, (27) should not hold in the first place, and we must have

Original: [Equation]
Revised: [Equation]

Original: Step 2: w ⋆α is picked with large probability.
Revised: Step 2: w ⋆α is picked with large probability.

Original: We now show that w ⋆α is picked by the algorithm with at least a constant probability.
Revised: We now show that w ⋆α is picked by the algorithm with at least a constant probability.

Original: Note that since w ⋆α and w ⋆ both satisfy the constraint E µ [ w ] − 1 = 0 , objectives L MAB α ( w ⋆α , v ) and L MAB α ( w α , v ) do not depend on the Lagrange multiplier variable v .
Revised: Note that since w ⋆α and w ⋆ both satisfy the constraint E µ [ w ] − 1 = 0 , objectives L MAB α ( w ⋆α , v ) and L MAB α ( w α , v ) do not depend on the Lagrange multiplier variable v .

Original: We argue that at the population level, we have the following lower bound on the gap L MAB α ( w ⋆α , v ) −
Revised: We argue that at the population level, we have the following lower bound on the gap L MAB α ( w ⋆α , v ) −

Original: L MAB α ( w ⋆ , v ) ≳
Revised: L MAB α ( w ⋆ , v ) ≳

Original: α .
Revised: α .

Original: Using the definition of L MAB α , one has
Revised: Using the definition of L MAB α , one has

Original: L MAB α
Revised: L MAB α

Original: [Equation]
Revised: [Equation]

Original: In (32), we used the convexity of regularization function f as well as the fact that f ( w ⋆ (2)) =
Revised: In (32), we used the convexity of regularization function f as well as the fact that f ( w ⋆ (2)) =

Original: f (0) = 0 .
Revised: f (0) = 0 .

Original: Moreover, (33) holds because
Revised: Moreover, (33) holds because

Original: [Equation]
Revised: [Equation]

Original: By KKT conditions we also have
Revised: By KKT conditions we also have

Original: [Equation]
Revised: [Equation]

Original: Plugging (34) back into (33), we obtain
Revised: Plugging (34) back into (33), we obtain

Original: [Equation]
Revised: [Equation]

Original: where (35) is based on the fact that f is M f -strongly convex
Revised: where (35) is based on the fact that f is M f -strongly convex

Original: [Equation]
Revised: [Equation]

Original: and that w ⋆α (2) >
Revised: and that w ⋆α (2) >

Original: c proved in Step 1.
Revised: c proved in Step 1.

Original: We now prove that such large lower bound on population objective difference leads the algorithm to select w ⋆α .
Revised: We now prove that such large lower bound on population objective difference leads the algorithm to select w ⋆α .

Original: Recall from Lemma 3 that with at least constant probability (e
Revised: Recall from Lemma 3 that with at least constant probability (e

Original: [Equation]
Revised: [Equation]

Original: g. setting δ = 0 . 1 ), for any v ∈ V , w ∈ W , one has the following bound on difference between the population and empirical objectives
Revised: g. setting δ = 0 . 1 ), for any v ∈ V , w ∈ W , one has the following bound on difference between the population and empirical objectives

Original: [Equation]
Revised: [Equation]

Original: where ϵ MABstat ,α is of order 1 / √ N as defined in (18).
Revised: where ϵ MABstat ,α is of order 1 / √ N as defined in (18).

Original: Combining the above inequality with (35), for any v, v ′ ∈ V we have
Revised: Combining the above inequality with (35), for any v, v ′ ∈ V we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, since β > − 1 / 2 , we conclude that w ⋆α is chosen by the algorithm with constant probability:
Revised: Therefore, since β > − 1 / 2 , we conclude that w ⋆α is chosen by the algorithm with constant probability:

Original: [Equation]
Revised: [Equation]

Original: Combining the above result with the suboptimality lower bound of π ⋆α in (31) completes the proof for α ≥ N β .
Revised: Combining the above result with the suboptimality lower bound of π ⋆α in (31) completes the proof for α ≥ N β .

Original: D.2. P ROOF FOR SMALL α
Revised: D.2. P ROOF FOR SMALL α

Original: Now suppose α ≤ (cid:101) O
Revised: Now suppose α ≤ (cid:101) O

Original: ( N − 1 2 ) , where (cid:101) O hides the logarithmic factors.
Revised: ( N − 1 2 ) , where (cid:101) O hides the logarithmic factors.

Original: In this case
Revised: In this case

Original: [Equation]
Revised: [Equation]

Original: we consider the following two-state two-arm contextual bandit instance:
Revised: we consider the following two-state two-arm contextual bandit instance:

Original: • State and reward distributions : We construct the states such that state 1 has a very small probability mass.
Revised: • State and reward distributions : We construct the states such that state 1 has a very small probability mass.

Original: For state 1, the first arm is optimal with a Bernoulli-distributed reward and the second arm is suboptimal with a deterministic reward.
Revised: For state 1, the first arm is optimal with a Bernoulli-distributed reward and the second arm is suboptimal with a deterministic reward.

Original: For state 2, both arms have deterministic rewards.
Revised: For state 2, both arms have deterministic rewards.

Original: Importantly, state 1 has a constant value gap in its suboptimal action.
Revised: Importantly, state 1 has a constant value gap in its suboptimal action.

Original: [Equation]
Revised: [Equation]

Original: • Data distribution : We assume that for both states, most of the probability density is concentrated on the optimal arm.
Revised: • Data distribution : We assume that for both states, most of the probability density is concentrated on the optimal arm.

Original: [Equation]
Revised: [Equation]

Original: Function
Revised: Function

Original: classes :
Revised: classes :

Original: Let w
Revised: Let w

Original: be defined as ˜ w (2 , a )
Revised: be defined as ˜ w (2 , a )

Original: =
Revised: =

Original: w
Revised: w

Original: ⋆α ( a )
Revised: ⋆α ( a )

Original: and ˜
Revised: and ˜

Original: w (
Revised: w (

Original: , a ) =
Revised: , a ) =

Original: 0 for
Revised: 0 for

Original: a = 1 ,
Revised: a = 1 ,

Original: Consider the following function classes W and V :
Revised: Consider the following function classes W and V :

Original: [Equation]
Revised: [Equation]

Original: The proof is broken down into 4 steps.
Revised: The proof is broken down into 4 steps.

Original: In the first step, we show that when α ≤ (cid:101) O ( N − 12 ) and N is sufficiently large, the regularized optimal policy is the same as the unregularized optimal policy, i.e., w ⋆α = w ⋆ .
Revised: In the first step, we show that when α ≤ (cid:101) O ( N − 12 ) and N is sufficiently large, the regularized optimal policy is the same as the unregularized optimal policy, i.e., w ⋆α = w ⋆ .

Original: Therefore, the function class W defined in (36) is realizable w ⋆α = w ⋆ ∈ W .
Revised: Therefore, the function class W defined in (36) is realizable w ⋆α = w ⋆ ∈ W .

Original: In the second step, we prove that with constant probability v ⋆α = argmin v ∈V ˆ L CB α
Revised: In the second step, we prove that with constant probability v ⋆α = argmin v ∈V ˆ L CB α

Original: ( ˜ w, v ) .
Revised: ( ˜ w, v ) .

Original: Then, we show that solving the saddle point of the empirical objective ˆ L CB α ( w, v ) selects ˜ w over w ⋆α with a constant probability.
Revised: Then, we show that solving the saddle point of the empirical objective ˆ L CB α ( w, v ) selects ˜ w over w ⋆α with a constant probability.

Original: Finally, we prove that ˜ w induces a policy π ˜ w that suffers from suboptimality of order
Revised: Finally, we prove that ˜ w induces a policy π ˜ w that suffers from suboptimality of order

Original: N − 1 / 4 , which completes the proof.
Revised: N − 1 / 4 , which completes the proof.

Original: Step 1: Regularized optimal weights coincides with unregularized optimal weights.
Revised: Step 1: Regularized optimal weights coincides with unregularized optimal weights.

Original: Since the population optimization problem (24) is independent across states at a population level, we can use the result of Lemma 1 to conclude that v ⋆α ( s ) =
Revised: Since the population optimization problem (24) is independent across states at a population level, we can use the result of Lemma 1 to conclude that v ⋆α ( s ) =

Original: r ⋆ ( s ) − c ( s ) α, and
Revised: r ⋆ ( s ) − c ( s ) α, and

Original: [Equation]
Revised: [Equation]

Original: where 0 ≤ c ( s ) ≤ f ′ ( C ⋆ ) for s ∈ { 1 , 2 } .
Revised: where 0 ≤ c ( s ) ≤ f ′ ( C ⋆ ) for s ∈ { 1 , 2 } .

Original: Since r ⋆ ( s ) − r ( s, 2) =
Revised: Since r ⋆ ( s ) − r ( s, 2) =

Original: 16 = Θ(1) , for N ≥ (6 f ′ ( C ⋆ )) 2 , we have w ⋆α ( s, 2) = 

Revised: 16 = Θ(1) , for N ≥ (6 f ′ ( C ⋆ )) 2 , we have w ⋆α ( s, 2) =

Original: for the suboptimal arm 2 .
Revised: for the suboptimal arm 2 .

Original: Thus w ⋆α ( s ) = w ⋆ ( s ) = 1 µ (1 | s ) .
Revised: Thus w ⋆α ( s ) = w ⋆ ( s ) = 1 µ (1 | s ) .

Original: Correspondingly, we can use the KKT conditions to compute v ⋆α ( s ) =
Revised: Correspondingly, we can use the KKT conditions to compute v ⋆α ( s ) =

Original: r ⋆ ( s ) − αf ′ (cid:16) 1 µ (1 | s ) (cid:17) .
Revised: r ⋆ ( s ) − αf ′ (cid:16) 1 µ (1 | s ) (cid:17) .

Original: Step 2: v ⋆α = argmin v ∈V ˆ L CB α
Revised: Step 2: v ⋆α = argmin v ∈V ˆ L CB α

Original: ( ˜ w, v ) with constant probability.
Revised: ( ˜ w, v ) with constant probability.

Original: Let ˆ µ denote the empirical statearm distribution and ˆ r denote the empirical mean reward.
Revised: Let ˆ µ denote the empirical statearm distribution and ˆ r denote the empirical mean reward.

Original: Define the following event:
Revised: Define the following event:

Original: [Equation]
Revised: [Equation]

Original: Recall that we defined ˜ w (1 , a ) = 0 and ˜ w (2 , a ) =
Revised: Recall that we defined ˜ w (1 , a ) = 0 and ˜ w (2 , a ) =

Original: w ⋆ (2 , a ) .
Revised: w ⋆ (2 , a ) .

Original: Thus, the above event can be equivalently written as
Revised: Thus, the above event can be equivalently written as

Original: [Equation]
Revised: [Equation]

Original: Here we used the fact that (cid:80) a µ ( a | 2) w ⋆α ( s, 2) = 1 .
Revised: Here we used the fact that (cid:80) a µ ( a | 2) w ⋆α ( s, 2) = 1 .

Original: Moreover, in Step 1 we showed that w ⋆α = w ⋆ , thus w ⋆α (2 , 2) =
Revised: Moreover, in Step 1 we showed that w ⋆α = w ⋆ , thus w ⋆α (2 , 2) =

Original: and (38) corresponds to the following event
Revised: and (38) corresponds to the following event

Original: [Equation]
Revised: [Equation]

Original: Since ˆ µ (1 | 2) is an empirical version of the conditional probability µ (1 | 2) , event E happens with probability 12 .
Revised: Since ˆ µ (1 | 2) is an empirical version of the conditional probability µ (1 | 2) , event E happens with probability 12 .

Original: We condition on the event E for the rest of the proof.
Revised: We condition on the event E for the rest of the proof.

Original: Using the fact that v ⋆α ( s ) ≤ r ⋆ ( s ) = v ⋆ ( s ) , we conclude that
Revised: Using the fact that v ⋆α ( s ) ≤ r ⋆ ( s ) = v ⋆ ( s ) , we conclude that

Original: [Equation]
Revised: [Equation]

Original: Step 3:
Revised: Step 3:

Original: Analyzing the probability of picking w ⋆ α .
Revised: Analyzing the probability of picking w ⋆ α .

Original: Now we compare the value of ˆ L CB α ( · , v ⋆ α ) evaluated at ˜ w and w ⋆α .
Revised: Now we compare the value of ˆ L CB α ( · , v ⋆ α ) evaluated at ˜ w and w ⋆α .

Original: We use the definition ˜ w (2 , a ) =
Revised: We use the definition ˜ w (2 , a ) =

Original: w ⋆α (2 , a ) and write ˆ L CB α ( w ⋆α , v ⋆α ) − ˆ L CB α
Revised: w ⋆α (2 , a ) and write ˆ L CB α ( w ⋆α , v ⋆α ) − ˆ L CB α

Original: ( ˜ w, v ⋆α )
Revised: ( ˜ w, v ⋆α )

Original: =
Revised: =

Original: ˆ µ (
Revised: ˆ µ (

Original: [Equation]
Revised: [Equation]

Original: Noting that v ⋆α (1) =
Revised: Noting that v ⋆α (1) =

Original: r (1 , 1) − αf ′ ( w ⋆α ( s 1 , a 1 )) , ˜ w (1 , a ) = 0 , and w ⋆α (1 , 1) = w ⋆ (1 , 1) =
Revised: r (1 , 1) − αf ′ ( w ⋆α ( s 1 , a 1 )) , ˜ w (1 , a ) = 0 , and w ⋆α (1 , 1) = w ⋆ (1 , 1) =

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: , we further simplify the above equation
Revised: , we further simplify the above equation

Original: [Equation]
Revised: [Equation]

Original: We then prove that with constant probability, the first term in (42) is negative with a magnitude larger than the second term:
Revised: We then prove that with constant probability, the first term in (42) is negative with a magnitude larger than the second term:

Original: [Equation]
Revised: [Equation]

Original: The proof of this inequality relies on anti-concentration bounds of binomial random variables and is presented at the end of this section.
Revised: The proof of this inequality relies on anti-concentration bounds of binomial random variables and is presented at the end of this section.

Original: By Inequality (43) combined with (40), we conclude that
Revised: By Inequality (43) combined with (40), we conclude that

Original: [Equation]
Revised: [Equation]

Original: which guarantees that the algorithm picks ˜ w with a constant probability.
Revised: which guarantees that the algorithm picks ˜ w with a constant probability.

Original: Step 4: Suboptimality of π w
Revised: Step 4: Suboptimality of π w

Original: Finally, for the policy π w induced by w , we have
Revised: Finally, for the policy π w induced by w , we have

Original: [Equation]
Revised: [Equation]

Original: for β = − 14 > − 12 , as desired.
Revised: for β = − 14 > − 12 , as desired.

Original: The proof for small α is thus complete.
Revised: The proof for small α is thus complete.

Original: Proof of Inequality (43) .
Revised: Proof of Inequality (43) .

Original: Using the Chernoff bounds for binomial random variables given in Lemma 2 (adapted from Proposition 7.3.2 of Matoušek & Vondrák (2001)), one can conclude that the following event E ′ happens with probability at least 0.5:
Revised: Using the Chernoff bounds for binomial random variables given in Lemma 2 (adapted from Proposition 7.3.2 of Matoušek & Vondrák (2001)), one can conclude that the following event E ′ happens with probability at least 0.5:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, E and E ′ are independent because the random variable
Revised: Furthermore, E and E ′ are independent because the random variable

Original: ˆ r
Revised: ˆ r

Original: ( s 1 , a 1 ) is independent from the arm distribution within state s 2 .
Revised: ( s 1 , a 1 ) is independent from the arm distribution within state s 2 .

Original: Therefore, conditioning on E ∩E ′ which happens with probability 0 .
Revised: Therefore, conditioning on E ∩E ′ which happens with probability 0 .

Original: 5 × 0 .
Revised: 5 × 0 .

Original: 5 = 0 .
Revised: 5 = 0 .

Original: 25 , we use the anti-concentration bounds for Binomial random variables Lemma 5 to obtain the following lower bound:
Revised: 25 , we use the anti-concentration bounds for Binomial random variables Lemma 5 to obtain the following lower bound:

Original: [Equation]
Revised: [Equation]

Original: where c ′ =
Revised: where c ′ =

Original: (cid:113) 20log(

Revised: (cid:113) 20log(

Original: c 1 ) cis a universal constant.
Revised: c 1 ) cis a universal constant.

Original: Therefore, we have established that (43) holds with constant probability.
Revised: Therefore, we have established that (43) holds with constant probability.

Original: Lemma 5 (Anti-concentration of Binomial random variables) Let X 1 , · · · , X n be independent random variables following the Bernoulli distribution with mean 12 , and let X = 1 n (cid:80) ni
Revised: Lemma 5 (Anti-concentration of Binomial random variables) Let X 1 , · · · , X n be independent random variables following the Bernoulli distribution with mean 12 , and let X = 1 n (cid:80) ni

Original: =1 X
Revised: =1 X

Original: i be the empirical mean.
Revised: i be the empirical mean.

Original: Then we have that for any t ∈ [0 , 18 ] and universal constants c 1 , c 2 ,
Revised: Then we have that for any t ∈ [0 , 18 ] and universal constants c 1 , c 2 ,

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF T HEOREM
Revised: D. P ROOF OF T HEOREM

Original: Proof of this theorem largely follows similar steps as the proof we presented for Theorem 1.
Revised: Proof of this theorem largely follows similar steps as the proof we presented for Theorem 1.

Original: In particular, we start by presenting two lemmas.
Revised: In particular, we start by presenting two lemmas.

Original: The first lemma leverages Hoeffding’s inequality to establish the closeness of the population objective (7) and empirical objective (8).
Revised: The first lemma leverages Hoeffding’s inequality to establish the closeness of the population objective (7) and empirical objective (8).

Original: Additionally, we show that this result leads to the closeness of population objective at w ⋆ and ˆ w .
Revised: Additionally, we show that this result leads to the closeness of population objective at w ⋆ and ˆ w .

Original: Proof of this lemma is presented at the end of this subsection.
Revised: Proof of this lemma is presented at the end of this subsection.

Original: Lemma 6 (Empirical and population closeness, CB)
Revised: Lemma 6 (Empirical and population closeness, CB)

Original: Fix δ > 0 and define
Revised: Fix δ > 0 and define

Original: [Equation]
Revised: [Equation]

Original: For any w ∈ W and v ∈ V , the following statements hold with probability at least 1 − δ
Revised: For any w ∈ W and v ∈ V , the following statements hold with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: In the second lemma, we prove that the ALM term enforces a lower bound on normalization factors d ˆ w
Revised: In the second lemma, we prove that the ALM term enforces a lower bound on normalization factors d ˆ w

Original: [Equation]
Revised: [Equation]

Original: significant states.
Revised: significant states.

Original: Lemma 7 (Occupancy validity enforced by the ALM)
Revised: Lemma 7 (Occupancy validity enforced by the ALM)

Original: Define the state space subset
Revised: Define the state space subset

Original: [Equation]
Revised: [Equation]

Original: For any fixed δ > 0 , the following statements hold with probability at least 1 − δ ,
Revised: For any fixed δ > 0 , the following statements hold with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where ϵ CBstat is defined in (48) .
Revised: where ϵ CBstat is defined in (48) .

Original: Given the two lemmas above, our suboptimality analysis can be broken down into two simple steps.
Revised: Given the two lemmas above, our suboptimality analysis can be broken down into two simple steps.

Original: First, we partition the states based on S s defined in (49) and decompose the policy suboptimality accordingly:
Revised: First, we partition the states based on S s defined in (49) and decompose the policy suboptimality accordingly:

Original: [Equation]
Revised: [Equation]

Original: In (50)
Revised: In (50)

Original: [Equation]
Revised: [Equation]

Original: we used part (II) in Lemma 7 to bound the first term and (51) uses the fact that by definition, for all s ̸∈ S s we have µ ( s ) <
Revised: we used part (II) in Lemma 7 to bound the first term and (51) uses the fact that by definition, for all s ̸∈ S s we have µ ( s ) <

Original: 2 ˆ d ( s ) and V ⋆ ( s ) − V ˆ π ( s ) ≥ 0 .
Revised: 2 ˆ d ( s ) and V ⋆ ( s ) − V ˆ π ( s ) ≥ 0 .

Original: Moreover, the second term in (51) is bounded by part (I) of Lemma 7 since
Revised: Moreover, the second term in (51) is bounded by part (I) of Lemma 7 since

Original: [Equation]
Revised: [Equation]

Original: where the equations follow from the definition of ˆ π .
Revised: where the equations follow from the definition of ˆ π .

Original: The final suboptimality bound is proved by noting that ( B w + 1) 2 ≍ B 2 w since B w ≥ 1 due to realizability of w ⋆ ( s, a ⋆ ) ≥ 1 .
Revised: The final suboptimality bound is proved by noting that ( B w + 1) 2 ≍ B 2 w since B w ≥ 1 due to realizability of w ⋆ ( s, a ⋆ ) ≥ 1 .

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: To prove part (I), notice that E µ (cid:104)
Revised: To prove part (I), notice that E µ (cid:104)

Original: ˆ L CBAL ( w, v ) (cid:105) =
Revised: ˆ L CBAL ( w, v ) (cid:105) =

Original: L CBAL ( w, v ) .
Revised: L CBAL ( w, v ) .

Original: Furthermore, ˆ L CBAL ( w, v ) is an empirical average of i.i.d.
Revised: Furthermore, ˆ L CBAL ( w, v ) is an empirical average of i.i.d.

Original: random variables which are bounded by
Revised: random variables which are bounded by

Original: [Equation]
Revised: [Equation]

Original: Applying Hoeffding’s inequality along with a union bound on w ∈ W and v ∈ V finishes the proof of part (I).
Revised: Applying Hoeffding’s inequality along with a union bound on w ∈ W and v ∈ V finishes the proof of part (I).

Original: We now prove part (II).
Revised: We now prove part (II).

Original: For the primal-dual objective without the AL term
Revised: For the primal-dual objective without the AL term

Original: [Equation]
Revised: [Equation]

Original: we have ( w ⋆ , v ⋆ ) ∈ argmax w ≥ 0 argmin v L CB ( w, v ) by strong duality.
Revised: we have ( w ⋆ , v ⋆ ) ∈ argmax w ≥ 0 argmin v L CB ( w, v ) by strong duality.

Original: Moreover, since w ⋆ is realizable, it satisfies the validity constraint E a ∼ µ ( ·| s ) [ w ⋆ ( s, a )]
Revised: Moreover, since w ⋆ is realizable, it satisfies the validity constraint E a ∼ µ ( ·| s ) [ w ⋆ ( s, a )]

Original: = 1 for all s .
Revised: = 1 for all s .

Original: Therefore, by
Revised: Therefore, by

Original: Lemma 14 adding the ALM term does not change the optimal solution and we have ( w ⋆ , v ⋆ ) ∈ argmax w ≥ 0 argmin v L CBAL ( w, v ) .
Revised: Lemma 14 adding the ALM term does not change the optimal solution and we have ( w ⋆ , v ⋆ ) ∈ argmax w ≥ 0 argmin v L CBAL ( w, v ) .

Original: We follow similar steps as in the proof of Lemma 1 and decompose L CBAL ( w ⋆ , v ) − L CBAL ( ˆ w, v ) according to
Revised: We follow similar steps as in the proof of Lemma 1 and decompose L CBAL ( w ⋆ , v ) − L CBAL ( ˆ w, v ) according to

Original: [Equation]
Revised: [Equation]

Original: CBAL
Revised: CBAL

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˆ v w = arg min v ∈V ˆ L CBAL ( w, v ) .
Revised: where ˆ v w = arg min v ∈V ˆ L CBAL ( w, v ) .

Original: Each term is bounded as follows:
Revised: Each term is bounded as follows:

Original: • T 1 = 0 because w ⋆ satisfies the optimization constraints.
Revised: • T 1 = 0 because w ⋆ satisfies the optimization constraints.

Original: • T 2 ≤ ϵ CBstat due to Lemma 6.
Revised: • T 2 ≤ ϵ CBstat due to Lemma 6.

Original: • T 3 ≤ 0 because ˆ w = arg max w ∈W ˆ L CBAL (ˆ v w , w ) .
Revised: • T 3 ≤ 0 because ˆ w = arg max w ∈W ˆ L CBAL (ˆ v w , w ) .

Original: • T 4 ≤ 0 because ˆ v = arg min v ∈V ˆ L CBAL ( v, ˆ w ) .
Revised: • T 4 ≤ 0 because ˆ v = arg min v ∈V ˆ L CBAL ( v, ˆ w ) .

Original: • T 5 ≤ ϵ CBstat due to Lemma 6.
Revised: • T 5 ≤ ϵ CBstat due to Lemma 6.

Original: Summing up the bounds on each term proves part (II).
Revised: Summing up the bounds on each term proves part (II).

Original: □
Revised: □

Original: Proof of Lemma 7.
Revised: Proof of Lemma 7.

Original: We leverage the closeness of the objective at w ⋆ and ˆ w established in Lemmato show that the ALM term at ˆ w is small.
Revised: We leverage the closeness of the objective at w ⋆ and ˆ w established in Lemmato show that the ALM term at ˆ w is small.

Original: Since w ⋆ satisfies the validity constraints, the objective at w ⋆ simplifies to
Revised: Since w ⋆ satisfies the validity constraints, the objective at w ⋆ simplifies to

Original: [Equation]
Revised: [Equation]

Original: w ⋆ , v ) =
Revised: w ⋆ , v ) =

Original: E s,a ∼ µ [ r ( s, a ) w ⋆ ( s, a )] + E s,a ∼ µ [ v ( s )(1 − w ⋆ ( s, a ))] (cid:124) (cid:123)(cid:122) (cid:125) =
Revised: E s,a ∼ µ [ r ( s, a ) w ⋆ ( s, a )] + E s,a ∼ µ [ v ( s )(1 − w ⋆ ( s, a ))] (cid:124) (cid:123)(cid:122) (cid:125) =

Original: − E s ∼ µ [( E a ∼ µ ( ·| s )
Revised: − E s ∼ µ [( E a ∼ µ ( ·| s )

Original: [ w ( s, a )] − 1) 2 ] (cid:124) (cid:123)(cid:122) (cid:125) =
Revised: [ w ( s, a )] − 1) 2 ] (cid:124) (cid:123)(cid:122) (cid:125) =

Original: =
Revised: =

Original: E s,a ∼ µ [ r ( s, a ) w ⋆ ( s, a )] .
Revised: E s,a ∼ µ [ r ( s, a ) w ⋆ ( s, a )] .

Original: Consider the objective difference at v ( s ) =
Revised: Consider the objective difference at v ( s ) =

Original: r ⋆ ( s ) :=
Revised: r ⋆ ( s ) :=

Original: [Equation]
Revised: [Equation]

Original: a r ( s, a ) :
Revised: a r ( s, a ) :

Original: [Equation]
Revised: [Equation]

Original: Since L CBAL ( w ⋆ , v ) − L CBAL ( ˆ w, v ) ≲ ϵ CBstat by Lemma 6, we conclude that
Revised: Since L CBAL ( w ⋆ , v ) − L CBAL ( ˆ w, v ) ≲ ϵ CBstat by Lemma 6, we conclude that

Original: [Equation]
Revised: [Equation]

Original: Moreover, since the first term is nonnegative due to ˆ w ( s, a ) ≥ 0 and r ⋆ ( s ) − r ( s, a ) ≥ 0 , both of the terms in the above inequality are bounded by ϵ CBstat and thereby proving part (I).
Revised: Moreover, since the first term is nonnegative due to ˆ w ( s, a ) ≥ 0 and r ⋆ ( s ) − r ( s, a ) ≥ 0 , both of the terms in the above inequality are bounded by ϵ CBstat and thereby proving part (I).

Original: The above result also allows us to bound the mass on the subset S s that contains the states that violate state occupancy validity
Revised: The above result also allows us to bound the mass on the subset S s that contains the states that violate state occupancy validity

Original: [Equation]
Revised: [Equation]

Original: d ˆ w µ ( s ) −(cid:19) 2 (cid:35) ≥ (cid:88) s ∈S s µ ( s ) (cid:34)(cid:18) d ˆ w µ ( s ) −(cid:19) 2 (cid:35) ≥ (cid:88) s ∈S s µ ( s ) ≳
Revised: d ˆ w µ ( s ) −(cid:19) 2 (cid:35) ≥ (cid:88) s ∈S s µ ( s ) (cid:34)(cid:18) d ˆ w µ ( s ) −(cid:19) 2 (cid:35) ≥ (cid:88) s ∈S s µ ( s ) ≳

Original: (cid:88) s ∈S s µ ( s ) , where we used the fact that d ˆ w µ ( s ) ≤ 12 and thus (cid:16) d
Revised: (cid:88) s ∈S s µ ( s ) , where we used the fact that d ˆ w µ ( s ) ≤ 12 and thus (cid:16) d

Original: ˆ w µ
Revised: ˆ w µ

Original: ( s ) − 1 (cid:17) 2 ≥ 14 by definition of S s .
Revised: ( s ) − 1 (cid:17) 2 ≥ 14 by definition of S s .

Original: This concludes the proof of part (II).
Revised: This concludes the proof of part (II).

Original: □
Revised: □

Original: E P ROOFS FOR MDP S
Revised: E P ROOFS FOR MDP S

Original: In this section, we begin by introducing some additional notation.
Revised: In this section, we begin by introducing some additional notation.

Original: The original primal-dual objective without ALM term is given by
Revised: The original primal-dual objective without ALM term is given by

Original: [Equation]
Revised: [Equation]

Original: Define w ⋆ ( s, a ) =
Revised: Define w ⋆ ( s, a ) =

Original: d π ⋆ ( s, a ) /µ
Revised: d π ⋆ ( s, a ) /µ

Original: ( s, a ) and v ⋆ ( s ) =
Revised: ( s, a ) and v ⋆ ( s ) =

Original: V ∗ ( s ) .
Revised: V ∗ ( s ) .

Original: By strong duality, one has ( w ⋆ , v ⋆ ) ∈ arg max w ≥ 0 arg min v L MDP ( w, v ) .
Revised: By strong duality, one has ( w ⋆ , v ⋆ ) ∈ arg max w ≥ 0 arg min v L MDP ( w, v ) .

Original: Additionally, define ζ ⋆w,u = arg max ζ< 

Revised: Additionally, define ζ ⋆w,u = arg max ζ<

Original: L model-freeAL ( w, v, u, ζ ) , ∀ w ∈ W , u ∈ U and ζ ⋆w = ζ ⋆w,u ⋆w ∀ w ∈ W where u ⋆w is defined in Theorem 4.
Revised: L model-freeAL ( w, v, u, ζ ) , ∀ w ∈ W , u ∈ U and ζ ⋆w = ζ ⋆w,u ⋆w ∀ w ∈ W where u ⋆w is defined in Theorem 4.

Original: Also, denote ζ ⋆ = ζ ⋆w ⋆ and u ⋆ = u ⋆w ⋆ .
Revised: Also, denote ζ ⋆ = ζ ⋆w ⋆ and u ⋆ = u ⋆w ⋆ .

Original: The rest of this section is organized as follows.
Revised: The rest of this section is organized as follows.

Original: In Appendix E.1, we provide some details regarding practical implementation of the offline learning algorithm with ALM.
Revised: In Appendix E.1, we provide some details regarding practical implementation of the offline learning algorithm with ALM.

Original: In Appendix E.2, we derive the objective of model-free ALMIS algorithm.
Revised: In Appendix E.2, we derive the objective of model-free ALMIS algorithm.

Original: Appendix E.3 contains the proof of performance upper bound on model-based and model-free ALMIS algorithms (Theorem 4), which relies on several lemmas subsequently proved in Appendices E.4 through E.7.
Revised: Appendix E.3 contains the proof of performance upper bound on model-based and model-free ALMIS algorithms (Theorem 4), which relies on several lemmas subsequently proved in Appendices E.4 through E.7.

Original: E. O N PRACTICAL IMPLEMENTATIONS
Revised: E. O N PRACTICAL IMPLEMENTATIONS

Original: In our algorithms for CB and MDP, we need to compute summations of form (cid:80) a ∈A .
Revised: In our algorithms for CB and MDP, we need to compute summations of form (cid:80) a ∈A .

Original: This can be implemented efficiently when |A| is small.
Revised: This can be implemented efficiently when |A| is small.

Original: When |A| is large or even infinite, one can utilize numerical methods to estimate the summation with desired precision.
Revised: When |A| is large or even infinite, one can utilize numerical methods to estimate the summation with desired precision.

Original: Additionally, in Algorithm 3, we need to evaluate a term (cid:80) s ′ ,a ′ P ( s ′ | s, a ) π w ( a ′ | s ′ )
Revised: Additionally, in Algorithm 3, we need to evaluate a term (cid:80) s ′ ,a ′ P ( s ′ | s, a ) π w ( a ′ | s ′ )

Original: u
Revised: u

Original: ( s ′ , a ′ ) .
Revised: ( s ′ , a ′ ) .

Original: In practice, we can evaluate this term by numerical integration.
Revised: In practice, we can evaluate this term by numerical integration.

Original: E. D ERIVATION OF THE MODEL - FREE ALMIS OBJECTIVE
Revised: E. D ERIVATION OF THE MODEL - FREE ALMIS OBJECTIVE

Original: For f ( x ) =
Revised: For f ( x ) =

Original: ( x − 1) 2 , the Fenchel conjugate f ∗ is given by
Revised: ( x − 1) 2 , the Fenchel conjugate f ∗ is given by

Original: [Equation]
Revised: [Equation]

Original: Since d w ( s ) /d π w ( s ) ≥ 0 , we have x ⋆w ( s, a ) ≥ − 2 and thus it is sufficient to only consider domain x ( s, a ) ≥ − 2 , over which f ∗ ( x ) is invertible.
Revised: Since d w ( s ) /d π w ( s ) ≥ 0 , we have x ⋆w ( s, a ) ≥ − 2 and thus it is sufficient to only consider domain x ( s, a ) ≥ − 2 , over which f ∗ ( x ) is invertible.

Original: Let g ( x ) =
Revised: Let g ( x ) =

Original: − f − 1 ∗ ( x ) = 2 − 2 √ x + 1 , which is a convex function on [ − 1 , + ∞ ) .
Revised: − f − 1 ∗ ( x ) = 2 − 2 √ x + 1 , which is a convex function on [ − 1 , + ∞ ) .

Original: Similar to Nachum et al.
Revised: Similar to Nachum et al.

Original: (2019a)
Revised: (2019a)

Original: , we use Fenchel duality to estimate g ( u ( s, a ) − γ P π w u ( s, a )) .
Revised: , we use Fenchel duality to estimate g ( u ( s, a ) − γ P π w u ( s, a )) .

Original: By Fenchel duality, any convex function g ( x ) can be written as g ( x )
Revised: By Fenchel duality, any convex function g ( x ) can be written as g ( x )

Original: = max ζ xζ − g ∗ ( ζ ) .
Revised: = max ζ xζ − g ∗ ( ζ ) .

Original: In the case of g ( x ) , the
Revised: In the case of g ( x ) , the

Original: Fenchel conjugate is given by g ∗ ( x ) =
Revised: Fenchel conjugate is given by g ∗ ( x ) =

Original: − x − 2 − 1 /x with domain x < 0 .
Revised: − x − 2 − 1 /x with domain x < 0 .

Original: Therefore, we write
Revised: Therefore, we write

Original: [Equation]
Revised: [Equation]

Original: The interchangeability principle (Rockafellar & Wets, 2009; Dai et al., 2017) allows us to convert the inner maximization step over scalar ζ to an overall maximization over ζ : S ×
Revised: The interchangeability principle (Rockafellar & Wets, 2009; Dai et al., 2017) allows us to convert the inner maximization step over scalar ζ to an overall maximization over ζ : S ×

Original: A → R − .
Revised: A → R − .

Original: Replacing this term in the objective (13) results in the following objective:
Revised: Replacing this term in the objective (13) results in the following objective:

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: E. P ROOF OF T HEOREM

Original: We start by deriving an expression for x ⋆w and characterizing bounds on u ⋆w and ζ ⋆w,v in the following lemma.
Revised: We start by deriving an expression for x ⋆w and characterizing bounds on u ⋆w and ζ ⋆w,v in the following lemma.

Original: The proof is presented in Appendix E.4.
Revised: The proof is presented in Appendix E.4.

Original: Lemma 8 For any w ∈ W and v ∈ V , one has x ⋆w ( s, a ) = 2 d w ( s ) /d
Revised: Lemma 8 For any w ∈ W and v ∈ V , one has x ⋆w ( s, a ) = 2 d w ( s ) /d

Original: π w ( s ) − 2 , | u ⋆w ( s, a ) |≤ 11 − γ ( B 2 x / 4 + B x ) , and | ζ ⋆w,v ( s, a ) |∈ (cid:104) 2 2+ B x
Revised: π w ( s ) − 2 , | u ⋆w ( s, a ) |≤ 11 − γ ( B 2 x / 4 + B x ) , and | ζ ⋆w,v ( s, a ) |∈ (cid:104) 2 2+ B x

Original: [Equation]
Revised: [Equation]

Original: Bounding the suboptimality of policies returned by both model-based and model-free variants of
Revised: Bounding the suboptimality of policies returned by both model-based and model-free variants of

Original: ALMIS follow a similar analysis.
Revised: ALMIS follow a similar analysis.

Original: We first characterize the statistical error in approximating population objectives by their empirical versions and use it to establish the closeness of ˆ w and w ⋆ .
Revised: We first characterize the statistical error in approximating population objectives by their empirical versions and use it to establish the closeness of ˆ w and w ⋆ .

Original: The lemma below captures these approximation errors for the model-based objective, whose proof can be found Appendix E.5.
Revised: The lemma below captures these approximation errors for the model-based objective, whose proof can be found Appendix E.5.

Original: Lemma 9 (Empirical and population closeness, model-based ALMIS)
Revised: Lemma 9 (Empirical and population closeness, model-based ALMIS)

Original: Fix δ > 0 and define ϵ model-basedstat :
Revised: Fix δ > 0 and define ϵ model-basedstat :

Original: [Equation]
Revised: [Equation]

Original: N .
Revised: N .

Original: For any w ∈ W , v ∈ V , and u ∈ U , the following statements hold with probability at least 1 − δ
Revised: For any w ∈ W , v ∈ V , and u ∈ U , the following statements hold with probability at least 1 − δ

Original: (I) (cid:12)(cid:12)(cid:12) L model-based
Revised: (I) (cid:12)(cid:12)(cid:12) L model-based

Original: [Equation]
Revised: [Equation]

Original: (II) L model-basedAL
Revised: (II) L model-basedAL

Original: [Equation]
Revised: [Equation]

Original: In Appendix E.6, we prove a similar lemma for the model-free objective.
Revised: In Appendix E.6, we prove a similar lemma for the model-free objective.

Original: Lemma 10 (Empirical and population closeness, model-free ALMIS)
Revised: Lemma 10 (Empirical and population closeness, model-free ALMIS)

Original: Fix δ > 0 and define
Revised: Fix δ > 0 and define

Original: [Equation]
Revised: [Equation]

Original: For any w ∈ W , v ∈ V , and u ∈ U , the following statements hold with probability at least 1 − δ
Revised: For any w ∈ W , v ∈ V , and u ∈ U , the following statements hold with probability at least 1 − δ

Original: (I) (cid:12)(cid:12)(cid:12) L model-freeAL ( w, v, u ) − ˆ L model-freeAL ( w, v, u ) (cid:12)(cid:12)(cid:12) ≤ ϵ model-freestat ; (II) L model-freeAL ( w ⋆ , v ⋆ , u ⋆ ) − L model-freeAL ( ˆ w, v ⋆ , u ⋆ ˆ w ) ≤ 2 ϵ model-freestat .
Revised: (I) (cid:12)(cid:12)(cid:12) L model-freeAL ( w, v, u ) − ˆ L model-freeAL ( w, v, u ) (cid:12)(cid:12)(cid:12) ≤ ϵ model-freestat ; (II) L model-freeAL ( w ⋆ , v ⋆ , u ⋆ ) − L model-freeAL ( ˆ w, v ⋆ , u ⋆ ˆ w ) ≤ 2 ϵ model-freestat .

Original: The final key lemma demonstrates that in model-based and model-free ALMIS, the ALM terms enforce lower bounds on the ratio of the estimated occupancy of learned weights d ˆ w ( s ) and the actual occupancy of the learned policy d π ˆ w ( s ) in most states.
Revised: The final key lemma demonstrates that in model-based and model-free ALMIS, the ALM terms enforce lower bounds on the ratio of the estimated occupancy of learned weights d ˆ w ( s ) and the actual occupancy of the learned policy d π ˆ w ( s ) in most states.

Original: The proof of this lemma is given in
Revised: The proof of this lemma is given in

Original: Appendix E.7.
Revised: Appendix E.7.

Original: Lemma 11 (Occupancy validity by the ALM, MDP)
Revised: Lemma 11 (Occupancy validity by the ALM, MDP)

Original: For ˆ w computed by the model-based ALMIS
Revised: For ˆ w computed by the model-based ALMIS

Original: Algorithm 3, define the state space subspace S s := (cid:8) s (cid:12)(cid:12)
Revised: Algorithm 3, define the state space subspace S s := (cid:8) s (cid:12)(cid:12)

Original: d ˆ w ( s ) ≤ 12 d π ˆ w ( s ) (cid:9) .
Revised: d ˆ w ( s ) ≤ 12 d π ˆ w ( s ) (cid:9) .

Original: For any fixed δ > 0 , the following statements hold with probability at least 1 − δ
Revised: For any fixed δ > 0 , the following statements hold with probability at least 1 − δ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly, for ˆ w computed by the model-free ALMIS Algorithm 4, define the state space subspace
Revised: Similarly, for ˆ w computed by the model-free ALMIS Algorithm 4, define the state space subspace

Original: S s :=
Revised: S s :=

Original: (cid:8) s (cid:12)(cid:12)
Revised: (cid:8) s (cid:12)(cid:12)

Original: d ˆ w ( s ) ≤ 12 d π ˆ w ( s ) (cid:9) .
Revised: d ˆ w ( s ) ≤ 12 d π ˆ w ( s ) (cid:9) .

Original: For any fixed δ > 0 , the following statements hold with probability at least 1 − δ
Revised: For any fixed δ > 0 , the following statements hold with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Given the above lemmas
Revised: Given the above lemmas

Original: [Equation]
Revised: [Equation]

Original: we proceed to prove the suboptimality bounds in terms of statistical errors defined in (55) and (56).
Revised: we proceed to prove the suboptimality bounds in terms of statistical errors defined in (55) and (56).

Original: In the rest of this section, we drop the superscripts model-based and model-free from statistical errors to avoid cluttered notation
Revised: In the rest of this section, we drop the superscripts model-based and model-free from statistical errors to avoid cluttered notation

Original: [Equation]
Revised: [Equation]

Original: In view of the performance difference lemma in Kakade & Langford (2002, Lemma 6.1)
Revised: In view of the performance difference lemma in Kakade & Langford (2002, Lemma 6.1)

Original: [Equation]
Revised: [Equation]

Original: one has
Revised: one has

Original: [Equation]
Revised: [Equation]

Original: where d ˆ π = d π ˆ w .
Revised: where d ˆ π = d π ˆ w .

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we used the fact that the expectation of the optimal advantage over optimal policy is zero (cid:80)
Revised: we used the fact that the expectation of the optimal advantage over optimal policy is zero (cid:80)

Original: a A ⋆ ( s, a ) π ⋆ ( a | s ) = 0 .
Revised: a A ⋆ ( s, a ) π ⋆ ( a | s ) = 0 .

Original: Lemma 11 links an expectation of − A ⋆ ( s, a ) to the statistical error.
Revised: Lemma 11 links an expectation of − A ⋆ ( s, a ) to the statistical error.

Original: With this lemma at hand and using the definition S s = { s | d ˆ w ( s ) ≤ d ˆ π ( s ) / 2 } , we continue to decompose and bound the suboptimality
Revised: With this lemma at hand and using the definition S s = { s | d ˆ w ( s ) ≤ d ˆ π ( s ) / 2 } , we continue to decompose and bound the suboptimality

Original: [Equation]
Revised: [Equation]

Original: In (57), we used part (II) in Lemma 11 and that − A ⋆ ( s, a ) ≤ 1 / (1 − γ ) and in (58) we used the definition of S s to bound the ratio d ˆ π ( s ) /d
Revised: In (57), we used part (II) in Lemma 11 and that − A ⋆ ( s, a ) ≤ 1 / (1 − γ ) and in (58) we used the definition of S s to bound the ratio d ˆ π ( s ) /d

Original: ˆ w ( s ) by 2 and the fact that d ˆ w ( s ) = 0 implies d ˆ π ( s ) = 0 for s / ∈ S s .
Revised: ˆ w ( s ) by 2 and the fact that d ˆ w ( s ) = 0 implies d ˆ π ( s ) = 0 for s / ∈ S s .

Original: We then apply part (I) in in Lemma 11 to bound the second term by E s,a ∼ µ [ − A ⋆ ( s, a ) ˆ w ( s, a )] and thus the overall suboptimality:
Revised: We then apply part (I) in in Lemma 11 to bound the second term by E s,a ∼ µ [ − A ⋆ ( s, a ) ˆ w ( s, a )] and thus the overall suboptimality:

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF L EMMA
Revised: E. P ROOF OF L EMMA

Original: Derivation of x ⋆w .
Revised: Derivation of x ⋆w .

Original: Recall from Appendix E.2 that for f ( x ) =
Revised: Recall from Appendix E.2 that for f ( x ) =

Original: ( x − 1) 2 , the Fenchel conjugate is f ∗ ( x ) =
Revised: ( x − 1) 2 , the Fenchel conjugate is f ∗ ( x ) =

Original: (cid:0) x +22 (cid:1) 2 − 1 .
Revised: (cid:0) x +22 (cid:1) 2 − 1 .

Original: Therefore, for any ( s, a ) ,
Revised: Therefore, for any ( s, a ) ,

Original: [Equation]
Revised: [Equation]

Original: ⇒ ˜ x
Revised: ⇒ ˜ x

Original: [Equation]
Revised: [Equation]

Original: ( s, a ) =
Revised: ( s, a ) =

Original: clip
Revised: clip

Original: [Equation]
Revised: [Equation]

Original: Bound on u ⋆w .
Revised: Bound on u ⋆w .

Original: Recall that u
Revised: Recall that u

Original: ⋆w
Revised: ⋆w

Original: is defined as the fixed point of the following Bellman-like equation
Revised: is defined as the fixed point of the following Bellman-like equation

Original: [Equation]
Revised: [Equation]

Original: The above equation has a solution since f ∗ (˜ x w ( s, a )) is bounded
Revised: The above equation has a solution since f ∗ (˜ x w ( s, a )) is bounded

Original: [Equation]
Revised: [Equation]

Original: One can view u ⋆w as the Q-function of policy π w with the reward function f ∗ (˜ x w ( s, a )) , which leads to | u ⋆w
Revised: One can view u ⋆w as the Q-function of policy π w with the reward function f ∗ (˜ x w ( s, a )) , which leads to | u ⋆w

Original: [Equation]
Revised: [Equation]

Original: s, a ) |
Revised: s, a ) |

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Bound on ζ ⋆w,u .
Revised: Bound on ζ ⋆w,u .

Original: To see the bound on ζ ⋆w,u , recall that by definition, ζ ⋆w,u = arg
Revised: To see the bound on ζ ⋆w,u , recall that by definition, ζ ⋆w,u = arg

Original: [Equation]
Revised: [Equation]

Original: It is easy to show that | ζ ⋆w,u ( s, a ) |
Revised: It is easy to show that | ζ ⋆w,u ( s, a ) |

Original: =
Revised: =

Original: ( u ( s, a ) − γ
Revised: ( u ( s, a ) − γ

Original: ( P π w u )( s ′ , a ′ )+1)
Revised: ( P π w u )( s ′ , a ′ )+1)

Original: − 1 / 2 =
Revised: − 1 / 2 =

Original: ( f ∗ (˜ x w ( s, a ))+1)
Revised: ( f ∗ (˜ x w ( s, a ))+1)

Original: − 1 / 2 .
Revised: − 1 / 2 .

Original: Since ˜ x w ( s, a ) ∈ [ − B x , B x ] , we have | ζ ⋆w,u ( s, a ) |∈ (cid:104) 2 2+ B x ,
Revised: Since ˜ x w ( s, a ) ∈ [ − B x , B x ] , we have | ζ ⋆w,u ( s, a ) |∈ (cid:104) 2 2+ B x ,

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF L EMMAE.5. P ROOF OF PART (I)
Revised: E. P ROOF OF L EMMAE.5. P ROOF OF PART (I)

Original: We decompose the difference between the population and empirical objective into three terms
Revised: We decompose the difference between the population and empirical objective into three terms

Original: L model-basedAL − ˆ L model-basedAL =
Revised: L model-basedAL − ˆ L model-basedAL =

Original: T 1 + T 2 + T 3 defined as follows
Revised: T 1 + T 2 + T 3 defined as follows

Original: T 1 :
Revised: T 1 :

Original: = (1 − γ ) E ρ (cid:34)
Revised: = (1 − γ ) E ρ (cid:34)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:32)
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: a ) π w ( a | s i ) (cid:33)
Revised: a ) π w ( a | s i ) (cid:33)

Original: T 2 := E µ (cid:34) w ( s, a )( r ( s, a ) + γ (cid:88) s ′
Revised: T 2 := E µ (cid:34) w ( s, a )( r ( s, a ) + γ (cid:88) s ′

Original: P ( s ′ | s
Revised: P ( s ′ | s

Original: [Equation]
Revised: [Equation]

Original: a ) v ( s ′ )
Revised: a ) v ( s ′ )

Original: − v ( s ))
Revised: − v ( s ))

Original: (cid:35) −
Revised: (cid:35) −

Original: N
Revised: N

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: w ( s i
Revised: w ( s i

Original: [Equation]
Revised: [Equation]

Original: a i )
Revised: a i )

Original: [ r i + γv ( s ′ i ) − v ( s i )]
Revised: [ r i + γv ( s ′ i ) − v ( s i )]

Original: T 3 := E µ (cid:2) w (
Revised: T 3 := E µ (cid:2) w (

Original: [Equation]
Revised: [Equation]

Original: We subsequently show that the absolute values of the above error terms satisfy the following high probability upper bounds:
Revised: We subsequently show that the absolute values of the above error terms satisfy the following high probability upper bounds:

Original: [Equation]
Revised: [Equation]

Original: Taking N 0 = N and noting that B w ≥ 1 due to realizability of w ⋆ yield that
Revised: Taking N 0 = N and noting that B w ≥ 1 due to realizability of w ⋆ yield that

Original: [Equation]
Revised: [Equation]

Original: Proof of bound (61a) on | T 1 | .
Revised: Proof of bound (61a) on | T 1 | .

Original: Since | v ( s ) |≤ B v , | u ( s, a ) |≤ B u for all v ∈ V and u ∈ U and s i are independent, we can apply Hoeffding’s inequality and union bound to conclude the advertised bound (61a) on | T 1 | .
Revised: Since | v ( s ) |≤ B v , | u ( s, a ) |≤ B u for all v ∈ V and u ∈ U and s i are independent, we can apply Hoeffding’s inequality and union bound to conclude the advertised bound (61a) on | T 1 | .

Original: Proof of the bound (61b) on | T 2 | .
Revised: Proof of the bound (61b) on | T 2 | .

Original: By boundedness of w, v , we have
Revised: By boundedness of w, v , we have

Original: [Equation]
Revised: [Equation]

Original: As before, due to boundedness and independence of variables w ( s i , a i )[ r i + γv ( s ′ i ) − v ( s i )] , Hoeffding’s inequality can be applied, giving the bound (61b) on | T 2 | .
Revised: As before, due to boundedness and independence of variables w ( s i , a i )[ r i + γv ( s ′ i ) − v ( s i )] , Hoeffding’s inequality can be applied, giving the bound (61b) on | T 2 | .

Original: Proof of the bound (61c) on | T 3 | .
Revised: Proof of the bound (61c) on | T 3 | .

Original: We decompose T 3 =
Revised: We decompose T 3 =

Original: T 3 , 1 + T 3 , 2 , where T 3 , 1 and T 3 , 2 are defined as
Revised: T 3 , 1 + T 3 , 2 , where T 3 , 1 and T 3 , 2 are defined as

Original: [Equation]
Revised: [Equation]

Original: Recall that f − 1 ∗ ( x ) = 2 √ x + 1 − 2 from Appendix E.2.
Revised: Recall that f − 1 ∗ ( x ) = 2 √ x + 1 − 2 from Appendix E.2.

Original: The absolute value of T 3 , 2 can be immediately bounded using Hoeffding’s inequality:
Revised: The absolute value of T 3 , 2 can be immediately bounded using Hoeffding’s inequality:

Original: [Equation]
Revised: [Equation]

Original: To bound | T 3 , 1 |
Revised: To bound | T 3 , 1 |

Original: [Equation]
Revised: [Equation]

Original: we first use the inequality given in Lemma 13, setting b i , x i , y i for each ( s, a ) according to
Revised: we first use the inequality given in Lemma 13, setting b i , x i , y i for each ( s, a ) according to

Original: [Equation]
Revised: [Equation]

Original: Thus by Lemma
Revised: Thus by Lemma

Original: [Equation]
Revised: [Equation]

Original: we obtain the following bound on T 23 ,
Revised: we obtain the following bound on T 23 ,

Original: T 23 , 1 =
Revised: T 23 , 1 =

Original: [Equation]
Revised: [Equation]

Original: Note that the terms under square root are always nonnegative because for any transition P 1 + u
Revised: Note that the terms under square root are always nonnegative because for any transition P 1 + u

Original: [Equation]
Revised: [Equation]

Original: s, a
Revised: s, a

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: Then, we use the concentration result on maximum likelihood model estimation stated in Theorem 6 and a union bound on w ∈ W and v ∈ V to conclude that
Revised: Then, we use the concentration result on maximum likelihood model estimation stated in Theorem 6 and a union bound on w ∈ W and v ∈ V to conclude that

Original: [Equation]
Revised: [Equation]

Original: E.5. P ROOF OF PART (II)
Revised: E.5. P ROOF OF PART (II)

Original: To prove the second part
Revised: To prove the second part

Original: [Equation]
Revised: [Equation]

Original: let ˆ v w and ˆ u w denote the solutions to the model-based empirical objective
Revised: let ˆ v w and ˆ u w denote the solutions to the model-based empirical objective

Original: [Equation]
Revised: [Equation]

Original: Decompose the objective difference according to
Revised: Decompose the objective difference according to

Original: [Equation]
Revised: [Equation]

Original: We bound each term:
Revised: We bound each term:

Original: • T 1 ≤ 0 because v ⋆ , u ⋆ = arg min v arg min u L model-basedAL
Revised: • T 1 ≤ 0 because v ⋆ , u ⋆ = arg min v arg min u L model-basedAL

Original: [Equation]
Revised: [Equation]

Original: w ⋆
Revised: w ⋆

Original: [Equation]
Revised: [Equation]

Original: ; • T 2 ≤ ϵ model-basedstat
Revised: ; • T 2 ≤ ϵ model-basedstat

Original: by Lemma 9;
Revised: by Lemma 9;

Original: • T 3 ≤ 0 because ˆ w = arg max w ∈W ˆ L model-basedAL ( w, ˆ v w , ˆ u w ) ; • T 4 ≤ 0 because ˆ v w , ˆ u w = arg min v ∈V arg min u ∈U ˆ L model-basedAL ( w, v, u ) ; • T 5 ≤ ϵ model-basedstat by Lemma 9.
Revised: • T 3 ≤ 0 because ˆ w = arg max w ∈W ˆ L model-basedAL ( w, ˆ v w , ˆ u w ) ; • T 4 ≤ 0 because ˆ v w , ˆ u w = arg min v ∈V arg min u ∈U ˆ L model-basedAL ( w, v, u ) ; • T 5 ≤ ϵ model-basedstat by Lemma 9.

Original: E. P ROOF OF L EMMAE.6. P ROOF OF PART (I)
Revised: E. P ROOF OF L EMMAE.6. P ROOF OF PART (I)

Original: We decompose the difference L model-freeAL − ˆ L model-freeAL = T 1 + T 2 + T 3 into three error terms
Revised: We decompose the difference L model-freeAL − ˆ L model-freeAL = T 1 + T 2 + T 3 into three error terms

Original: T 1 :
Revised: T 1 :

Original: = (1 − γ ) E ρ (cid:34) v ( s ) + (cid:88) a u ( s, a ) π w ( a | s ) (cid:35) − (1 − γ ) 1 N
Revised: = (1 − γ ) E ρ (cid:34) v ( s ) + (cid:88) a u ( s, a ) π w ( a | s ) (cid:35) − (1 − γ ) 1 N

Original: N 0 (cid:88) i = (cid:32) v ( s i ) + (cid:88) a u
Revised: N 0 (cid:88) i = (cid:32) v ( s i ) + (cid:88) a u

Original: ( s i , a ) π w ( a | s i ) (cid:33)
Revised: ( s i , a ) π w ( a | s i ) (cid:33)

Original: T 2 := E µ (cid:34) w ( s, a )( r ( s, a ) + γ (cid:88) s ′
Revised: T 2 := E µ (cid:34) w ( s, a )( r ( s, a ) + γ (cid:88) s ′

Original: P ( s ′ | s, a ) v ( s ′ )
Revised: P ( s ′ | s, a ) v ( s ′ )

Original: − v ( s ))
Revised: − v ( s ))

Original: (cid:35) −
Revised: (cid:35) −

Original: N
Revised: N

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: w ( s i , a i )
Revised: w ( s i , a i )

Original: [ r i + γv ( s ′ i ) − v ( s i )]
Revised: [ r i + γv ( s ′ i ) − v ( s i )]

Original: T 3 :=
Revised: T 3 :=

Original: E ( s,a,s ′ ) ∼ µ,a ′ ∼ π w ( ·| s ′ )
Revised: E ( s,a,s ′ ) ∼ µ,a ′ ∼ π w ( ·| s ′ )

Original: [ w ( s, a ) (( u ( s, a ) − γu ( s ′ , a ′ ))
Revised: [ w ( s, a ) (( u ( s, a ) − γu ( s ′ , a ′ ))

Original: ζ ( s, a ) − g ⋆ ( ζ ( s, a )))] −
Revised: ζ ( s, a ) − g ⋆ ( ζ ( s, a )))] −

Original: N
Revised: N

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: w ( s i , a i ) (cid:34)(cid:32) u
Revised: w ( s i , a i ) (cid:34)(cid:32) u

Original: ( s i , a i ) − γ (cid:88) a ′ ∈A u
Revised: ( s i , a i ) − γ (cid:88) a ′ ∈A u

Original: ( s ′ i , a ′ )
Revised: ( s ′ i , a ′ )

Original: π w ( a ′ | s ′ i )
Revised: π w ( a ′ | s ′ i )

Original: [Equation]
Revised: [Equation]

Original: The absolute values of the error terms above satisfy the following upper bounds with high probability
Revised: The absolute values of the error terms above satisfy the following upper bounds with high probability

Original: [Equation]
Revised: [Equation]

Original: The bounds on the first two error terms | T 1 | and | T 2 | are already shown in Appendix E.5.1.
Revised: The bounds on the first two error terms | T 1 | and | T 2 | are already shown in Appendix E.5.1.

Original: To bound | T 3 | , recall that g ⋆ ( x ) =
Revised: To bound | T 3 | , recall that g ⋆ ( x ) =

Original: − x − 2 − 1 x , ∀ x < 0 .
Revised: − x − 2 − 1 x , ∀ x < 0 .

Original: Also, | ζ ( s, a ) |∈ ( B ζ,L , B ζ,U ) for any ζ ∈ Z and any ( s, a ) , and B ζ ≜ max { B ζ,U , B − 1 ζ,L } .
Revised: Also, | ζ ( s, a ) |∈ ( B ζ,L , B ζ,U ) for any ζ ∈ Z and any ( s, a ) , and B ζ ≜ max { B ζ,U , B − 1 ζ,L } .

Original: Therefore, the individual error terms in | T 3 | satisfy the following bound
Revised: Therefore, the individual error terms in | T 3 | satisfy the following bound

Original: [Equation]
Revised: [Equation]

Original: Thus, by Hoeffding’s inequality and a union bound on W , U
Revised: Thus, by Hoeffding’s inequality and a union bound on W , U

Original: [Equation]
Revised: [Equation]

Original: and Z
Revised: and Z

Original: [Equation]
Revised: [Equation]

Original: we obtain the upper bound (65c) on | T 3 | .
Revised: we obtain the upper bound (65c) on | T 3 | .

Original: Summing up the bounds given in (65a)
Revised: Summing up the bounds given in (65a)

Original: [Equation]
Revised: [Equation]

Original: (65b)
Revised: (65b)

Original: [Equation]
Revised: [Equation]

Original: and (65c) and noting that B w ≥ 1 due to realizability of w ⋆ , we obtain
Revised: and (65c) and noting that B w ≥ 1 due to realizability of w ⋆ , we obtain

Original: [Equation]
Revised: [Equation]

Original: E.6. P ROOF OF PART (II)
Revised: E.6. P ROOF OF PART (II)

Original: Define the following solutions to the empirical model-free objective
Revised: Define the following solutions to the empirical model-free objective

Original: [Equation]
Revised: [Equation]

Original: w,
Revised: w,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , ζ
Revised: , ζ

Original: [Equation]
Revised: [Equation]

Original: Decompose the objective difference according to
Revised: Decompose the objective difference according to

Original: L model-freeAL
Revised: L model-freeAL

Original: [Equation]
Revised: [Equation]

Original: We bound each term:
Revised: We bound each term:

Original: • T 1 ≤ 0 because v ⋆ , u ⋆ = arg min v,u L model-freeAL ( w ⋆ , v, u, ζ ⋆ ( w ⋆ , u )) ; • T 2 ≤ ϵ model-free by part (I); • T 3 ≤ 0 because ˆ ζ w ⋆ = ˆ ζ w ⋆ , ˆ u w⋆ = arg max ζ ∈Z ˆ L model-freeAL ( w ⋆ , ˆ v w ⋆ , ˆ u w ⋆ , ζ ) ; • T 4 ≤ 0 because ˆ w = arg max w ∈W ˆ L model-freeAL ( w, ˆ v w , ˆ u w , ˆ ζ w ) ; • T 5 ≤ 0 because ˆ v ˆ w , ˆ u ˆ w = arg min v ∈V ,u ∈U
Revised: • T 1 ≤ 0 because v ⋆ , u ⋆ = arg min v,u L model-freeAL ( w ⋆ , v, u, ζ ⋆ ( w ⋆ , u )) ; • T 2 ≤ ϵ model-free by part (I); • T 3 ≤ 0 because ˆ ζ w ⋆ = ˆ ζ w ⋆ , ˆ u w⋆ = arg max ζ ∈Z ˆ L model-freeAL ( w ⋆ , ˆ v w ⋆ , ˆ u w ⋆ , ζ ) ; • T 4 ≤ 0 because ˆ w = arg max w ∈W ˆ L model-freeAL ( w, ˆ v w , ˆ u w , ˆ ζ w ) ; • T 5 ≤ 0 because ˆ v ˆ w , ˆ u ˆ w = arg min v ∈V ,u ∈U

Original: ˆ L model-freeAL ( ˆ w, v, u, ˆ ζ ˆ w,u ) ; • T 6 ≤ ϵ model-free by part (I); • T 7 ≤ 0 because ζ ⋆ ˆ w = ζ ⋆ ˆ w,u ⋆ ˆ w = arg max ζ< 0 L model-freeAL ( ˆ w, v ⋆ , u ⋆ ˆ w , ζ ) .
Revised: ˆ L model-freeAL ( ˆ w, v, u, ˆ ζ ˆ w,u ) ; • T 6 ≤ ϵ model-free by part (I); • T 7 ≤ 0 because ζ ⋆ ˆ w = ζ ⋆ ˆ w,u ⋆ ˆ w = arg max ζ< 0 L model-freeAL ( ˆ w, v ⋆ , u ⋆ ˆ w , ζ ) .

Original: E. P ROOF OF L EMMA
Revised: E. P ROOF OF L EMMA

Original: We provide proof only for the model-based algorithm and let ˆ w = ˆ w model-based for notation convenience.
Revised: We provide proof only for the model-based algorithm and let ˆ w = ˆ w model-based for notation convenience.

Original: The proof for a model-free algorithm follows analogously, noting the fact that
Revised: The proof for a model-free algorithm follows analogously, noting the fact that

Original: L model-free AL ( w, v ⋆ , u ⋆w , ζ ⋆w ) =
Revised: L model-free AL ( w, v ⋆ , u ⋆w , ζ ⋆w ) =

Original: L model-based AL ( w, v ⋆ , u ⋆w ) and we can replace Lemma 9 with Lemmato prove the model-free version.
Revised: L model-based AL ( w, v ⋆ , u ⋆w ) and we can replace Lemma 9 with Lemmato prove the model-free version.

Original: E.7. P ROOF OF PART (I)
Revised: E.7. P ROOF OF PART (I)

Original: Consider the expression of the model-based objective L model-basedAL (
Revised: Consider the expression of the model-based objective L model-basedAL (

Original: [Equation]
Revised: [Equation]

Original: ⋆ ,
Revised: ⋆ ,

Original: [Equation]
Revised: [Equation]

Original: ⋆ , u ⋆ ) at the optimal solution where u ⋆ := u ⋆w ⋆ :
Revised: ⋆ , u ⋆ ) at the optimal solution where u ⋆ := u ⋆w ⋆ :

Original: [Equation]
Revised: [Equation]

Original: The first equation comes from the fact that u ⋆ is the optimal solution to the variational lower bound, making it equal to the f -divergence.
Revised: The first equation comes from the fact that u ⋆ is the optimal solution to the variational lower bound, making it equal to the f -divergence.

Original: To see this, recall from Lemma 8 that x ⋆w ( s, a ) =
Revised: To see this, recall from Lemma 8 that x ⋆w ( s, a ) =

Original: 2 d w ( s ) /d
Revised: 2 d w ( s ) /d

Original: π w ( s ) − 2 and ˜ x w ( s, a )
Revised: π w ( s ) − 2 and ˜ x w ( s, a )

Original: = clip( x ⋆w ( s, a ) , − B x , B x ) .
Revised: = clip( x ⋆w ( s, a ) , − B x , B x ) .

Original: Since x ⋆w ⋆ ( s, a ) = 0 , we have ˜ x w ⋆ ( s, a ) =
Revised: Since x ⋆w ⋆ ( s, a ) = 0 , we have ˜ x w ⋆ ( s, a ) =

Original: x ⋆w ⋆ ( s, a ) and thus u ⋆ recovers the f -divergence.
Revised: x ⋆w ⋆ ( s, a ) and thus u ⋆ recovers the f -divergence.

Original: In Equation (66), we wrote v ⋆ ( s ) =
Revised: In Equation (66), we wrote v ⋆ ( s ) =

Original: V ⋆ ( s ) since v ⋆ ( s ) is the optimal solution to the primal-dual program without the ALM term and is equal to the optimal value function (Zhan et al., 2022).
Revised: V ⋆ ( s ) since v ⋆ ( s ) is the optimal solution to the primal-dual program without the ALM term and is equal to the optimal value function (Zhan et al., 2022).

Original: We also used the fact that e v ⋆ ( s, a ) =
Revised: We also used the fact that e v ⋆ ( s, a ) =

Original: r ( s, a )+ γ (cid:80) s ′ P
Revised: r ( s, a )+ γ (cid:80) s ′ P

Original: ( s ′ | s, a ) v ⋆ ( s ′ )
Revised: ( s ′ | s, a ) v ⋆ ( s ′ )

Original: − v ⋆ ( s )
Revised: − v ⋆ ( s )

Original: =
Revised: =

Original: A ⋆ ( s, a ) is the optimal advantage function, and that d w ⋆ ( s ) =
Revised: A ⋆ ( s, a ) is the optimal advantage function, and that d w ⋆ ( s ) =

Original: d π w⋆ ( s ) by definition and realizability of w ⋆ .
Revised: d π w⋆ ( s ) by definition and realizability of w ⋆ .

Original: Moreover, the second term in (66) is zero since it captures the optimal advantage of optimal policy.
Revised: Moreover, the second term in (66) is zero since it captures the optimal advantage of optimal policy.

Original: Therefore, we conclude that
Revised: Therefore, we conclude that

Original: [Equation]
Revised: [Equation]

Original: Given the above expression of the objective at ( w ⋆ , v ⋆ , u ⋆ )
Revised: Given the above expression of the objective at ( w ⋆ , v ⋆ , u ⋆ )

Original: [Equation]
Revised: [Equation]

Original: we write the following objective difference
Revised: we write the following objective difference

Original: [Equation]
Revised: [Equation]

Original: The last line uses e v ⋆ ( s, a ) =
Revised: The last line uses e v ⋆ ( s, a ) =

Original: A ⋆ ( s, a ) as well as the definition of u ⋆ ˆ w as the fixed point solution to
Revised: A ⋆ ( s, a ) as well as the definition of u ⋆ ˆ w as the fixed point solution to

Original: [Equation]
Revised: [Equation]

Original: which allows us to write (68) in the original f -divergence variational form (11) with ˜ x ˆ w as variable
Revised: which allows us to write (68) in the original f -divergence variational form (11) with ˜ x ˆ w as variable

Original: [Equation]
Revised: [Equation]

Original: Lemma 9 asserts that L
Revised: Lemma 9 asserts that L

Original: [Equation]
Revised: [Equation]

Original: AL
Revised: AL

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: We next argue that both terms in inequality above are nonnegative and conclude that
Revised: We next argue that both terms in inequality above are nonnegative and conclude that

Original: [Equation]
Revised: [Equation]

Original: The first term is nonnegative because for the optimal advantage function we have A ⋆ ( s, a ) ≤ 0 for all s ∈ S and a ∈ A .
Revised: The first term is nonnegative because for the optimal advantage function we have A ⋆ ( s, a ) ≤ 0 for all s ∈ S and a ∈ A .

Original: We write the second term as
Revised: We write the second term as

Original: [Equation]
Revised: [Equation]

Original: We then show that each term inside the expectation is nonnegative:
Revised: We then show that each term inside the expectation is nonnegative:

Original: [Equation]
Revised: [Equation]

Original: Proof of bound (71) .
Revised: Proof of bound (71) .

Original: we separate the argument into three cases and use the expression of ˜ x w given in Lemma
Revised: we separate the argument into three cases and use the expression of ˜ x w given in Lemma

Original: [Equation]
Revised: [Equation]

Original: When 1 − B x / 2 ≤ d w ( s ) d πw ( s ) ≤ B x / 2 + 1 , we have ˜ x w ( s, a )
Revised: When 1 − B x / 2 ≤ d w ( s ) d πw ( s ) ≤ B x / 2 + 1 , we have ˜ x w ( s, a )

Original: =
Revised: =

Original: (cid:16) 2 d w ( s )
Revised: (cid:16) 2 d w ( s )

Original: d πw ( s ) − 2 (cid:17) and therefore d w ( s )
Revised: d πw ( s ) − 2 (cid:17) and therefore d w ( s )

Original: [Equation]
Revised: [Equation]

Original: ≥ 0 .
Revised: ≥ 0 .

Original: When d w ( s ) d πw ( s ) >
Revised: When d w ( s ) d πw ( s ) >

Original: B x / 2 + 1 , substitute ˜ x w ( s, a ) =
Revised: B x / 2 + 1 , substitute ˜ x w ( s, a ) =

Original: B x to arrive at
Revised: B x to arrive at

Original: [Equation]
Revised: [Equation]

Original: Similarly, when d w ( s ) d πw ( s ) < 1 − B x / 2 , substitute ˜ x w ( s, a ) =
Revised: Similarly, when d w ( s ) d πw ( s ) < 1 − B x / 2 , substitute ˜ x w ( s, a ) =

Original: − B x to arrive at
Revised: − B x to arrive at

Original: [Equation]
Revised: [Equation]

Original: E.7. P ROOF OF PART (II)
Revised: E.7. P ROOF OF PART (II)

Original: We derive the second part by using the bound (70b) restricted on the set S s .
Revised: We derive the second part by using the bound (70b) restricted on the set S s .

Original: When s ∈ S s , we have d ˆ w ( s ) d π ˆ w ( s ) ≤ 12 and thus the variational form falls into the case 3 in the proof of bound (71).
Revised: When s ∈ S s , we have d ˆ w ( s ) d π ˆ w ( s ) ≤ 12 and thus the variational form falls into the case 3 in the proof of bound (71).

Original: Therefore, for s ∈ S s , we have ˜ x
Revised: Therefore, for s ∈ S s , we have ˜ x

Original: ˆ w ( s, a ) =
Revised: ˆ w ( s, a ) =

Original: − B x and d ˆ w ( s ) d
Revised: − B x and d ˆ w ( s ) d

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: We use the bound in (70b) as well as (73) to conclude that
Revised: We use the bound in (70b) as well as (73) to conclude that

Original: [Equation]
Revised: [Equation]

Original: which leads to the second advertised claim (cid:80)
Revised: which leads to the second advertised claim (cid:80)

Original: [Equation]
Revised: [Equation]

Original: E. ALM BASED ON B ELLMAN FLOW ERROR CONSTRAINT IS INSUFFICIENT
Revised: E. ALM BASED ON B ELLMAN FLOW ERROR CONSTRAINT IS INSUFFICIENT

Original: We demonstrated that Lagrange multipliers are not sufficient to enforce occupancy validity and we need to use additional penalty terms.
Revised: We demonstrated that Lagrange multipliers are not sufficient to enforce occupancy validity and we need to use additional penalty terms.

Original: Furthermore, we discussed why ensuring ratio-based occupancy validity is compatible with the single-policy concentrability definition, resulting in learning a policy whose actual occupancy is within the data distribution.
Revised: Furthermore, we discussed why ensuring ratio-based occupancy validity is compatible with the single-policy concentrability definition, resulting in learning a policy whose actual occupancy is within the data distribution.

Original: However, one might wonder whether a more standard application of the ALM term, which involves adding a squared penalty on Bellman flow error, leads to a similar policy validity guarantee.
Revised: However, one might wonder whether a more standard application of the ALM term, which involves adding a squared penalty on Bellman flow error, leads to a similar policy validity guarantee.

Original: This idea is appealing because it avoids variational lower bound and additional variables.
Revised: This idea is appealing because it avoids variational lower bound and additional variables.

Original: However, here we provide an intuitive argument that a penalty term on Bellman flow error does not appear to be sufficient to ensure a ratio-based occupancy validity guarantee.
Revised: However, here we provide an intuitive argument that a penalty term on Bellman flow error does not appear to be sufficient to ensure a ratio-based occupancy validity guarantee.

Original: We use ϵ ( s ) to denote Bellman flow error defined as
Revised: We use ϵ ( s ) to denote Bellman flow error defined as

Original: [Equation]
Revised: [Equation]

Original: We show that even such a strong state-vise guarantee on Bellman error cannot generally lead to d π ˆ w ( s ) d ˆ w ( s ) being bounded by a constant.
Revised: We show that even such a strong state-vise guarantee on Bellman error cannot generally lead to d π ˆ w ( s ) d ˆ w ( s ) being bounded by a constant.

Original: We argue this by contradiction.
Revised: We argue this by contradiction.

Original: Assume that for 0 ≤ c <
Revised: Assume that for 0 ≤ c <

Original: [Equation]
Revised: [Equation]

Original: Since d π ˆ w satisfies the Bellman flow equations, we can show d π ˆ w − d ˆ w =
Revised: Since d π ˆ w satisfies the Bellman flow equations, we can show d π ˆ w − d ˆ w =

Original: ( I − γP π ˆ w ) − 1 ϵ .
Revised: ( I − γP π ˆ w ) − 1 ϵ .

Original: Moreover, we have d π ˆ w =
Revised: Moreover, we have d π ˆ w =

Original: ( I − γP π ˆ w )
Revised: ( I − γP π ˆ w )

Original: − 1 ρ .
Revised: − 1 ρ .

Original: Substituting these equations to (74), we conclude that for 0 ≤ c <( I − γP π ˆ w ) − 1 ϵ ≤ c ( I − γP π ˆ w ) − 1 ρ ⇐⇒ ϵ ≤ cρ.
Revised: Substituting these equations to (74), we conclude that for 0 ≤ c <( I − γP π ˆ w ) − 1 ϵ ≤ c ( I − γP π ˆ w ) − 1 ρ ⇐⇒ ϵ ≤ cρ.

Original: Therefore, to ensure a constant bound on d π ˆ w ( s ) d ˆ w ( s ) , we require Bellman flow error to be pointwise smaller than the initial distribution.
Revised: Therefore, to ensure a constant bound on d π ˆ w ( s ) d ˆ w ( s ) , we require Bellman flow error to be pointwise smaller than the initial distribution.

Original: For state s with ρ ( s ) = 0 , this means that the Bellman flow error is required to be nonpositive: ϵ ( s ) ≤ 0 .
Revised: For state s with ρ ( s ) = 0 , this means that the Bellman flow error is required to be nonpositive: ϵ ( s ) ≤ 0 .

Original: However, even state-wise minimization of squared penalty terms such as ϵ 2 ( s ) can only ensure | ϵ ( s ) | to be small.
Revised: However, even state-wise minimization of squared penalty terms such as ϵ 2 ( s ) can only ensure | ϵ ( s ) | to be small.

Original: F R OBUSTNESS TO MODEL MISSPECIFICATION AND OPTIMIZATION ERROR
Revised: F R OBUSTNESS TO MODEL MISSPECIFICATION AND OPTIMIZATION ERROR

Original: In this section, we study the sample complexity of our algorithm in the presence of model misspecification and optimization error similar to Zhan et al.
Revised: In this section, we study the sample complexity of our algorithm in the presence of model misspecification and optimization error similar to Zhan et al.

Original: Since in practice, it might be the case that our function classes W , V do not contain w ⋆ , v ⋆ , similar to Zhan et al.
Revised: Since in practice, it might be the case that our function classes W , V do not contain w ⋆ , v ⋆ , similar to Zhan et al.

Original: (2022), we measure the approximation errors of W and V by
Revised: (2022), we measure the approximation errors of W and V by

Original: [Equation]
Revised: [Equation]

Original: where w ⋆ = d ⋆ /µ
Revised: where w ⋆ = d ⋆ /µ

Original: and
Revised: and

Original: d ⋆ is the (discounted) occupancy frequency of any optimal policy π ⋆ , ∥·∥ 1 ,ρ is weighted l 1 norm w.r.t.
Revised: d ⋆ is the (discounted) occupancy frequency of any optimal policy π ⋆ , ∥·∥ 1 ,ρ is weighted l 1 norm w.r.t.

Original: ρ , and µ ′ ( s ) = (cid:80) s ′ ,a ′ P ( s
Revised: ρ , and µ ′ ( s ) = (cid:80) s ′ ,a ′ P ( s

Original: | s ′ , a ′ ) µ ( s ′ , a ′ ) .
Revised: | s ′ , a ′ ) µ ( s ′ , a ′ ) .

Original: The model misspecification error is measured in l 1 norm, which is weaker than l ∞ norm.
Revised: The model misspecification error is measured in l 1 norm, which is weaker than l ∞ norm.

Original: Furthermore, we also consider the optimization error of practical optimization algorithms since in real-world scenarios it is unlikely that an algorithm can recover the exact optimal solution.
Revised: Furthermore, we also consider the optimization error of practical optimization algorithms since in real-world scenarios it is unlikely that an algorithm can recover the exact optimal solution.

Original: Instead, a typical optimization algorithm is able to find an approximate solution that is close enough to the true optimal solution.
Revised: Instead, a typical optimization algorithm is able to find an approximate solution that is close enough to the true optimal solution.

Original: Formally, we assume that the solution ( ˆ w, ˆ v ) that the optimizer obtained satisfies
Revised: Formally, we assume that the solution ( ˆ w, ˆ v ) that the optimizer obtained satisfies

Original: [Equation]
Revised: [Equation]

Original: where the objective ˆ L can be substituted by any objective with ALM term in different settings (e.g., ˆ
Revised: where the objective ˆ L can be substituted by any objective with ALM term in different settings (e.g., ˆ

Original: L =
Revised: L =

Original: ˆ L CBAL in contextual bandits).
Revised: ˆ L CBAL in contextual bandits).

Original: The assumption above is also similar to Zhan et al.
Revised: The assumption above is also similar to Zhan et al.

Original: , and it assumes that ˆ L ( ˆ w, ˆ v ) ≈ max w ∈W min v ∈V ˆ L ( w, v ) , which shows that ( ˆ w, ˆ v ) is an approximate max-min solution of ˆ L .
Revised: , and it assumes that ˆ L ( ˆ w, ˆ v ) ≈ max w ∈W min v ∈V ˆ L ( w, v ) , which shows that ( ˆ w, ˆ v ) is an approximate max-min solution of ˆ L .