Original: LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation
Revised: LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We consider the problem of learning from observation (LfO), in which the agentaims to mimic the expert’s behavior from the state-only demonstrations by experts.
Revised: We consider the problem of learning from observation (LfO), in which the agentaims to mimic the expert’s behavior from the state-only demonstrations by experts.

Original: We additionally assume that the agent cannot interact with the environment buthas access to the action-labeled transition data collected by some agents withunknown qualities.
Revised: We additionally assume that the agent cannot interact with the environment buthas access to the action-labeled transition data collected by some agents withunknown qualities.

Original: This offline setting for LfO is appealing in many real-worldscenarios where the ground-truth expert actions are inaccessible and the arbitraryenvironment interactions are costly or risky.
Revised: This offline setting for LfO is appealing in many real-worldscenarios where the ground-truth expert actions are inaccessible and the arbitraryenvironment interactions are costly or risky.

Original: In this paper, we present LobsDICE,an offline LfO algorithm that learns to imitate the expert policy via optimizationin the space of stationary distributions.
Revised: In this paper, we present LobsDICE,an offline LfO algorithm that learns to imitate the expert policy via optimizationin the space of stationary distributions.

Original: Our algorithm solves a single convexminimization problem, which minimizes the divergence between the two statetransition distributions induced by the expert and the agent policy.
Revised: Our algorithm solves a single convexminimization problem, which minimizes the divergence between the two statetransition distributions induced by the expert and the agent policy.

Original: Through anextensive set of offline LfO tasks, we show that LobsDICE outperforms strongbaseline methods.
Revised: Through anextensive set of offline LfO tasks, we show that LobsDICE outperforms strongbaseline methods.

Original:  Introduction
Revised:  Introduction

Original: The ability to learn from experience is one of the core aspects of an intelligent agent.
Revised: The ability to learn from experience is one of the core aspects of an intelligent agent.

Original: Reinforcementlearning (RL) [36] provides a framework to acquire such an intelligent behavior autonomouslythrough interactions with the environment while receiving reward feedback.
Revised: Reinforcementlearning (RL) [36] provides a framework to acquire such an intelligent behavior autonomouslythrough interactions with the environment while receiving reward feedback.

Original: However, the practicalapplicability of RL to real-world domains has been limited for two reasons.
Revised: However, the practicalapplicability of RL to real-world domains has been limited for two reasons.

Original: First, designing asuitable reward function for complex tasks can be extremely difficult.
Revised: First, designing asuitable reward function for complex tasks can be extremely difficult.

Original: The RL agent learns behaviorsincentivized by the reward function rather than the ones intended, which can be nontrivial to specifyin terms of reward.
Revised: The RL agent learns behaviorsincentivized by the reward function rather than the ones intended, which can be nontrivial to specifyin terms of reward.

Original: Second, the need for online interaction with the environment during the RLtraining loop has hindered its adoption in many real-world domains, where environment interactionsare costly or risky.
Revised: Second, the need for online interaction with the environment during the RLtraining loop has hindered its adoption in many real-world domains, where environment interactionsare costly or risky.

Original: Imitation learning (IL) [1, 33, 34] circumvents the difficulty of reward design in RL by leveragingdemonstrations given by experts, where the goal is to mimic the expert’s behavior.
Revised: Imitation learning (IL) [1, 33, 34] circumvents the difficulty of reward design in RL by leveragingdemonstrations given by experts, where the goal is to mimic the expert’s behavior.

Original: However, thestandard IL requires the expert demonstrations to contain not only the state information (e.g. robotjoint angles) but also the precise action (e.g. robot joint torques) executed by the expert at each timestep.
Revised: However, thestandard IL requires the expert demonstrations to contain not only the state information (e.g. robotjoint angles) but also the precise action (e.g. robot joint torques) executed by the expert at each timestep.

Original: This demand for explicit action labels is in contrast to the way human imitates (e.g. learningby watching videos) and precludes leveraging a massive amount of data in which the action label ismissing.
Revised: This demand for explicit action labels is in contrast to the way human imitates (e.g. learningby watching videos) and precludes leveraging a massive amount of data in which the action label ismissing.

Original: Therefore, developing an imitation learning algorithm that can learn from observing theexperts’ state-only trajectories is a promising direction for creating a practical, autonomous intelligentagent.
Revised: Therefore, developing an imitation learning algorithm that can learn from observing theexperts’ state-only trajectories is a promising direction for creating a practical, autonomous intelligentagent.

Original: Learning from observation (LfO) [2, 14, 26, 39, 45] concerns this particular learning scenarioand has been gaining interest in recent years.
Revised: Learning from observation (LfO) [2, 14, 26, 39, 45] concerns this particular learning scenarioand has been gaining interest in recent years.

Original: In this paper, we are particularly interested in solving the LfO problem in an offline setting: Given thestate-only demonstrations by experts and abundant state-action demonstrations by imperfect agentsof arbitrary levels of optimality, we aim to find a policy that follows the sequence of expert’s states without further interaction with the environment.
Revised: In this paper, we are particularly interested in solving the LfO problem in an offline setting: Given thestate-only demonstrations by experts and abundant state-action demonstrations by imperfect agentsof arbitrary levels of optimality, we aim to find a policy that follows the sequence of expert’s states without further interaction with the environment.

Original: This problem setting is appealing in many practicalsituations where environment interactions are costly or risky.
Revised: This problem setting is appealing in many practicalsituations where environment interactions are costly or risky.

Original: Yet, it cannot be straightforwardlytackled by existing approaches.
Revised: Yet, it cannot be straightforwardlytackled by existing approaches.

Original: Most of the existing LfO methods are on-policy algorithms [25, 35,39, 40], which is not directly applicable to the offline setting.
Revised: Most of the existing LfO methods are on-policy algorithms [25, 35,39, 40], which is not directly applicable to the offline setting.

Original: One of the few exceptions is BCO [38],which performs behavior cloning on the inferred action via an inverse dynamics model.
Revised: One of the few exceptions is BCO [38],which performs behavior cloning on the inferred action via an inverse dynamics model.

Original: This methodcould be suboptimal unless the inverse dynamics disagreement is always zero, i.e. the underlyingenvironment dynamics is deterministic and injective.
Revised: This methodcould be suboptimal unless the inverse dynamics disagreement is always zero, i.e. the underlyingenvironment dynamics is deterministic and injective.

Original: Another one is OPOLO [45], an off-policy LfOalgorithm, but it relies on nested min-max optimization as well as out-of-distribution (OOD) actionvalues, which can be unstable especially in the offline setting.
Revised: Another one is OPOLO [45], an off-policy LfOalgorithm, but it relies on nested min-max optimization as well as out-of-distribution (OOD) actionvalues, which can be unstable especially in the offline setting.

Original: IQ-Learn [8] solves an Inverse RLand can be applied to the offline LfO by learning a state-only reward function.
Revised: IQ-Learn [8] solves an Inverse RLand can be applied to the offline LfO by learning a state-only reward function.

Original: However, it suffersfrom numerical instability due to using OOD action values in the offline setting.
Revised: However, it suffersfrom numerical instability due to using OOD action values in the offline setting.

Original: Lastly, RCE [5], anexample-based control algorithm, can in principle be applied to the offline LfO setting by providingthe expert trajectories as successful example states; however, its empirical performance is known tobe limited without online data collection.
Revised: Lastly, RCE [5], anexample-based control algorithm, can in principle be applied to the offline LfO setting by providingthe expert trajectories as successful example states; however, its empirical performance is known tobe limited without online data collection.

Original: We present an offline LfO algorithm that minimizes the divergence between state-transition distributions induced by the expert and the learned policy, without requiring an inverse dynamicsmodel.
Revised: We present an offline LfO algorithm that minimizes the divergence between state-transition distributions induced by the expert and the learned policy, without requiring an inverse dynamicsmodel.

Original: Our algorithm, offline Learning from OBServation via stationary DIstribution Correction
Revised: Our algorithm, offline Learning from OBServation via stationary DIstribution Correction

Original: Estimation (LobsDICE), essentially optimizes in the space of state-action stationary distributionsand state-transition stationary distributions, rather than in the space of policies.
Revised: Estimation (LobsDICE), essentially optimizes in the space of state-action stationary distributionsand state-transition stationary distributions, rather than in the space of policies.

Original: We show that ourformulation can be reduced to a single convex minimization problem that can be solved efficiently inpractice, unlike existing imitation learning (from observation) algorithms that rely on nested min-maxoptimization.
Revised: We show that ourformulation can be reduced to a single convex minimization problem that can be solved efficiently inpractice, unlike existing imitation learning (from observation) algorithms that rely on nested min-maxoptimization.

Original: In the experiments, we demonstrate that LobsDICE can successfully recover the statevisitations by the expert, outperforming strong baseline methods.
Revised: In the experiments, we demonstrate that LobsDICE can successfully recover the statevisitations by the expert, outperforming strong baseline methods.

Original:  Preliminaries2.1 Markov decision process
Revised:  Preliminaries2.1 Markov decision process

Original: We consider an environment modeled as a Markov Decision Process (MDP), defined by M =⟨ S, A, T, R, p 0 , γ ⟩ [36], where S is the set of states, A is the set of actions, T : S
Revised: We consider an environment modeled as a Markov Decision Process (MDP), defined by M =⟨ S, A, T, R, p 0 , γ ⟩ [36], where S is the set of states, A is the set of actions, T : S

Original: ×
Revised: ×

Original: A → ∆( S ) isthe transition probability, R : S ×
Revised: A → ∆( S ) isthe transition probability, R : S ×

Original: A → R is the reward function, p 0 ∈ ∆( S ) is the distribution of theinitial state, and γ ∈ (0 , 1) is the discount factor.
Revised: A → R is the reward function, p 0 ∈ ∆( S ) is the distribution of theinitial state, and γ ∈ (0 , 1) is the discount factor.

Original: The policy π : S → ∆( A ) is a mapping from statesto distribution over actions.
Revised: The policy π : S → ∆( A ) is a mapping from statesto distribution over actions.

Original: For the given policy π , its state-action stationary distribution d π ( s, a )and state-transition stationary ¯ d π ( s, s ′ ) are defined as:
Revised: For the given policy π , its state-action stationary distribution d π ( s, a )and state-transition stationary ¯ d π ( s, s ′ ) are defined as:

Original: [Equation]
Revised: [Equation]

Original: where s 0 ∼ p 0 , a t ∼ π ( ·| s t ) , and s t
Revised: where s 0 ∼ p 0 , a t ∼ π ( ·| s t ) , and s t

Original: +1 ∼ T ( ·| s t , a t ) for all timesteps t ≥ 0 .
Revised: +1 ∼ T ( ·| s t , a t ) for all timesteps t ≥ 0 .

Original: For brevity, the barnotation ¯( · ) will be used to denote the distributions for ( s, s ′ ) , e.g. ¯ d π ( s, s ′ ) .
Revised: For brevity, the barnotation ¯( · ) will be used to denote the distributions for ( s, s ′ ) , e.g. ¯ d π ( s, s ′ ) .

Original: We assume offline LfO setting, where direct, online interactions with the environment are not allowed,and the policy should be optimized solely from a pre-collected dataset.
Revised: We assume offline LfO setting, where direct, online interactions with the environment are not allowed,and the policy should be optimized solely from a pre-collected dataset.

Original: We denote the datasetof state-only demonstrations collected by experts as D E
Revised: We denote the datasetof state-only demonstrations collected by experts as D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: i } N E i =1 and the dataset of stateaction demonstrations by some imperfect agents as D I = { ( s, a, s ′ )
Revised: i } N E i =1 and the dataset of stateaction demonstrations by some imperfect agents as D I = { ( s, a, s ′ )

Original: i } N I
Revised: i } N I

Original: i =1 .
Revised: i =1 .

Original: That is, we do not haveinformation about the actions taken by the expert, but instead, we have additional action-labeledtransition data collected by some other agents with unknown levels of optimality.
Revised: That is, we do not haveinformation about the actions taken by the expert, but instead, we have additional action-labeledtransition data collected by some other agents with unknown levels of optimality.

Original: We denote thecorresponding distributions of the datasets D E and D I by ¯ d E and d I , respectively.
Revised: We denote thecorresponding distributions of the datasets D E and D I by ¯ d E and d I , respectively.

Original: For brevity, wewill abuse notation d I to represent ( s, a ) ∼ d
Revised: For brevity, wewill abuse notation d I to represent ( s, a ) ∼ d

Original: I , ( s, a, s ′ ) ∼ d I , and ( s, s ′ )
Revised: I , ( s, a, s ′ ) ∼ d I , and ( s, s ′ )

Original: ∼ ¯ d I unless ambiguous.
Revised: ∼ ¯ d I unless ambiguous.

Original: 2.2 Imitation learning and learning from observation
Revised: 2.2 Imitation learning and learning from observation

Original: Imitation learning (IL) aims to mimic the expert policy from its state-action demonstrations.
Revised: Imitation learning (IL) aims to mimic the expert policy from its state-action demonstrations.

Original: IL canbe naturally formulated as a distribution matching problem that minimizes the divergence betweenstate-action stationary distributions induced by the expert and the target policy [11, 13].
Revised: IL canbe naturally formulated as a distribution matching problem that minimizes the divergence betweenstate-action stationary distributions induced by the expert and the target policy [11, 13].

Original: For example,one can consider minimizing KL-divergence [17]:
Revised: For example,one can consider minimizing KL-divergence [17]:

Original: [Equation]
Revised: [Equation]

Original: However, the standard IL requires action labels in the expert demonstrations, which may be a toostrong requirement for various practical situations.
Revised: However, the standard IL requires action labels in the expert demonstrations, which may be a toostrong requirement for various practical situations.

Original: Learning from observation (LfO) relaxes the requirement on action labels, and aims to imitate the expert’s behavior only from the state observations.
Revised: Learning from observation (LfO) relaxes the requirement on action labels, and aims to imitate the expert’s behavior only from the state observations.

Original: Since the expert’s action information is missing in the demonstrations, the distribution matching forstate-action stationary distribution is no longer readily applicable.
Revised: Since the expert’s action information is missing in the demonstrations, the distribution matching forstate-action stationary distribution is no longer readily applicable.

Original: Therefore, LfO is reformulatedas another distribution matching problem that minimizes the divergence between state-transitionstationary distributions induced by the expert and the target policy [39, 40, 45] 1 :
Revised: Therefore, LfO is reformulatedas another distribution matching problem that minimizes the divergence between state-transitionstationary distributions induced by the expert and the target policy [39, 40, 45] 1 :

Original: [Equation]
Revised: [Equation]

Original: Still, optimizing Eq.
Revised: Still, optimizing Eq.

Original: (2) in a purely offline manner is challenging since naively estimating theexpectation would require the knowledge of T ( ·| s, a ) for the OOD action a ∼ π ( s ) by the targetpolicy, which is inaccessible in the offline LfO setting; see Section 9.8 in [45] for more discussions.
Revised: (2) in a purely offline manner is challenging since naively estimating theexpectation would require the knowledge of T ( ·| s, a ) for the OOD action a ∼ π ( s ) by the targetpolicy, which is inaccessible in the offline LfO setting; see Section 9.8 in [45] for more discussions.

Original: OPOLO [45], an off-policy LfO algorithm, mitigates this challenge by minimizing the followingupper bound of the divergence D KL (
Revised: OPOLO [45], an off-policy LfO algorithm, mitigates this challenge by minimizing the followingupper bound of the divergence D KL (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and applying DualDICE trick [29] to the RHS of (3).
Revised: and applying DualDICE trick [29] to the RHS of (3).

Original: The upper bound gap is given by the inversedynamics disagreement between the target policy and the imperfect demonstrator:
Revised: The upper bound gap is given by the inversedynamics disagreement between the target policy and the imperfect demonstrator:

Original: [Equation]
Revised: [Equation]

Original: which usually gets larger as the stochasticity of the environment transition increases.
Revised: which usually gets larger as the stochasticity of the environment transition increases.

Original:  LobsDICE
Revised:  LobsDICE

Original: We present offline Learning from OBServation via stationary DIstribution Correction Estimation(LobsDICE), a principled offline LfO algorithm that further extends the recent progress made by the
Revised: We present offline Learning from OBServation via stationary DIstribution Correction Estimation(LobsDICE), a principled offline LfO algorithm that further extends the recent progress made by the

Original: DIstribution Correction Estimation (DICE) methods for offline RL.
Revised: DIstribution Correction Estimation (DICE) methods for offline RL.

Original: LobsDICE essentially optimizesthe stationary distributions of the target policy to match the expert’s state visitations.
Revised: LobsDICE essentially optimizesthe stationary distributions of the target policy to match the expert’s state visitations.

Original: In the context of offline RL, the policy constraint principle (i.e. prevent deviating too much from thedata support) is one of the common approaches to avoid severe performance degradation [7, 12, 18,22, 30].
Revised: In the context of offline RL, the policy constraint principle (i.e. prevent deviating too much from thedata support) is one of the common approaches to avoid severe performance degradation [7, 12, 18,22, 30].

Original: In the same manner, we use KL divergence minimization between ¯ d π ( s, s ′ ) and ¯ d E ( s, s ′ )with additional KL regularization on the deviation from d I :
Revised: In the same manner, we use KL divergence minimization between ¯ d π ( s, s ′ ) and ¯ d E ( s, s ′ )with additional KL regularization on the deviation from d I :

Original: [Equation]
Revised: [Equation]

Original: Here the hyperparameter α > 0 balances between encouraging state-transition matching and preventing distribution shift from the distribution of imperfect demonstrations.
Revised: Here the hyperparameter α > 0 balances between encouraging state-transition matching and preventing distribution shift from the distribution of imperfect demonstrations.

Original: All the proofs can be foundin Appendix F.
Revised: All the proofs can be foundin Appendix F.

Original: 3.1 Lagrange dual formulation
Revised: 3.1 Lagrange dual formulation

Original: The derivation of our algorithm starts by rewriting the (regularized) distribution matching problem (5)in terms of directly optimizing stationary distribution, rather than policy:
Revised: The derivation of our algorithm starts by rewriting the (regularized) distribution matching problem (5)in terms of directly optimizing stationary distribution, rather than policy:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The Bellman flow constraint (7) ensures d ( s, a ) to be a valid state-action stationary distribution ofsome policy, where d ( s, a ) can be interpreted as a normalized occupancy measure of ( s, a ) .
Revised: The Bellman flow constraint (7) ensures d ( s, a ) to be a valid state-action stationary distribution ofsome policy, where d ( s, a ) can be interpreted as a normalized occupancy measure of ( s, a ) .

Original: Themarginalization constraint (8) enforces ¯ d ( s, s ′ ) to be the state-transition stationary distribution thatis directly induced by d ( s, a ) .
Revised: Themarginalization constraint (8) enforces ¯ d ( s, s ′ ) to be the state-transition stationary distribution thatis directly induced by d ( s, a ) .

Original: In essence, the constrained optimization problem (6-8) seeks thestationary distributions of an optimal policy, which best matches the state-transition trajectories ofthe expert.
Revised: In essence, the constrained optimization problem (6-8) seeks thestationary distributions of an optimal policy, which best matches the state-transition trajectories ofthe expert.

Original: Once we have computed the optimal solution ( d ∗ , ¯ d ∗ ) , its corresponding optimal policycan also be obtained by normalizing d ∗ for each state [32]: π
Revised: Once we have computed the optimal solution ( d ∗ , ¯ d ∗ ) , its corresponding optimal policycan also be obtained by normalizing d ∗ for each state [32]: π

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s )
Revised: s )

Original: [Equation]
Revised: [Equation]

Original: Note that DemoDICE [15] considers a similar optimization problem to ours, but it deals with theoffline IL (i.e. state-action stationary distribution matching), whereas we consider the offline LfO(e.g state-transition stationary distribution matching).
Revised: Note that DemoDICE [15] considers a similar optimization problem to ours, but it deals with theoffline IL (i.e. state-action stationary distribution matching), whereas we consider the offline LfO(e.g state-transition stationary distribution matching).

Original: Accordingly, the optimization variable ¯ d ( s, s ′ )and the marginalization constraint (8) are newly added in our formulation.
Revised: Accordingly, the optimization variable ¯ d ( s, s ′ )and the marginalization constraint (8) are newly added in our formulation.

Original: We then consider the Lagrangian for the constrained optimization (6-8):
Revised: We then consider the Lagrangian for the constrained optimization (6-8):

Original: [Equation]
Revised: [Equation]

Original: where ν ( s ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (7), and µ ( s, s ′ )
Revised: where ν ( s ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (7), and µ ( s, s ′ )

Original: ∈
Revised: ∈

Original: Rare the the Lagrange multipliers for the marginalization constraint (8).
Revised: Rare the the Lagrange multipliers for the marginalization constraint (8).

Original: Note that the Lagrangian(9) cannot be naively optimized in an offline manner since it requires evaluation of T ( s ′ | s, a ) for( s, a ) ∼ d , which is not accessible in the offline LfO setting.
Revised: Note that the Lagrangian(9) cannot be naively optimized in an offline manner since it requires evaluation of T ( s ′ | s, a ) for( s, a ) ∼ d , which is not accessible in the offline LfO setting.

Original: Therefore, we rearrange the terms in (9)to eliminate the direct dependence on d and ¯ d , introducing new optimization variables w and ¯ w thatdenote stationary distribution correction ratios for ( s, a ) and ( s, s ′ ) , respectively:
Revised: Therefore, we rearrange the terms in (9)to eliminate the direct dependence on d and ¯ d , introducing new optimization variables w and ¯ w thatdenote stationary distribution correction ratios for ( s, a ) and ( s, s ′ ) , respectively:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similar to the assumption of full coverage which is fairly standard across a broad set of recent offline
Revised: Similar to the assumption of full coverage which is fairly standard across a broad set of recent offline

Original: RL approaches [19, 22, 27], we make a milder assumption that ¯ d I ( s, s ′ ) > 0 whenever ¯ d E ( s, s ′ ) > 0 .
Revised: RL approaches [19, 22, 27], we make a milder assumption that ¯ d I ( s, s ′ ) > 0 whenever ¯ d E ( s, s ′ ) > 0 .

Original: This assumption is necessary to recover the expert’s behavior successfully.
Revised: This assumption is necessary to recover the expert’s behavior successfully.

Original: We introduce the log ratio r ( s, s ′ )
Revised: We introduce the log ratio r ( s, s ′ )

Original: =
Revised: =

Original: log ¯ d E ( s,s ′ )
Revised: log ¯ d E ( s,s ′ )

Original: ¯ d I ( s,s ′ ) in (10) to take an expectation under ¯ d I (insteadof ¯ d E ), which is assumed to have a broader support than ¯ d E .
Revised: ¯ d I ( s,s ′ ) in (10) to take an expectation under ¯ d I (insteadof ¯ d E ), which is assumed to have a broader support than ¯ d E .

Original: This log ratio r ( s, s ′ ) can be easilyestimated using a pretrained discriminator for two datasets D E and D I , which will be explained indetail in the following section.
Revised: This log ratio r ( s, s ′ ) can be easilyestimated using a pretrained discriminator for two datasets D E and D I , which will be explained indetail in the following section.

Original: In summary, LobsDICE aims to solve the minimax optimization,
Revised: In summary, LobsDICE aims to solve the minimax optimization,

Original: [Equation]
Revised: [Equation]

Original: The optimal solution ( w ∗ , ¯ w ∗ ) of (12) represents stationary distribution corrections of an optimalpolicy π ∗ : w ∗ ( s, a )
Revised: The optimal solution ( w ∗ , ¯ w ∗ ) of (12) represents stationary distribution corrections of an optimalpolicy π ∗ : w ∗ ( s, a )

Original: =
Revised: =

Original: d π
Revised: d π

Original: [Equation]
Revised: [Equation]

Original: ( s,a ) d
Revised: ( s,a ) d

Original: I ( s,a ) and ¯ w ∗ ( s, s ′ ) =
Revised: I ( s,a ) and ¯ w ∗ ( s, s ′ ) =

Original: ¯ d
Revised: ¯ d

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ) ¯ d I ( s,s ′ ) .
Revised: ) ¯ d I ( s,s ′ ) .

Original: 3.2 Log ratio estimation via a pretrained discriminator
Revised: 3.2 Log ratio estimation via a pretrained discriminator

Original: To optimize (11), an estimate of the log ratio r ( s, s ′ )
Revised: To optimize (11), an estimate of the log ratio r ( s, s ′ )

Original: =
Revised: =

Original: log ¯ d E ( s,s ′ )
Revised: log ¯ d E ( s,s ′ )

Original: ¯ d I ( s,s ′ ) is required.
Revised: ¯ d I ( s,s ′ ) is required.

Original: The log ratioestimation is straightforward for tabular MDPs since we can use empirical distributions from thedatasets to estimate ¯ d E ( s, s ′ ) and ¯ d I ( s, s ′ ) .
Revised: The log ratioestimation is straightforward for tabular MDPs since we can use empirical distributions from thedatasets to estimate ¯ d E ( s, s ′ ) and ¯ d I ( s, s ′ ) .

Original: For continuous MDPs, we train a discriminator c :
Revised: For continuous MDPs, we train a discriminator c :

Original: S × S → [0 , 1] by solving the following maximization problem [9, 45]:
Revised: S × S → [0 , 1] by solving the following maximization problem [9, 45]:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: It is easy to show that the optimal discriminator satisfies c ∗
Revised: It is easy to show that the optimal discriminator satisfies c ∗

Original: [Equation]
Revised: [Equation]

Original: s, s ′
Revised: s, s ′

Original: [Equation]
Revised: [Equation]

Original: Thus, r ( s, s ′ )can be derived from the optimal discriminator c ∗ as,
Revised: Thus, r ( s, s ′ )can be derived from the optimal discriminator c ∗ as,

Original: [Equation]
Revised: [Equation]

Original: 3.3 Minimax to min: a closed-form solution
Revised: 3.3 Minimax to min: a closed-form solution

Original: Exploiting the strict convexity of x log x , we can derive a closed-form solution to the inner maximization for ( w, ¯ w ) in (11).
Revised: Exploiting the strict convexity of x log x , we can derive a closed-form solution to the inner maximization for ( w, ¯ w ) in (11).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.( w µ,ν , ¯ w µ ) =
Revised: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.( w µ,ν , ¯ w µ ) =

Original: arg max w, ¯ w ≥ 

Revised: arg max w, ¯ w ≥

Original: L ( w, ¯ w, µ, ν ) , is given by:
Revised: L ( w, ¯ w, µ, ν ) , is given by:

Original: [Equation]
Revised: [Equation]

Original: Based on the above result, we reduce the nested min-max optimization of (11) to a single minimizationby plugging the closed-form solution ( w µ,ν , ¯ w µ ) into L ( w, ¯ w, µ, ν ) :
Revised: Based on the above result, we reduce the nested min-max optimization of (11) to a single minimizationby plugging the closed-form solution ( w µ,ν , ¯ w µ ) into L ( w, ¯ w, µ, ν ) :

Original: [Equation]
Revised: [Equation]

Original: We can even show that L ( w µ,ν , ¯ w µ , µ, ν ) is a convex function of µ and ν .
Revised: We can even show that L ( w µ,ν , ¯ w µ , µ, ν ) is a convex function of µ and ν .

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .
Revised: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .

Original: In short, by operating in the space of stationary distributions, offline LfO can, in principle, be resolvedby solving a convex minimization problem.
Revised: In short, by operating in the space of stationary distributions, offline LfO can, in principle, be resolvedby solving a convex minimization problem.

Original: This is in contrast to the existing LfO algorithms, whichtypically involve either an adversarial training that optimizes the policy and the discriminator [14, 39,40] or a nested min-max optimization for the policy and the critic [45].
Revised: This is in contrast to the existing LfO algorithms, whichtypically involve either an adversarial training that optimizes the policy and the discriminator [14, 39,40] or a nested min-max optimization for the policy and the critic [45].

Original: 3.4 Policy extraction
Revised: 3.4 Policy extraction

Original: So far, we have derived an algorithm that essentially solves the state-transition distribution matchingproblem via convex minimization.
Revised: So far, we have derived an algorithm that essentially solves the state-transition distribution matchingproblem via convex minimization.

Original: However, we obtain ( µ ∗ , ν ∗ ) as the solution of (16), instead of theoptimal policy π ∗ .
Revised: However, we obtain ( µ ∗ , ν ∗ ) as the solution of (16), instead of theoptimal policy π ∗ .

Original: The remaining problem is to extract the optimal policy from ( µ ∗ , ν ∗ ) .
Revised: The remaining problem is to extract the optimal policy from ( µ ∗ , ν ∗ ) .

Original: The first stepis to see that we can obtain the state-action stationary distribution correction w µ ∗ ,ν ∗ ( s, a ) =
Revised: The first stepis to see that we can obtain the state-action stationary distribution correction w µ ∗ ,ν ∗ ( s, a ) =

Original: d π ∗ ( s,a ) d
Revised: d π ∗ ( s,a ) d

Original: I ( s,a )of the optimal policy from the closed-form solution (15).
Revised: I ( s,a )of the optimal policy from the closed-form solution (15).

Original: Among many possibilities to extract thepolicy from the state-action distribution correction, we adopt weighted behavior cloning (WBC):
Revised: Among many possibilities to extract thepolicy from the state-action distribution correction, we adopt weighted behavior cloning (WBC):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which aims to maximize the predicted probability of actions chosen by optimal policy π ∗
Revised: which aims to maximize the predicted probability of actions chosen by optimal policy π ∗

Original: This isdone by BC on the offline dataset D I where each sample ( s, a ) is weighted by w µ ∗ ,ν ∗ ( s, a ) .
Revised: This isdone by BC on the offline dataset D I where each sample ( s, a ) is weighted by w µ ∗ ,ν ∗ ( s, a ) .

Original: Fortabular MDPs, we can formally show that WBC extracts an optimal policy π ∗ (Appendix C).
Revised: Fortabular MDPs, we can formally show that WBC extracts an optimal policy π ∗ (Appendix C).

Original: 3.5 Practical algorithm with sample-based approximation
Revised: 3.5 Practical algorithm with sample-based approximation

Original: In practice, we estimate L ( w, ¯ w, µ, ν ) in (11) using samples from distribution d I .
Revised: In practice, we estimate L ( w, ¯ w, µ, ν ) in (11) using samples from distribution d I .

Original: Let ˆE x ∈ D
Revised: Let ˆE x ∈ D

Original: [ f ( x )] :=1 | D | (cid:80) x ∈ D f ( x ) be a Monte-Carlo estimate of E x ∼ p [ f ( x )] where D = { x i } | D | i =1 ∼ p .
Revised: [ f ( x )] :=1 | D | (cid:80) x ∈ D f ( x ) be a Monte-Carlo estimate of E x ∼ p [ f ( x )] where D = { x i } | D | i =1 ∼ p .

Original: We denoteeach sample ( s, a, s ′ ) in D I as x for brevity.
Revised: We denoteeach sample ( s, a, s ′ ) in D I as x for brevity.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˆ e µ,ν ( s, a, s ′ ) :
Revised: where ˆ e µ,ν ( s, a, s ′ ) :

Original: = − µ ( s, s ′ )
Revised: = − µ ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) is a single-sample estimate of e µ,ν ( s, a ) .
Revised: − ν ( s ) is a single-sample estimate of e µ,ν ( s, a ) .

Original: Notethat this sample-based objective function (cid:98) L ( w, ¯ w, µ, ν ) can be estimated only from samples in theoffline dataset D I and is an unbiased estimator of L ( w, ¯ w, µ, ν ) as long as every sample in D I wascollected by interacting with the underlying MDP.
Revised: Notethat this sample-based objective function (cid:98) L ( w, ¯ w, µ, ν ) can be estimated only from samples in theoffline dataset D I and is an unbiased estimator of L ( w, ¯ w, µ, ν ) as long as every sample in D I wascollected by interacting with the underlying MDP.

Original: To the best of our knowledge, this is the firstresult to directly solve the state-transition distribution matching problem in a fully offline manner.
Revised: To the best of our knowledge, this is the firstresult to directly solve the state-transition distribution matching problem in a fully offline manner.

Original: Incontrast, OPOLO [45] relies on the (potentially loose) upper bound of the objective in (3).
Revised: Incontrast, OPOLO [45] relies on the (potentially loose) upper bound of the objective in (3).

Original: To reduce the minimax optimization to a single minimization, we apply the non-parametric closedform solution for each sample x
Revised: To reduce the minimax optimization to a single minimization, we apply the non-parametric closedform solution for each sample x

Original: = ( s, a, s ′ ) in D I :(cid:98)
Revised: = ( s, a, s ′ ) in D I :(cid:98)

Original: w µ,ν ( x ) = exp (cid:16) 1 α ˆ e µ,ν ( s, a, s ′ ) − 1 (cid:17) and (cid:98)
Revised: w µ,ν ( x ) = exp (cid:16) 1 α ˆ e µ,ν ( s, a, s ′ ) − 1 (cid:17) and (cid:98)

Original: ¯ w µ ( x ) =
Revised: ¯ w µ ( x ) =

Original: exp( r ( s, s ′ )
Revised: exp( r ( s, s ′ )

Original: + µ ( s, s ′ )
Revised: + µ ( s, s ′ )

Original: − 1) .
Revised: − 1) .

Original: which is analogous to (15).
Revised: which is analogous to (15).

Original: Plugging this result into (18) yields a sample-based objective function forminimization 2 :
Revised: Plugging this result into (18) yields a sample-based objective function forminimization 2 :

Original: [Equation]
Revised: [Equation]

Original: Still, the variable µ ∈ R S × S is much higher dimensional than ν ∈ R S .
Revised: Still, the variable µ ∈ R S × S is much higher dimensional than ν ∈ R S .

Original: So, it works as the mainbottleneck for the overall optimization.
Revised: So, it works as the mainbottleneck for the overall optimization.

Original: Fortunately, we can further simplify (20) by eliminating itsdependence on µ via exploiting an additional closed-form solution.
Revised: Fortunately, we can further simplify (20) by eliminating itsdependence on µ via exploiting an additional closed-form solution.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e.µ ν = arg min µ
Revised: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e.µ ν = arg min µ

Original: [Equation]
Revised: [Equation]

Original: L ( µ, ν ) , is
Revised: L ( µ, ν ) , is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the above result in (20), we obtain the following minimization problem:min
Revised: Using the above result in (20), we obtain the following minimization problem:min

Original: [Equation]
Revised: [Equation]

Original: (22) where (cid:98)
Revised: (22) where (cid:98)

Original: A ν ( s, a, s ′ ) :
Revised: A ν ( s, a, s ′ ) :

Original: = r ( s, s ′ )
Revised: = r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) .
Revised: − ν ( s ) .

Original: The remaining issue is that optimizing (22) is notpractical because exp( · ) often causes numerical instability and gradient explosion.
Revised: The remaining issue is that optimizing (22) is notpractical because exp( · ) often causes numerical instability and gradient explosion.

Original: To address this,we use a numerically-stable alternative of (22):
Revised: To address this,we use a numerically-stable alternative of (22):

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: Let (cid:101) L ( (cid:101) ν ) be the function:
Revised: Let (cid:101) L ( (cid:101) ν ) be the function:

Original: [Equation]
Revised: [Equation]

Original: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )
Revised: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )

Original: = min (cid:101) ν (cid:101)
Revised: = min (cid:101) ν (cid:101)

Original: L ( (cid:101) ν ) holds.
Revised: L ( (cid:101) ν ) holds.

Original: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .
Revised: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .

Original: In order see why minimizing (cid:101) L ( (cid:101) ν ) no longer suffers from numerical instability, note that the gradient∇ x log E x ∼ p [exp( h ( x ))]
Revised: In order see why minimizing (cid:101) L ( (cid:101) ν ) no longer suffers from numerical instability, note that the gradient∇ x log E x ∼ p [exp( h ( x ))]

Original: =
Revised: =

Original: E x ∼ p [
Revised: E x ∼ p [

Original: [Equation]
Revised: [Equation]

Original: ] ∇ x h ( x )] normalizes exp( · ) by softmax and thustames large numerical values.
Revised: ] ∇ x h ( x )] normalizes exp( · ) by softmax and thustames large numerical values.

Original: Finally, we can show the following connection between (cid:98)
Revised: Finally, we can show the following connection between (cid:98)

Original: ν ∗ and (cid:101)
Revised: ν ∗ and (cid:101)

Original: ν ∗ :
Revised: ν ∗ :

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: Let (cid:98) V and (cid:101)
Revised: Let (cid:98) V and (cid:101)

Original: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.
Revised: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.

Original: Then,(cid:101)
Revised: Then,(cid:101)

Original: V = { (cid:98) ν
Revised: V = { (cid:98) ν

Original: ∗ + C | (cid:98) ν ∗ ∈ (cid:98) V , C ∈ R } holds.
Revised: ∗ + C | (cid:98) ν ∗ ∈ (cid:98) V , C ∈ R } holds.

Original: Proposition 3.5 implies that an unnormalized stationary distribution corrections of an optimal policywould be obtained from (cid:101)
Revised: Proposition 3.5 implies that an unnormalized stationary distribution corrections of an optimal policywould be obtained from (cid:101)

Original: ν ∗ ∈ (cid:101) V .
Revised: ν ∗ ∈ (cid:101) V .

Original: [Equation]
Revised: [Equation]

Original: Policy extraction We must take caution when using (cid:101) w (cid:101) ν ∗ ( x ) since it is an unnormalized densityratio, i.e. E x ∼ d I [ (cid:101) w (cid:101)
Revised: Policy extraction We must take caution when using (cid:101) w (cid:101) ν ∗ ( x ) since it is an unnormalized densityratio, i.e. E x ∼ d I [ (cid:101) w (cid:101)

Original: ν ∗ ( x )]
Revised: ν ∗ ( x )]

Original: ̸ = 1 .
Revised: ̸ = 1 .

Original: Therefore, to extract a policy, we perform weighted BC usingself-normalized importance sampling [31]:
Revised: Therefore, to extract a policy, we perform weighted BC usingself-normalized importance sampling [31]:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which completes the derivation of the practical version of LobsDICE.
Revised: which completes the derivation of the practical version of LobsDICE.

Original: To sum up, LobsDICE solves(cid:101)
Revised: To sum up, LobsDICE solves(cid:101)

Original: ν ∗ = arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) of (23) via gradient descent, and extracts a policy via self-normalized weighted
Revised: ν ∗ = arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) of (23) via gradient descent, and extracts a policy via self-normalized weighted

Original: BC of (24).
Revised: BC of (24).

Original: Pseudocode for LobsDICE can be found in Appendix G.2.
Revised: Pseudocode for LobsDICE can be found in Appendix G.2.

Original: L ( µ, ν ) in (20) is an upper bound of max w, ¯ w (cid:98) L ( w, ¯ w, µ, ν ) in (18), due to applying per-sample closed-form solutions.
Revised: L ( µ, ν ) in (20) is an upper bound of max w, ¯ w (cid:98) L ( w, ¯ w, µ, ν ) in (18), due to applying per-sample closed-form solutions.

Original: While unbiasedness has been sacrificed at the cost of eliminating the nested optimization, it enables much stable optimization in practice.
Revised: While unbiasedness has been sacrificed at the cost of eliminating the nested optimization, it enables much stable optimization in practice.

Original: The upper bound gap vanishes when the transition dynamics are deterministic.
Revised: The upper bound gap vanishes when the transition dynamics are deterministic.

Original:  Related Work
Revised:  Related Work

Original: Learning from observation (LfO)
Revised: Learning from observation (LfO)

Original: Recent approaches for LfO are mostly on-policy [25, 26, 35,39, 40] algorithms and are not directly applicable to the offline LfO setting considered in this work.
Revised: Recent approaches for LfO are mostly on-policy [25, 26, 35,39, 40] algorithms and are not directly applicable to the offline LfO setting considered in this work.

Original: MobILE [14] is a model-based LfO algorithm, but it encourages uncertainty for online exploration,which is not suitable for the offline setting.
Revised: MobILE [14] is a model-based LfO algorithm, but it encourages uncertainty for online exploration,which is not suitable for the offline setting.

Original: BCO [38] uses an inverse dynamics model (IDM) to inferthe missing expert actions and performs BC on the generated expert’s state-action dataset.
Revised: BCO [38] uses an inverse dynamics model (IDM) to inferthe missing expert actions and performs BC on the generated expert’s state-action dataset.

Original: In additionto common issues by vanilla BC, BCO is not guaranteed to recover the expert’s behavior in general.
Revised: In additionto common issues by vanilla BC, BCO is not guaranteed to recover the expert’s behavior in general.

Original: OPOLO [45] is a principled off-policy LfO algorithm, but it solves a nested optimization and requiresevaluation on OOD action values during training, which suffers from numerical instability in theoffline setting.
Revised: OPOLO [45] is a principled off-policy LfO algorithm, but it solves a nested optimization and requiresevaluation on OOD action values during training, which suffers from numerical instability in theoffline setting.

Original: IQ-Learn [8] solves online and offline IL problems while avoiding adversarial trainingby learning a single Q-function.
Revised: IQ-Learn [8] solves online and offline IL problems while avoiding adversarial trainingby learning a single Q-function.

Original: However, it also suffers from numerical instability in the offlinesetting by overestimating Q due to using OOD action values.
Revised: However, it also suffers from numerical instability in the offlinesetting by overestimating Q due to using OOD action values.

Original: RCE [5] aims to solve example-basedcontrol tasks in which a collection of example success states is assumed to be provided instead ofentire expert trajectories.
Revised: RCE [5] aims to solve example-basedcontrol tasks in which a collection of example success states is assumed to be provided instead ofentire expert trajectories.

Original: While RCE can be applied to LfO in principle, it tends to stay in a fewexpert states that are easy to reach in the offline setting, as discussed in Section C.2 of [5].
Revised: While RCE can be applied to LfO in principle, it tends to stay in a fewexpert states that are easy to reach in the offline setting, as discussed in Section C.2 of [5].

Original: Stationary DIstribution Correction Estimation (DICE) DICE-family algorithms perform stationary distribution estimation, and many of them have been proposed for off-policy evaluation [4, 29, 42–44 ].
Revised: Stationary DIstribution Correction Estimation (DICE) DICE-family algorithms perform stationary distribution estimation, and many of them have been proposed for off-policy evaluation [4, 29, 42–44 ].

Original: Other lines of works consider reinforcement learning [22, 23, 30], offline policy selection [41].
Revised: Other lines of works consider reinforcement learning [22, 23, 30], offline policy selection [41].

Original: ValueDICE [17] and OPOLO [45] derive off-policy IL and LfO objectives using DICE.
Revised: ValueDICE [17] and OPOLO [45] derive off-policy IL and LfO objectives using DICE.

Original: However, theysuffer from numerical instability in the offline setting due to the nested min-max optimization and
Revised: However, theysuffer from numerical instability in the offline setting due to the nested min-max optimization and

Original: OOD action evaluation.
Revised: OOD action evaluation.

Original: DemoDICE [15] is an offline IL algorithm that directly optimizes stationarydistribution as ours and reduces to solving a convex minimization.
Revised: DemoDICE [15] is an offline IL algorithm that directly optimizes stationarydistribution as ours and reduces to solving a convex minimization.

Original: Yet, it requires expert actionlabels and is not directly applicable to LfO. Concurrent to our work, SMODICE [28] is an offline
Revised: Yet, it requires expert actionlabels and is not directly applicable to LfO. Concurrent to our work, SMODICE [28] is an offline

Original: LfO method, aiming to match stationary state distributions by minimizing an objective similar to
Revised: LfO method, aiming to match stationary state distributions by minimizing an objective similar to

Original: OPOLO [45] in (3).
Revised: OPOLO [45] in (3).

Original: Thus it also relies on potentially loose upper bound of the divergence.
Revised: Thus it also relies on potentially loose upper bound of the divergence.

Original: In addition,matching the state distributions may not be sufficient to recover the expert’s behavior, on which weprovide detailed discussion in Appendix A and Remark B.1.
Revised: In addition,matching the state distributions may not be sufficient to recover the expert’s behavior, on which weprovide detailed discussion in Appendix A and Remark B.1.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate LobsDICE on both tabular and continuous MDPs.
Revised: In this section, we evaluate LobsDICE on both tabular and continuous MDPs.

Original: We use four baselinemethods in tabular MDPs: BC on imperfect demonstrations, BCO [38], and OPOLO [45].
Revised: We use four baselinemethods in tabular MDPs: BC on imperfect demonstrations, BCO [38], and OPOLO [45].

Original: Additionally, we designed a strong baseline DemoDICEfO, which extends the state-of-the-art offline
Revised: Additionally, we designed a strong baseline DemoDICEfO, which extends the state-of-the-art offline

Original: IL algorithm, DemoDICE [15].
Revised: IL algorithm, DemoDICE [15].

Original: DemoDICEfO trains an inverse dynamics model, uses it to fill themissing actions in the expert demonstrations, and then runs DemoDICE using the approximate expertdemonstrations and the imperfect demonstrations.
Revised: DemoDICEfO trains an inverse dynamics model, uses it to fill themissing actions in the expert demonstrations, and then runs DemoDICE using the approximate expertdemonstrations and the imperfect demonstrations.

Original: For continuous control tasks, we use two additionalbaselines: IQ-Learn [8] and RCE [5].
Revised: For continuous control tasks, we use two additionalbaselines: IQ-Learn [8] and RCE [5].

Original: 5.1 Random MDPs
Revised: 5.1 Random MDPs

Original: We first evaluate LobsDICE and baseline algorithms on randomly generated finite MDPs using avarying number of expert/imperfect trajectories and different degrees of stochasticity in the environment .
Revised: We first evaluate LobsDICE and baseline algorithms on randomly generated finite MDPs using avarying number of expert/imperfect trajectories and different degrees of stochasticity in the environment .

Original: We follow the experimental protocol in previous offline RL works [20–22] but with additionalcontrol on the stochasticity of transition probabilities.
Revised: We follow the experimental protocol in previous offline RL works [20–22] but with additionalcontrol on the stochasticity of transition probabilities.

Original: We conduct repeated experiments for 1K runs.
Revised: We conduct repeated experiments for 1K runs.

Original: For each run, (1) a random MDP is generated, (2) expert trajectories and imperfect trajectories arecollected, and (3) each offline LfO algorithm is tested on the collected offline dataset.
Revised: For each run, (1) a random MDP is generated, (2) expert trajectories and imperfect trajectories arecollected, and (3) each offline LfO algorithm is tested on the collected offline dataset.

Original: We evaluatethe performance of each algorithm by measuring the total variation distance between state-transitionstationary distributions by the expert policy and the learned policy, i.e. D TV ( ¯ d π ( s, s ′ )
Revised: We evaluatethe performance of each algorithm by measuring the total variation distance between state-transitionstationary distributions by the expert policy and the learned policy, i.e. D TV ( ¯ d π ( s, s ′ )

Original: ∥ ¯ d π E ( s, s ′ )) .
Revised: ∥ ¯ d π E ( s, s ′ )) .

Original: For the tabular MDP experiments, we adopt tabular methods but not function approximation.
Revised: For the tabular MDP experiments, we adopt tabular methods but not function approximation.

Original: Ourtabular LobsDICE optimizes (16) on the empirical MDP model constructed from the action-labeleddataset D I while extracting the policy through (17).
Revised: Ourtabular LobsDICE optimizes (16) on the empirical MDP model constructed from the action-labeleddataset D I while extracting the policy through (17).

Original: The pseudocode for tabular LobsDICE can befound in Appendix G.1.
Revised: The pseudocode for tabular LobsDICE can befound in Appendix G.1.

Original: The detailed experimental setup such as random MDP generation and offlinedataset generation can be found in Appendix I.1.
Revised: The detailed experimental setup such as random MDP generation and offlinedataset generation can be found in Appendix I.1.

Original: Results Figure 1 presents the results in random MDP experiments, where β is the hyperparameterthat controls the stochasticity of the underlying MDP.
Revised: Results Figure 1 presents the results in random MDP experiments, where β is the hyperparameterthat controls the stochasticity of the underlying MDP.

Original: The first row corresponds to the case whenβ = 0 .
Revised: The first row corresponds to the case whenβ = 0 .

Original: 01 (nearly deterministic MDP).
Revised: 01 (nearly deterministic MDP).

Original: In this situation, the inverse dynamics disagreement will beclose to zero, i.e. D KL ( d π 1 ( a | s, s ′ )
Revised: In this situation, the inverse dynamics disagreement will beclose to zero, i.e. D KL ( d π 1 ( a | s, s ′ )

Original: ||
Revised: ||

Original: d π 2 ( a | s, s ′ ))
Revised: d π 2 ( a | s, s ′ ))

Original: ≈ 0 for any two policies π 1 and π 2 .
Revised: ≈ 0 for any two policies π 1 and π 2 .

Original: Thus, thealgorithms whose performance directly relies on the IDM’s accuracy (i.e. BCO and DemoDICEfO)
Revised: Thus, thealgorithms whose performance directly relies on the IDM’s accuracy (i.e. BCO and DemoDICEfO)

Original: [Figure]
Revised: [Figure]

Original: OPOLO
Revised: OPOLO

Original: (ours)
Revised: (ours)

Original: , even goes even perform very well since it is very easy to learn a perfect inverse dynamics model in thisscenario.
Revised: , even goes even perform very well since it is very easy to learn a perfect inverse dynamics model in thisscenario.

Original: OPOLO’s upper bound gap (4) will also be close to zero, thus OPOLO directly minimizesthe divergence of state-transition distributions.
Revised: OPOLO’s upper bound gap (4) will also be close to zero, thus OPOLO directly minimizesthe divergence of state-transition distributions.

Original: As a result, there is no performance gap amongdifferent algorithms, except for BC whose performance is determined by the quality of imperfectdemonstrations.
Revised: As a result, there is no performance gap amongdifferent algorithms, except for BC whose performance is determined by the quality of imperfectdemonstrations.

Original: Also, the performance of all algorithms (except for BC) improves as more data isgiven, which is natural.
Revised: Also, the performance of all algorithms (except for BC) improves as more data isgiven, which is natural.

Original: The second row and the third row in Figure 1 presents the result when β = 0 .
Revised: The second row and the third row in Figure 1 presents the result when β = 0 .

Original: 1 (weakly stochastic
Revised: 1 (weakly stochastic

Original: MDP) and β = 1 . 

Revised: MDP) and β = 1 .

Original: (highly stochastic MDP) respectively.
Revised: (highly stochastic MDP) respectively.

Original: In the stochastic MDPs, the IDM trainedby the imperfect demonstrations faces a challenge in predicting the expert’s actions accurately (morechallenging as β gets larger).
Revised: In the stochastic MDPs, the IDM trainedby the imperfect demonstrations faces a challenge in predicting the expert’s actions accurately (morechallenging as β gets larger).

Original: As a consequence, BCO gets suboptimal and its suboptimality cannotbe improved even if more data is given.
Revised: As a consequence, BCO gets suboptimal and its suboptimality cannotbe improved even if more data is given.

Original: DemoDICEfO performs better than BCO since it additionallyconsiders the distributional shift by considering state distribution matching.
Revised: DemoDICEfO performs better than BCO since it additionallyconsiders the distributional shift by considering state distribution matching.

Original: However, it is stillsuboptimal due to its nature that directly depends on the quality of inferred action by the learned
Revised: However, it is stillsuboptimal due to its nature that directly depends on the quality of inferred action by the learned

Original: IDM.
Revised: IDM.

Original: OPOLO does not rely on the learned IDM and outperforms both BCO and DemoDICEfO.
Revised: OPOLO does not rely on the learned IDM and outperforms both BCO and DemoDICEfO.

Original: Still, OPOLO can be inherently suboptimal due to its nature of optimizing the upper bound unlessthe underlying transition dynamics are deterministic and injective.
Revised: Still, OPOLO can be inherently suboptimal due to its nature of optimizing the upper bound unlessthe underlying transition dynamics are deterministic and injective.

Original: This upper bound gap (4) is notcontrollable by the algorithm and implies that OPOLO can be suboptimal even given an infiniteamount of data with sufficient dataset coverage, which can be seen in the rightmost figures.
Revised: This upper bound gap (4) is notcontrollable by the algorithm and implies that OPOLO can be suboptimal even given an infiniteamount of data with sufficient dataset coverage, which can be seen in the rightmost figures.

Original: Finally,our tabular LobsDICE using (16) essentially solves the exact state-transition distribution matchingproblem (as α → 0 ).
Revised: Finally,our tabular LobsDICE using (16) essentially solves the exact state-transition distribution matchingproblem (as α → 0 ).

Original: LobsDICE is the only offline LfO algorithm that can asymptotically recoversthe expert’s state demonstrations even though the underlying MDP is stochastic.
Revised: LobsDICE is the only offline LfO algorithm that can asymptotically recoversthe expert’s state demonstrations even though the underlying MDP is stochastic.

Original: 5.2 Continuous control tasks (Gym-MuJoCo)
Revised: 5.2 Continuous control tasks (Gym-MuJoCo)

Original: We present the empirical performance of LobsDICE and baselines on MuJoCo [37] continuouscontrol tasks using the OpenAI Gym [3] framework.
Revised: We present the empirical performance of LobsDICE and baselines on MuJoCo [37] continuouscontrol tasks using the OpenAI Gym [3] framework.

Original: We utilize the D4RL dataset [6] for offline
Revised: We utilize the D4RL dataset [6] for offline

Original: LfO tasks in four MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant.
Revised: LfO tasks in four MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant.

Original: Implementationdetails for LobsDICE and baseline algorithms such as hyperparameters and evaluation metric areprovided in Appendix I.2.
Revised: Implementationdetails for LobsDICE and baseline algorithms such as hyperparameters and evaluation metric areprovided in Appendix I.2.

Original: Task setup For each MuJoCo environments, we employ expert-v2 , medium-v2 , and random-v2from D4RL datasets [6].
Revised: Task setup For each MuJoCo environments, we employ expert-v2 , medium-v2 , and random-v2from D4RL datasets [6].

Original: Across all environments, we consider three tasks, each of which uses differentimperfect demonstrations while sharing the same expert observations.
Revised: Across all environments, we consider three tasks, each of which uses differentimperfect demonstrations while sharing the same expert observations.

Original: First, we construct the stateonly expert demonstration D E
Revised: First, we construct the stateonly expert demonstration D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: i } N E i =
Revised: i } N E i =

Original: 1 using the first 5 trajectories in expert-v2 .
Revised: 1 using the first 5 trajectories in expert-v2 .

Original: Then, weuse trajectories in expert-v2 , medium-v2 , and random-v2 to construct imperfect demonstrations
Revised: Then, weuse trajectories in expert-v2 , medium-v2 , and random-v2 to construct imperfect demonstrations

Original: [Figure]
Revised: [Figure]

Original: We build state-only expert demonstrations using 5 trajectories from expert-v2 .
Revised: We build state-only expert demonstrations using 5 trajectories from expert-v2 .

Original: For each task ( X, Y, Z ) we construct imperfect demonstrations using X , Y , and Z trajectories from expert-v2 , medium-v2 , and random-v2 , respectively.
Revised: For each task ( X, Y, Z ) we construct imperfect demonstrations using X , Y , and Z trajectories from expert-v2 , medium-v2 , and random-v2 , respectively.

Original: We plot the mean and the standard errors (shaded area) of the normalized scores over five random seeds.
Revised: We plot the mean and the standard errors (shaded area) of the normalized scores over five random seeds.

Original: with different ratios.
Revised: with different ratios.

Original: We denote the composition of imperfect demonstrations as ( X, Y, Z ) in the titleof each subplot in Figure 2, which means that the imperfect dataset consists of X trajectories fromexpert-v2 , Y trajectories from medium-v2 , and Z trajectories from random-v2 .
Revised: We denote the composition of imperfect demonstrations as ( X, Y, Z ) in the titleof each subplot in Figure 2, which means that the imperfect dataset consists of X trajectories fromexpert-v2 , Y trajectories from medium-v2 , and Z trajectories from random-v2 .

Original: Results Figure 2 summarizes that the empirical results of LobsDICE and baselines on continuouscontrol tasks.
Revised: Results Figure 2 summarizes that the empirical results of LobsDICE and baselines on continuouscontrol tasks.

Original: We first remark that LobsDICE (blue) significantly outperforms OPOLO (green) in alltasks across all domains, although both LobsDICE and OPOLO are DICE-based algorithms.
Revised: We first remark that LobsDICE (blue) significantly outperforms OPOLO (green) in alltasks across all domains, although both LobsDICE and OPOLO are DICE-based algorithms.

Original: Thefailure of OPOLO comes from its numerical instability due to its dependence on nested optimizationand using OOD action values during training.
Revised: Thefailure of OPOLO comes from its numerical instability due to its dependence on nested optimizationand using OOD action values during training.

Original: In contrast, LobsDICE solves a single minimization (23)while it does not involve any evaluation on OOD actions, thus it is optimized stably.
Revised: In contrast, LobsDICE solves a single minimization (23)while it does not involve any evaluation on OOD actions, thus it is optimized stably.

Original: IQ-Learn (purple)also suffers from numerical instability due to the usage of OOD action values during training, showingpoor performance similar to OPOLO.
Revised: IQ-Learn (purple)also suffers from numerical instability due to the usage of OOD action values during training, showingpoor performance similar to OPOLO.

Original: RCE (brown) tends to stay in a few states that are easy to reachin the offline setting, rather than following the entire expert trajectories.
Revised: RCE (brown) tends to stay in a few states that are easy to reachin the offline setting, rather than following the entire expert trajectories.

Original: Naive BC on imperfectdemonstrations (black) is inherently suboptimal since it does not consider distribution-matchingwith the expert’s observation at all.
Revised: Naive BC on imperfectdemonstrations (black) is inherently suboptimal since it does not consider distribution-matchingwith the expert’s observation at all.

Original: While BCO (orange) exploits the expert’s demonstrations withthe inferred actions by the IDM, its policy learning is done only on the very scarce expert dataset(i.e.
Revised: While BCO (orange) exploits the expert’s demonstrations withthe inferred actions by the IDM, its policy learning is done only on the very scarce expert dataset(i.e.

Original: 5 trajectories), which makes the algorithm perform not well.
Revised: 5 trajectories), which makes the algorithm perform not well.

Original: DemoDICEfO (red) exploits bothexpert demonstrations (where the missing actions are filled with the IDM) and the abundant imperfectdemonstrations, but its performance is affected by the quality of the learned IDM.
Revised: DemoDICEfO (red) exploits bothexpert demonstrations (where the missing actions are filled with the IDM) and the abundant imperfectdemonstrations, but its performance is affected by the quality of the learned IDM.

Original: We empiricallyobserve that the IDM error (on the true expert data) increases as the proportion of the non-expertdata (i.e. medium-v2 and random-v2 ) increases, resulting in performance degradation of both
Revised: We empiricallyobserve that the IDM error (on the true expert data) increases as the proportion of the non-expertdata (i.e. medium-v2 and random-v2 ) increases, resulting in performance degradation of both

Original: BCO and DemoDICEfO.
Revised: BCO and DemoDICEfO.

Original: Finally, LobsDICE is the only algorithm that was able to fully recover theexpert’s performance regardless of the increase of non-expert data in the imperfect demonstrations,significantly outperforming baseline algorithms.
Revised: Finally, LobsDICE is the only algorithm that was able to fully recover theexpert’s performance regardless of the increase of non-expert data in the imperfect demonstrations,significantly outperforming baseline algorithms.

Original: This result highlights the effectiveness of ourmethod that solves a state-transition stationary matching problem in a principled manner.
Revised: This result highlights the effectiveness of ourmethod that solves a state-transition stationary matching problem in a principled manner.

Original: We provideadditional experiments in Appendix H.
Revised: We provideadditional experiments in Appendix H.

Original:  Conclusion
Revised:  Conclusion

Original: We presented LobsDICE, an algorithm for offline learning from observations (LfO), which successfully achieves state-of-the-art performance on various tabular and continuous tasks.
Revised: We presented LobsDICE, an algorithm for offline learning from observations (LfO), which successfully achieves state-of-the-art performance on various tabular and continuous tasks.

Original: We formulatedthe offline LfO as a state-transition stationary distribution matching problem, where the stationarydistribution is optimized via convex minimization.
Revised: We formulatedthe offline LfO as a state-transition stationary distribution matching problem, where the stationarydistribution is optimized via convex minimization.

Original: Experimental results demonstrated that LobsDICEachieves promising performance in both tabular and continuous offline LfO tasks.
Revised: Experimental results demonstrated that LobsDICEachieves promising performance in both tabular and continuous offline LfO tasks.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] See abstract and introduction (Section 1)
Revised: [Yes] See abstract and introduction (Section 1)

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Due to space limit, we discuss it
Revised: [Yes] Due to space limit, we discuss it

Original: in the supplementary material (Section L)
Revised: in the supplementary material (Section L)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Due tospace limit, we discuss it in the supplementary material (Section N).
Revised: [Yes] Due tospace limit, we discuss it in the supplementary material (Section N).

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] In appropriate
Revised: [Yes] In appropriate

Original: places.
Revised: places.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Due to space limit,we provide proofs in the supplementary material.
Revised: [Yes] Due to space limit,we provide proofs in the supplementary material.

Original: See Section F
Revised: See Section F

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We submit thecode as supplementary material.
Revised: [Yes] We submit thecode as supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We specify them in the supplementary material.
Revised: [Yes] We specify them in the supplementary material.

Original: See Section I(c) Did you report error bars (e
Revised: See Section I(c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] We report them in each figure.
Revised: [Yes] We report them in each figure.

Original: See Figure 1 and 2.
Revised: See Figure 1 and 2.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We specify them in the
Revised: [Yes] We specify them in the

Original: supplementary material.
Revised: supplementary material.

Original: See Section K
Revised: See Section K

Original: [Table]
Revised: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite Random
Revised: [Yes] We cite Random

Original: MDP protocols and D4RL dataset.
Revised: MDP protocols and D4RL dataset.

Original: See Section 5.
Revised: See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] D4RL dataset is licensed under the
Revised: [Yes] D4RL dataset is licensed under the

Original: Apache 2.0.
Revised: Apache 2.0.

Original: See Section M.
Revised: See Section M.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A] All the datasets we used are public.
Revised: [N/A] All the datasets we used are public.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: None of datasets contain identifiable contents.
Revised: None of datasets contain identifiable contents.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: Why State-transition Occupancy Matching Instead of State Occupancy
Revised: Why State-transition Occupancy Matching Instead of State Occupancy

Original: [Equation]
Revised: [Equation]

Original: aunique solution of π 2 , exploiting the directionality information (i.e. moving counterclockwise) of theexpert’s trajectories.
Revised: aunique solution of π 2 , exploiting the directionality information (i.e. moving counterclockwise) of theexpert’s trajectories.

Original: B LobsDICE for State Occupancy Matching
Revised: B LobsDICE for State Occupancy Matching

Original: While we have provided a counter-example where d ( s ) -matching can fail to recover the expert’sbehavior in Appendix A, we still can derive an offline LfO algorithm for d ( s ) -matching.
Revised: While we have provided a counter-example where d ( s ) -matching can fail to recover the expert’sbehavior in Appendix A, we still can derive an offline LfO algorithm for d ( s ) -matching.

Original: For thegiven policy π , its state stationary distribution is defined as:
Revised: For thegiven policy π , its state stationary distribution is defined as:

Original: [Equation]
Revised: [Equation]

Original: Then, similar to the state-transition occupancy matching objective (5), we formulate a state occupancymatching problem as follows:
Revised: Then, similar to the state-transition occupancy matching objective (5), we formulate a state occupancymatching problem as follows:

Original: [Equation]
Revised: [Equation]

Original: (25) where the hyperparameter α > 0 balances between encouraging state matching and preventingdistribution shift from the distribution of imperfect demonstrations.
Revised: (25) where the hyperparameter α > 0 balances between encouraging state matching and preventingdistribution shift from the distribution of imperfect demonstrations.

Original: This objective can be reformulatedin terms of optimizing stationary distribution:
Revised: This objective can be reformulatedin terms of optimizing stationary distribution:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, we consider the Lagrangian for the constrained optimization (26-28):
Revised: Then, we consider the Lagrangian for the constrained optimization (26-28):

Original: [Equation]
Revised: [Equation]

Original: where ν ( s ′ ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (27), and µ
Revised: where ν ( s ′ ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (27), and µ

Original: ( s ) ∈
Revised: ( s ) ∈

Original: R arethe Lagrange multipliers for the marginalization constraints (28).
Revised: R arethe Lagrange multipliers for the marginalization constraints (28).

Original: To make the optimization tractablein the offline setting, we rearrange the terms in (29), introducing new optimization variables w and ¯ wthat denote stationary distribution correction ratios for ( s, a ) and s respectively.
Revised: To make the optimization tractablein the offline setting, we rearrange the terms in (29), introducing new optimization variables w and ¯ wthat denote stationary distribution correction ratios for ( s, a ) and s respectively.

Original: [Equation]
Revised: [Equation]

Original: We introduce the log ratio r ( s ) = log d E ( s )
Revised: We introduce the log ratio r ( s ) = log d E ( s )

Original: d I ( s ) to make expectation for d I instead of d E .
Revised: d I ( s ) to make expectation for d I instead of d E .

Original: The log ratior ( s ) can be estimated by using a pretrained discriminator c : S → [0 , 1] , where c is trained by:
Revised: The log ratior ( s ) can be estimated by using a pretrained discriminator c : S → [0 , 1] , where c is trained by:

Original: [Equation]
Revised: [Equation]

Original: Then, it is proven that the optimal discriminator c ∗ satisfies c ∗ ( s )
Revised: Then, it is proven that the optimal discriminator c ∗ satisfies c ∗ ( s )

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: d I ( s ) .
Revised: d I ( s ) .

Original: Therefore, r ( s ) canbe estimated by:
Revised: Therefore, r ( s ) canbe estimated by:

Original: [Equation]
Revised: [Equation]

Original: Similar to Proposition 3.1, we can easily derive the closed-form solution to the inner maximization in(30):
Revised: Similar to Proposition 3.1, we can easily derive the closed-form solution to the inner maximization in(30):

Original: [Equation]
Revised: [Equation]

Original: Finally, by plugging these closed-form solutions into (30), the nested min-max optimization of (30)is reduced to a single minimization:
Revised: Finally, by plugging these closed-form solutions into (30), the nested min-max optimization of (30)is reduced to a single minimization:

Original: [Equation]
Revised: [Equation]

Original: which is a convex function of µ and ν .
Revised: which is a convex function of µ and ν .

Original: Finally, once we obtain the optimal solution ( µ ∗ , ν ∗ ) of (31),we have w µ ∗ ,ν ∗ ( s, a ) =
Revised: Finally, once we obtain the optimal solution ( µ ∗ , ν ∗ ) of (31),we have w µ ∗ ,ν ∗ ( s, a ) =

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: s,a
Revised: s,a

Original: [Equation]
Revised: [Equation]

Original: I ( s,a ) .
Revised: I ( s,a ) .

Original: Then, we can extract an optimal policy via weighted BC:
Revised: Then, we can extract an optimal policy via weighted BC:

Original: [Equation]
Revised: [Equation]

Original: Remark B.1.
Revised: Remark B.1.

Original: To the best of our knowledge, (31) is the first result to directly solve the state distributionmatching problem in an offline setting.
Revised: To the best of our knowledge, (31) is the first result to directly solve the state distributionmatching problem in an offline setting.

Original: Although SMODICE [28], a concurrent work with ours, alsoaims to solve offline LfO in terms of optimizing stationary distribution, it essentially minimizes the(potentially loose) upper bound , i.e.,
Revised: Although SMODICE [28], a concurrent work with ours, alsoaims to solve offline LfO in terms of optimizing stationary distribution, it essentially minimizes the(potentially loose) upper bound , i.e.,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The upper bound gap (33) − (35) is given by D KL ( π ( a | s )
Revised: The upper bound gap (33) − (35) is given by D KL ( π ( a | s )

Original: ||
Revised: ||

Original: π I ( a | s )) .
Revised: π I ( a | s )) .

Original: Unlike OPOLO of (4), thisupper bound gap does not vanish even when the transition is deterministic.
Revised: Unlike OPOLO of (4), thisupper bound gap does not vanish even when the transition is deterministic.

Original: Consequently, SMODICEmay not be able to precisely recover the expert’s state visitations even given an infinite amount ofdata due to the upper bound gap unless d I is collected purely by expert.
Revised: Consequently, SMODICEmay not be able to precisely recover the expert’s state visitations even given an infinite amount ofdata due to the upper bound gap unless d I is collected purely by expert.

Original: In contrast, we are directlyminimizing the divergence: solving (31) is exactly equivalent to solving (25).
Revised: In contrast, we are directlyminimizing the divergence: solving (31) is exactly equivalent to solving (25).

Original: As can be seen in
Revised: As can be seen in

Original: Figure B, SMODICE optimizing (33) fails to recover the expert’s behavior due to the upper boundgap that is irreducible even for deterministic MDPs.
Revised: Figure B, SMODICE optimizing (33) fails to recover the expert’s behavior due to the upper boundgap that is irreducible even for deterministic MDPs.

Original: [Figure]
Revised: [Figure]

Original: C Validity of Weighted Behavior Cloning for Policy Extraction
Revised: C Validity of Weighted Behavior Cloning for Policy Extraction

Original: For tabular MDPs, we can show that the weighted behavior cloning of (17)
Revised: For tabular MDPs, we can show that the weighted behavior cloning of (17)

Original: [Equation]
Revised: [Equation]

Original: (17) exactly yields an optimal policy π ∗ of the original optimization problem (5).
Revised: (17) exactly yields an optimal policy π ∗ of the original optimization problem (5).

Original: First, the optimizationproblem (5) and the constrained optimization problem (6-8) are equivalent with the followingrelationship between π ∗ and d ∗ :
Revised: First, the optimizationproblem (5) and the constrained optimization problem (6-8) are equivalent with the followingrelationship between π ∗ and d ∗ :

Original: [Equation]
Revised: [Equation]

Original: (36) where d ∗ is the optimal solution of both (6-8) and (10).
Revised: (36) where d ∗ is the optimal solution of both (6-8) and (10).

Original: Then, w µ ∗ ,ν ∗ ( s, a ) is computed by (15) with( µ ∗ , ν ∗ ) , the optimal solution of both (16) and (11).
Revised: Then, w µ ∗ ,ν ∗ ( s, a ) is computed by (15) with( µ ∗ , ν ∗ ) , the optimal solution of both (16) and (11).

Original: Consequently, w µ ∗ ,ν ∗ ( s, a ) =
Revised: Consequently, w µ ∗ ,ν ∗ ( s, a ) =

Original: d ∗ ( s,a ) d I ( s,a ) holds, andthus the weighted behavior cloning of (17) essentially performs the following:
Revised: d ∗ ( s,a ) d I ( s,a ) holds, andthus the weighted behavior cloning of (17) essentially performs the following:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, consider the Lagrangian for the constrained optimization (37-38):
Revised: Now, consider the Lagrangian for the constrained optimization (37-38):

Original: [Equation]
Revised: [Equation]

Original: where λ ( s ) is the Lagrange multiplier for the constraint (38).
Revised: where λ ( s ) is the Lagrange multiplier for the constraint (38).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Finally, considering the constraint (cid:80) a x ∗ ( a | s ) = 1 ∀ s , we obtain:
Revised: Finally, considering the constraint (cid:80) a x ∗ ( a | s ) = 1 ∀ s , we obtain:

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: 41) which is identical to (36).
Revised: 41) which is identical to (36).

Original: In summary, the weighted behavior cloning performs the process ofextracting the optimal policy π ∗ from w µ ∗ ,ν ∗ .
Revised: In summary, the weighted behavior cloning performs the process ofextracting the optimal policy π ∗ from w µ ∗ ,ν ∗ .

Original: D Challenges of Extending DemoDICE to Offline Learning from Observation
Revised: D Challenges of Extending DemoDICE to Offline Learning from Observation

Original: DemoDICE [15] is an algorithm that optimizes the state-action stationary distribution d
Revised: DemoDICE [15] is an algorithm that optimizes the state-action stationary distribution d

Original: ( s, a ) , whereits naive application to ( s, s ′ )
Revised: ( s, a ) , whereits naive application to ( s, s ′ )

Original: -distribution matching problem yields:
Revised: -distribution matching problem yields:

Original: [Equation]
Revised: [Equation]

Original: However, estimating E ( s,a ) ∼
Revised: However, estimating E ( s,a ) ∼

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: (cid:105) is intractable in the offline setting due to itsinclusion of marginalization over a inside the log( · ) .
Revised: (cid:105) is intractable in the offline setting due to itsinclusion of marginalization over a inside the log( · ) .

Original: This challenge can be addressed by using theupper bound proposed by OPOLO [45]:
Revised: This challenge can be addressed by using theupper bound proposed by OPOLO [45]:

Original: [Equation]
Revised: [Equation]

Original: Replacing D KL ( ¯ d π ( s, s ′ )
Revised: Replacing D KL ( ¯ d π ( s, s ′ )

Original: ||
Revised: ||

Original: d E ( s, s ′ )) with the (potentially loose) upper bound in the objective function yields the following tractable (but biased) optimization problem:
Revised: d E ( s, s ′ )) with the (potentially loose) upper bound in the objective function yields the following tractable (but biased) optimization problem:

Original: [Equation]
Revised: [Equation]

Original: In contrast, we address the intractability challenge by introducing an additional optimization variable¯ d ( s, s ′ ) along with the marginalization constraint (8).
Revised: In contrast, we address the intractability challenge by introducing an additional optimization variable¯ d ( s, s ′ ) along with the marginalization constraint (8).

Original: This allows for tractable offline optimizationby (11,16) without introducing bias, which is our novel contribution.
Revised: This allows for tractable offline optimizationby (11,16) without introducing bias, which is our novel contribution.

Original: E More Discussions on LobsDICE with Finite Samples
Revised: E More Discussions on LobsDICE with Finite Samples

Original: In this section, we provide additional discussions on LobsDICE with finite samples.
Revised: In this section, we provide additional discussions on LobsDICE with finite samples.

Original: E.1 LobsDICE with sample-based approximation: distribution matching on the MLE MDP
Revised: E.1 LobsDICE with sample-based approximation: distribution matching on the MLE MDP

Original: We first show that LobsDICE that optimizes (18) is equivalent to performing optimization of (6-8)on the MLE MDP constructed by D I .
Revised: We first show that LobsDICE that optimizes (18) is equivalent to performing optimization of (6-8)on the MLE MDP constructed by D I .

Original: Let ˆ M \ R = ⟨ S, A, ˆ T, (cid:98) p 0 , γ ⟩ be the MLE MDP constructedusing D
Revised: Let ˆ M \ R = ⟨ S, A, ˆ T, (cid:98) p 0 , γ ⟩ be the MLE MDP constructedusing D

Original: I .
Revised: I .

Original: Then, constrained optimization (6-8) for the MLE MDP ˆ M \ R can be formulated as:
Revised: Then, constrained optimization (6-8) for the MLE MDP ˆ M \ R can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) p 0 , (cid:98) d E , and (cid:98) d
Revised: where (cid:98) p 0 , (cid:98) d E , and (cid:98) d

Original: I are empirical distributions of D 0 , D E
Revised: I are empirical distributions of D 0 , D E

Original: [Equation]
Revised: [Equation]

Original: and D I , respectively.
Revised: and D I , respectively.

Original: We consider
Revised: We consider

Original: Lagrangian for the above constrained optimization:
Revised: Lagrangian for the above constrained optimization:

Original: [Equation]
Revised: [Equation]

Original: Note that this is identical to (18).
Revised: Note that this is identical to (18).

Original: We can also derive the closed-form solution ( w Iµ,ν , ¯ w Iµ ) to themax w, ¯ w
Revised: We can also derive the closed-form solution ( w Iµ,ν , ¯ w Iµ ) to themax w, ¯ w

Original: [Equation]
Revised: [Equation]

Original: L I ( w, ¯ w, µ,
Revised: L I ( w, ¯ w, µ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that this is different from (19) in that (49) is the closed-form solution for each ( s, a ) and ( s, s ′ ) ,while (19) is the non-parametric closed-form solution for each sample x ∈ D I .
Revised: Note that this is different from (19) in that (49) is the closed-form solution for each ( s, a ) and ( s, s ′ ) ,while (19) is the non-parametric closed-form solution for each sample x ∈ D I .

Original: Exploiting (49) doesnot introduce an additional bias, but requires evaluation of the expectation within the exp( · ) , i.e.transition model is needed.
Revised: Exploiting (49) doesnot introduce an additional bias, but requires evaluation of the expectation within the exp( · ) , i.e.transition model is needed.

Original: The last step is to plug the solution ( w Iµ,ν , ¯ w Iµ ) in (49) into L I ( w, ¯ w, µ, ν ) :
Revised: The last step is to plug the solution ( w Iµ,ν , ¯ w Iµ ) in (49) into L I ( w, ¯ w, µ, ν ) :

Original: [Equation]
Revised: [Equation]

Original: Finally, this can be rewritten as:
Revised: Finally, this can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: which is the objective function for our tabular LobsDICE (52).
Revised: which is the objective function for our tabular LobsDICE (52).

Original: E.2 When will LobsDICE reduce to BCO?
Revised: E.2 When will LobsDICE reduce to BCO?

Original: LobsDICE reduces to BCO when D I is collected by an expert policy and D E is a dataset identical to
Revised: LobsDICE reduces to BCO when D I is collected by an expert policy and D E is a dataset identical to

Original: D I except that action is missing, i.e., D E
Revised: D I except that action is missing, i.e., D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: | ( s, a, s ′ )
Revised: | ( s, a, s ′ )

Original: ∈
Revised: ∈

Original: D I } .
Revised: D I } .

Original: In this situation, note that (cid:98) d
Revised: In this situation, note that (cid:98) d

Original: I ( s, s ′ ) and (cid:98) d
Revised: I ( s, s ′ ) and (cid:98) d

Original: I ( s, a ) are valid stationary distributions on the MLE MDPconstructed by D I , since they satisfy both the Bellman flow constraint (47) and the marginalizationconstraint (48) on the MLE MDP.
Revised: I ( s, a ) are valid stationary distributions on the MLE MDPconstructed by D I , since they satisfy both the Bellman flow constraint (47) and the marginalizationconstraint (48) on the MLE MDP.

Original: The (cid:98) d I ( s, s ′ ) and (cid:98) d
Revised: The (cid:98) d I ( s, s ′ ) and (cid:98) d

Original: I ( s, a ) are also the optimal solution of (46-48),making the KL divergences to 0:
Revised: I ( s, a ) are also the optimal solution of (46-48),making the KL divergences to 0:

Original: [Equation]
Revised: [Equation]

Original: Finally, the policy obtained by LobsDICE will simply reduce to BC on D I by noting that:
Revised: Finally, the policy obtained by LobsDICE will simply reduce to BC on D I by noting that:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, one can easily show that BC on D I is identical to BCO (i.e. BC on D E where themissing actions are inferred by the IDM trained by D I ), which concludes that LobsDICE is equivalentto BCO in this special case.
Revised: Furthermore, one can easily show that BC on D I is identical to BCO (i.e. BC on D E where themissing actions are inferred by the IDM trained by D I ), which concludes that LobsDICE is equivalentto BCO in this special case.

Original: However, D I will include demonstrations collected by non-expert policies in general, which makesthe LobsDICE’s solution different from (and usually better than) the BCO’s solution in general cases.
Revised: However, D I will include demonstrations collected by non-expert policies in general, which makesthe LobsDICE’s solution different from (and usually better than) the BCO’s solution in general cases.

Original: F Theoretical Analysis
Revised: F Theoretical Analysis

Original: F.1 Closed-form solutions
Revised: F.1 Closed-form solutions

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.
Revised: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.

Original: [Equation]
Revised: [Equation]

Original: ν ) , is given by:
Revised: ν ) , is given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let e µ,ν ( s, a, s ′ )
Revised: Let e µ,ν ( s, a, s ′ )

Original: = − µ ( s, s ′ )
Revised: = − µ ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) .
Revised: − ν ( s ) .

Original: Then, e µ,ν ( s, a ) =
Revised: Then, e µ,ν ( s, a ) =

Original: E s ′ ∼ T ( ·| s,a ) [ e µ,ν ( s, a, s ′ )] .
Revised: E s ′ ∼ T ( ·| s,a ) [ e µ,ν ( s, a, s ′ )] .

Original: For ( s, a ) with d I ( s, a ) > 0 , we can compute the derivative ∂
Revised: For ( s, a ) with d I ( s, a ) > 0 , we can compute the derivative ∂

Original: L ∂w ( s,a ) of L w.r.t.
Revised: L ∂w ( s,a ) of L w.r.t.

Original: w ( s, a ) as follows:
Revised: w ( s, a ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Similar to the aforementioned derivation, when ¯ d I ( s, s ′ ) > 0 , we can derive the derivative of L w.r.t.¯ w ( s, s ′ ) :
Revised: Similar to the aforementioned derivation, when ¯ d I ( s, s ′ ) > 0 , we can derive the derivative of L w.r.t.¯ w ( s, s ′ ) :

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .
Revised: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .

Original: Proof.
Revised: Proof.

Original: Using the fact that exp( · ) is a convex function, we can easily prove the convexity of
Revised: Using the fact that exp( · ) is a convex function, we can easily prove the convexity of

Original: L ( w µ,ν , ¯ w µ,ν , µ, ν ) .
Revised: L ( w µ,ν , ¯ w µ,ν , µ, ν ) .

Original: For brevity, let L ( µ, ν ) :
Revised: For brevity, let L ( µ, ν ) :

Original: = L ( w µ,ν , ¯ w µ,ν , µ, ν ) .
Revised: = L ( w µ,ν , ¯ w µ,ν , µ, ν ) .

Original: Then, for given ( µ 1 , ν 1 ) ,( µ 2 , ν 2 ) and t ∈ [0 , 1] ,
Revised: Then, for given ( µ 1 , ν 1 ) ,( µ 2 , ν 2 ) and t ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: 1 + (1 − t ) ν 2 ) =
Revised: 1 + (1 − t ) ν 2 ) =

Original: (1 − γ ) E p 

Revised: (1 − γ ) E p

Original: [ tν 1 ( s ) + (1 − t ) ν 2 ( s )]
Revised: [ tν 1 ( s ) + (1 − t ) ν 2 ( s )]

Original: + E d
Revised: + E d

Original: I
Revised: I

Original: [exp( r ( s, s ′ )
Revised: [exp( r ( s, s ′ )

Original: + tµ 1 ( s, s ′ )
Revised: + tµ 1 ( s, s ′ )

Original: + (1 − t ) µ 2 ( s, s ′ ) − 1)]
Revised: + (1 − t ) µ 2 ( s, s ′ ) − 1)]

Original: + α E d I (cid:104) exp (cid:16) 

Revised: + α E d I (cid:104) exp (cid:16)

Original: α E s ′ (cid:2) − tµ 1 ( s, s ′ ) − (1 − t ) µ 2 ( s, s ′ )
Revised: α E s ′ (cid:2) − tµ 1 ( s, s ′ ) − (1 − t ) µ 2 ( s, s ′ )

Original: + γtν 1 ( s ′ )
Revised: + γtν 1 ( s ′ )

Original: + γ (1 − t ) ν 

Revised: + γ (1 − t ) ν

Original: ( s ′ )
Revised: ( s ′ )

Original: − tν 1 ( s ) − (1 − t )
Revised: − tν 1 ( s ) − (1 − t )

Original: ν 2 ( s ) (cid:3) − 1 (cid:17)(cid:105) =
Revised: ν 2 ( s ) (cid:3) − 1 (cid:17)(cid:105) =

Original: t (1 − γ ) E p 

Revised: t (1 − γ ) E p

Original: [ ν 1 ( s )]
Revised: [ ν 1 ( s )]

Original: + (1 − t )(1 − γ ) E p 

Revised: + (1 − t )(1 − γ ) E p

Original: [ ν 2 ( s )]
Revised: [ ν 2 ( s )]

Original: + E d
Revised: + E d

Original: I
Revised: I

Original: [exp( r ( s, s ′ )
Revised: [exp( r ( s, s ′ )

Original: + tµ 1 + (1 − t ) µ 2 − 1)]
Revised: + tµ 1 + (1 − t ) µ 2 − 1)]

Original: + α E d I (cid:104) exp (cid:16) 

Revised: + α E d I (cid:104) exp (cid:16)

Original: α E s ′ (cid:2) − tµ 1 ( s, s ′ ) − (1 − t ) µ 2 ( s, s ′ )
Revised: α E s ′ (cid:2) − tµ 1 ( s, s ′ ) − (1 − t ) µ 2 ( s, s ′ )

Original: + γtν 1 ( s ′ )
Revised: + γtν 1 ( s ′ )

Original: + γ (1 − t ) ν 

Revised: + γ (1 − t ) ν

Original: ( s ′ )
Revised: ( s ′ )

Original: − tν 1 ( s ) − (1 − t )
Revised: − tν 1 ( s ) − (1 − t )

Original: ν 2 ( s ) (cid:3) − 1 (cid:17)(cid:105)
Revised: ν 2 ( s ) (cid:3) − 1 (cid:17)(cid:105)

Original: [Equation]
Revised: [Equation]

Original: For the inequalities in the above formulation
Revised: For the inequalities in the above formulation

Original: [Equation]
Revised: [Equation]

Original: we use the fact that E d I [exp( · )] is a instance of convexfunctions.
Revised: we use the fact that E d I [exp( · )] is a instance of convexfunctions.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e.µ ν = arg min µ (cid:98) L ( µ, ν ) , is
Revised: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e.µ ν = arg min µ (cid:98) L ( µ, ν ) , is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: For simplicity, let (cid:98) L ( µ, ν ) :
Revised: For simplicity, let (cid:98) L ( µ, ν ) :

Original: = (cid:98) L ( w µ,ν , ¯ w µ,ν , µ, ν ) and x :=
Revised: = (cid:98) L ( w µ,ν , ¯ w µ,ν , µ, ν ) and x :=

Original: ( s, a, s ′ ) .
Revised: ( s, a, s ′ ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: F.2 Surrogate Objective
Revised: F.2 Surrogate Objective

Original: We first show a property of ˜ L that will be used to prove propositions:
Revised: We first show a property of ˜ L that will be used to prove propositions:

Original: Lemma F.1.
Revised: Lemma F.1.

Original: For given function ν : S → R ,
Revised: For given function ν : S → R ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any C ∈ R ,˜ L ( ν + C )
Revised: For any C ∈ R ,˜ L ( ν + C )

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E s ∼ p 

Revised: E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 

Revised: ∼ d I (cid:104) exp (cid:16) 1

Original: + α ˆ
Revised: + α ˆ

Original: A
Revised: A

Original: ν
Revised: ν

Original: + C ( s, a, s ′ ) (cid:17)(cid:105) =
Revised: + C ( s, a, s ′ ) (cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )
Revised: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )

Original: + γ ( ν ( s ′ )
Revised: + γ ( ν ( s ′ )

Original: + C ) − ( ν ( s ) + C ) (cid:1)(cid:17)(cid:105) =
Revised: + C ) − ( ν ( s ) + C ) (cid:1)(cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )
Revised: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) (cid:1)(cid:17) exp (cid:16) ( γ − 1) C 1+ α (cid:17)(cid:105) =
Revised: − ν ( s ) (cid:1)(cid:17) exp (cid:16) ( γ − 1) C 1+ α (cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 1+ α ( r ( s, s ′ )
Revised: ∼ d I (cid:104) exp (cid:16) 1 1+ α ( r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ))
Revised: − ν ( s ))

Original: (cid:17)(cid:105) + ( γ − 1) C
Revised: (cid:17)(cid:105) + ( γ − 1) C

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E s ∼ p 

Revised: E s ∼ p

Original: [ ν ( s )]
Revised: [ ν ( s )]

Original: + (1 + α ) log E ( s,a,s ′ ) ∼ d I (cid:104) exp (cid:16) 1 1+ α
Revised: + (1 + α ) log E ( s,a,s ′ ) ∼ d I (cid:104) exp (cid:16) 1 1+ α

Original: r ( s, s ′ )
Revised: r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ))
Revised: − ν ( s ))

Original: (cid:17)(cid:105) =
Revised: (cid:17)(cid:105) =

Original: ˜ L ( ν ) .
Revised: ˜ L ( ν ) .

Original: Now, we prove Proposition 3.4 and Proposition 3.5:
Revised: Now, we prove Proposition 3.4 and Proposition 3.5:

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: Let (cid:101) L ( (cid:101) ν ) be the function:
Revised: Let (cid:101) L ( (cid:101) ν ) be the function:

Original: [Equation]
Revised: [Equation]

Original: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )
Revised: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )

Original: = min (cid:101) ν (cid:101)
Revised: = min (cid:101) ν (cid:101)

Original: L ( (cid:101) ν ) holds.
Revised: L ( (cid:101) ν ) holds.

Original: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .
Revised: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .

Original: Proof.
Revised: Proof.

Original: First of all, from the fact that log( x + 1) ≤ x for all x > − 1 , we can easily concludethat min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) .
Revised: First of all, from the fact that log( x + 1) ≤ x for all x > − 1 , we can easily concludethat min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) .

Original: Now, we will show that min ν ˆ L ( ν ) ≤ min ν ˜ L ( ν ) .
Revised: Now, we will show that min ν ˆ L ( ν ) ≤ min ν ˜ L ( ν ) .

Original: For givenν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:
Revised: For givenν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Then, from the following two equations:ˆ L ( ν ∗ + C )
Revised: Then, from the following two equations:ˆ L ( ν ∗ + C )

Original: [Equation]
Revised: [Equation]

Original: and˜
Revised: and˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we can conclude that
Revised: we can conclude that

Original: [Equation]
Revised: [Equation]

Original: From the Lemma F.1, we obtain
Revised: From the Lemma F.1, we obtain

Original: [Equation]
Revised: [Equation]

Original: We show that min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) and min ν ˆ L ( ν )
Revised: We show that min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) and min ν ˆ L ( ν )

Original: ≤ min ν ˜ L ( ν ) , so min ν ˆ L ( ν ) =
Revised: ≤ min ν ˜ L ( ν ) , so min ν ˆ L ( ν ) =

Original: min ν ˜ L ( ν ) .
Revised: min ν ˜ L ( ν ) .

Original: Finally, similar to the proof steps for Proposition 3.2, we can easily show that ˆ L FD ( ν ) is convex w.r.t.ν (Remark that log-sum-exp is a convex function).
Revised: Finally, similar to the proof steps for Proposition 3.2, we can easily show that ˆ L FD ( ν ) is convex w.r.t.ν (Remark that log-sum-exp is a convex function).

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: Let (cid:98) V and (cid:101)
Revised: Let (cid:98) V and (cid:101)

Original: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.
Revised: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.

Original: Then,
Revised: Then,

Original: } holds.
Revised: } holds.

Original: Proof.
Revised: Proof.

Original: We will prove this proposition by showing ˜ V ⊆ { ν ∗ + C ′ | ν ∗ ∈
Revised: We will prove this proposition by showing ˜ V ⊆ { ν ∗ + C ′ | ν ∗ ∈

Original: ˆ V , C ∈ R } and ˜ V ⊇{ ν ∗ + C ′ | ν ∗ ∈
Revised: ˆ V , C ∈ R } and ˜ V ⊇{ ν ∗ + C ′ | ν ∗ ∈

Original: ˆ V , C ∈ R } .
Revised: ˆ V , C ∈ R } .

Original: ( ⊆ )
Revised: ( ⊆ )

Original: For given ν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:
Revised: For given ν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Then, by the proof steps of Proposition 3.4, we obtain
Revised: Then, by the proof steps of Proposition 3.4, we obtain

Original: [Equation]
Revised: [Equation]

Original: It means ( ν ∗ + C ) ∈
Revised: It means ( ν ∗ + C ) ∈

Original: arg min
Revised: arg min

Original: ν ˆ L ( ν ) and thus,
Revised: ν ˆ L ( ν ) and thus,

Original: [Equation]
Revised: [Equation]

Original: i.e.,
Revised: i.e.,

Original: [Equation]
Revised: [Equation]

Original: ( ⊇ ) Let ˆ ν ∗ ∈ arg min
Revised: ( ⊇ ) Let ˆ ν ∗ ∈ arg min

Original: ν ˆ L ( ν ) .
Revised: ν ˆ L ( ν ) .

Original: Then,d
Revised: Then,d

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: Let C ∈ R .
Revised: Let C ∈ R .

Original: Then, we derive the following equation:
Revised: Then, we derive the following equation:

Original: [Equation]
Revised: [Equation]

Original: Here, we apply Lemma F.1 to derive the first equality.
Revised: Here, we apply Lemma F.1 to derive the first equality.

Original: Becausemin
Revised: Becausemin

Original: [Equation]
Revised: [Equation]

Original: we can conclude that L FD (ˆ ν ∗ + C ) =
Revised: we can conclude that L FD (ˆ ν ∗ + C ) =

Original: min
Revised: min

Original: ν
Revised: ν

Original: L FD ( ν ) , i.e., (ˆ ν ∗ + C ) ∈ arg min ν ˜ L ( ν ) .
Revised: L FD ( ν ) , i.e., (ˆ ν ∗ + C ) ∈ arg min ν ˜ L ( ν ) .

Original: Consequently,arg min ν
Revised: Consequently,arg min ν

Original: [Equation]
Revised: [Equation]

Original: dual formulation
Revised: dual formulation

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then we can provide following proposition:
Revised: Then we can provide following proposition:

Original: Proposition F.2.
Revised: Proposition F.2.

Original: We can rewrite the optimization problem (6-8) as
Revised: We can rewrite the optimization problem (6-8) as

Original: [Equation]
Revised: [Equation]

Original: Then, the dual problem of (50) is given by
Revised: Then, the dual problem of (50) is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first define following three functions
Revised: We first define following three functions

Original: [Equation]
Revised: [Equation]

Original: and conjugate functions,
Revised: and conjugate functions,

Original: [Equation]
Revised: [Equation]

Original: Then, the dual formulation of the primal (50) can be derived as follows:
Revised: Then, the dual formulation of the primal (50) can be derived as follows:

Original: [Equation]
Revised: [Equation]

Original: | s,a )
Revised: | s,a )

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = max d ≥ 0 − f
Revised: = max d ≥ 0 − f

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: B ) ∗ d (cid:1)
Revised: B ) ∗ d (cid:1)

Original: [Equation]
Revised: [Equation]

Original: E ( s
Revised: E ( s

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: s ′
Revised: s ′

Original: [Equation]
Revised: [Equation]

Original: T ( ·
Revised: T ( ·

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: log d
Revised: log d

Original: [Equation]
Revised: [Equation]

Original: ¯ T
Revised: ¯ T

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:125)
Revised: (cid:125)

Original: [Equation]
Revised: [Equation]

Original: = r
Revised: = r

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: (cid:21) − αD KL ( d ∥ d I ) = max d ≥ 0 − f (cid:0)
Revised: (cid:21) − αD KL ( d ∥ d I ) = max d ≥ 0 − f (cid:0)

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: d (cid:1)
Revised: d (cid:1)

Original: [Equation]
Revised: [Equation]

Original: g ( ¯
Revised: g ( ¯

Original: [Equation]
Revised: [Equation]

Original: r )
Revised: r )

Original: [Equation]
Revised: [Equation]

Original: αD KL ( d ∥ d I ) =
Revised: αD KL ( d ∥ d I ) =

Original: max d ≥
Revised: max d ≥

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: − (cid:26) max µ
Revised: − (cid:26) max µ

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: ( µ ; r ) + ⟨ ¯ T ∗ d, µ ⟩ (cid:27) − αD KL ( d ∥ d I ) = max d ≥ 0 min µ − f
Revised: ( µ ; r ) + ⟨ ¯ T ∗ d, µ ⟩ (cid:27) − αD KL ( d ∥ d I ) = max d ≥ 0 min µ − f

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: ( µ
Revised: ( µ

Original: [Equation]
Revised: [Equation]

Original: r ) − ⟨ ¯
Revised: r ) − ⟨ ¯

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: − αD KL ( d ∥ d I ) = max d ≥ 0 min µ − f (cid:0) − ( γ T
Revised: − αD KL ( d ∥ d I ) = max d ≥ 0 min µ − f (cid:0) − ( γ T

Original: [Equation]
Revised: [Equation]

Original: B ) ∗ d (cid:1)
Revised: B ) ∗ d (cid:1)

Original: [Equation]
Revised: [Equation]

Original: g ∗ (
Revised: g ∗ (

Original: [Equation]
Revised: [Equation]

Original: r ) − ⟨ d
Revised: r ) − ⟨ d

Original: [Equation]
Revised: [Equation]

Original: ¯ T µ ⟩ − αD KL ( d ∥ d I ) = max d ≥ 0 min µ − f (cid:0) − ( γ T
Revised: ¯ T µ ⟩ − αD KL ( d ∥ d I ) = max d ≥ 0 min µ − f (cid:0) − ( γ T

Original: [Equation]
Revised: [Equation]

Original: B ) ∗ d (cid:1) + g ∗ ( µ
Revised: B ) ∗ d (cid:1) + g ∗ ( µ

Original: [Equation]
Revised: [Equation]

Original: r ) − h
Revised: r ) − h

Original: ( d
Revised: ( d

Original: [Equation]
Revised: [Equation]

Original: µ )
Revised: µ )

Original: = max d ≥ 0 min µ − (cid:26) max ν
Revised: = max d ≥ 0 min µ − (cid:26) max ν

Original: ⟨− ( γ T
Revised: ⟨− ( γ T

Original: [Equation]
Revised: [Equation]

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we can reorder the maximin to minimax and therefore
Revised: we can reorder the maximin to minimax and therefore

Original: [Equation]
Revised: [Equation]

Original: we can derive the
Revised: we can derive the

Original: [Equation]
Revised: [Equation]

Original: We can rewrite the last term asmin ν,µ (1 − γ )
Revised: We can rewrite the last term asmin ν,µ (1 − γ )

Original: E s ∼
Revised: E s ∼

Original: [Equation]
Revised: [Equation]

Original: [ ν ( s )]
Revised: [ ν ( s )]

Original: + log E
Revised: + log E

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: a ) (cid:17)(cid:105) .
Revised: a ) (cid:17)(cid:105) .

Original: Lemma F.3.
Revised: Lemma F.3.

Original: For given function µ : S × S → R and ν : S → R ,
Revised: For given function µ : S × S → R and ν : S → R ,

Original: [Equation]
Revised: [Equation]

Original: , ν
Revised: , ν

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Finally, we can show that the relation between arg min µ,ν L and arg min µ,ν L FD :
Revised: Finally, we can show that the relation between arg min µ,ν L and arg min µ,ν L FD :

Original: Proposition F.4.
Revised: Proposition F.4.

Original: Let V and V FD be the set of optimal solutions ( µ ∗ ,
Revised: Let V and V FD be the set of optimal solutions ( µ ∗ ,

Original: ν ∗ ) ofarg min µ,ν L ( w ν,µ , ¯ w ν,µ , µ, ν ) and arg min µ,ν L FD ( µ, ν ) , respectively.
Revised: ν ∗ ) ofarg min µ,ν L ( w ν,µ , ¯ w ν,µ , µ, ν ) and arg min µ,ν L FD ( µ, ν ) , respectively.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Proof.
Revised: Proof.

Original: For brevity, we will denote L ( w ν,µ , ¯ w ν,µ , µ, ν ) as L ( µ, ν ) .
Revised: For brevity, we will denote L ( w ν,µ , ¯ w ν,µ , µ, ν ) as L ( µ, ν ) .

Original: ( ⊆ )
Revised: ( ⊆ )

Original: For given (ˆ µ ∗ , ˆ ν ∗ ) ∈ arg min µ,ν L FD ( µ, ν ) , we define two constants as follows:
Revised: For given (ˆ µ ∗ , ˆ ν ∗ ) ∈ arg min µ,ν L FD ( µ, ν ) , we define two constants as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, from the following two equations:
Revised: Then, from the following two equations:

Original: [Equation]
Revised: [Equation]

Original: µ ∗ + C, ˆ ν ∗ + C ′ )
Revised: µ ∗ + C, ˆ ν ∗ + C ′ )

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E s ∼ p 

Revised: E s ∼ p

Original: [ˆ ν ∗ ( s ) + C ′ ]
Revised: [ˆ ν ∗ ( s ) + C ′ ]

Original: + E ( s,a,s ′ )
Revised: + E ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + C
Revised: + C

Original: + r ( s, s ′ ) − 1)]
Revised: + r ( s, s ′ ) − 1)]

Original: + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E s ′
Revised: + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: + γC
Revised: + γC

Original: ′ − ˆ µ ∗ ( s, s ′ )
Revised: ′ − ˆ µ ∗ ( s, s ′ )

Original: − C ] − ˆ ν ∗ ( s ′ )
Revised: − C ] − ˆ ν ∗ ( s ′ )

Original: − C ′ (cid:1) − 1 (cid:17)(cid:105) =
Revised: − C ′ (cid:1) − 1 (cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s )]
Revised: [ˆ ν ∗ ( s )]

Original: + (1 − γ ) C ′
Revised: + (1 − γ ) C ′

Original: + E ( s,a,s ′ )
Revised: + E ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + r ( s, s ′ ) − 1)] exp( C ) + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E
Revised: + r ( s, s ′ ) − 1)] exp( C ) + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E

Original: s ′
Revised: s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: − ˆ µ ∗ ( s, s ′ )]
Revised: − ˆ µ ∗ ( s, s ′ )]

Original: − ˆ ν ∗ ( s ′ )
Revised: − ˆ ν ∗ ( s ′ )

Original: (cid:1) − 1 (cid:17)(cid:105) exp (cid:16) 1 α (cid:0) − (1 − γ ) C ′ − C
Revised: (cid:1) − 1 (cid:17)(cid:105) exp (cid:16) 1 α (cid:0) − (1 − γ ) C ′ − C

Original: (cid:1)(cid:17) =
Revised: (cid:1)(cid:17) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s )]
Revised: [ˆ ν ∗ ( s )]

Original: + (1 − γ ) C ′
Revised: + (1 − γ ) C ′

Original: +
Revised: +

Original: + α, and
Revised: + α, and

Original: L FD (ˆ µ ∗ + C, ˆ ν ∗ + C ′ )
Revised: L FD (ˆ µ ∗ + C, ˆ ν ∗ + C ′ )

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E s ∼ p 

Revised: E s ∼ p

Original: [ˆ ν ∗ ( s ) + C ′ ]
Revised: [ˆ ν ∗ ( s ) + C ′ ]

Original: + log E
Revised: + log E

Original: ( s,a,s ′ )
Revised: ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + C
Revised: + C

Original: + r ( s, s ′ ))]
Revised: + r ( s, s ′ ))]

Original: + α log E ( s,a ) ∼ d I (cid:104) exp (cid:16) 

Revised: + α log E ( s,a ) ∼ d I (cid:104) exp (cid:16)

Original: α (cid:0) E s ′
Revised: α (cid:0) E s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: + γC
Revised: + γC

Original: ′ − ˆ µ ∗ ( s, s ′ )
Revised: ′ − ˆ µ ∗ ( s, s ′ )

Original: − C ] − ˆ ν ∗ ( s ′ )
Revised: − C ] − ˆ ν ∗ ( s ′ )

Original: − C ′ (cid:1)(cid:17)(cid:105) =
Revised: − C ′ (cid:1)(cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s )]
Revised: [ˆ ν ∗ ( s )]

Original: + (1 − γ ) C ′ + log E
Revised: + (1 − γ ) C ′ + log E

Original: ( s,a,s ′ )
Revised: ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + r ( s, s ′ ))]
Revised: + r ( s, s ′ ))]

Original: + C
Revised: + C

Original: + α log E ( s,a ) ∼ d I (cid:104) exp (cid:16) 

Revised: + α log E ( s,a ) ∼ d I (cid:104) exp (cid:16)

Original: α (cid:0) E s ′
Revised: α (cid:0) E s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: − ˆ µ ∗ ( s, s ′ )]
Revised: − ˆ µ ∗ ( s, s ′ )]

Original: − ˆ ν ∗ ( s ′ ) (cid:1)(cid:17)(cid:105) + ( γC ′ − C − C ′ ) =
Revised: − ˆ ν ∗ ( s ′ ) (cid:1)(cid:17)(cid:105) + ( γC ′ − C − C ′ ) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s ) + C ′ ]
Revised: [ˆ ν ∗ ( s ) + C ′ ]

Original: + 1 + α, we can conclude that
Revised: + 1 + α, we can conclude that

Original: [Equation]
Revised: [Equation]

Original: µ,
Revised: µ,

Original: [Equation]
Revised: [Equation]

Original: It means (ˆ µ ∗ + C, ˆ ν ∗ + C ′ )
Revised: It means (ˆ µ ∗ + C, ˆ ν ∗ + C ′ )

Original: ∈ arg min µ,ν L ( µ, ν ) and thus,
Revised: ∈ arg min µ,ν L ( µ, ν ) and thus,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( ⊇ ) Let ( µ ∗ , ν ∗ ) ∈ arg min µ,ν L ( µ, ν ) .
Revised: ( ⊇ ) Let ( µ ∗ , ν ∗ ) ∈ arg min µ,ν L ( µ, ν ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Let C ∈ R and C ′ ∈ R .
Revised: Let C ∈ R and C ′ ∈ R .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we derive the following equation:
Revised: we derive the following equation:

Original: [Equation]
Revised: [Equation]

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we apply Lemma F
Revised: we apply Lemma F

Original: [Equation]
Revised: [Equation]

Original: 1 to derive the first equality.
Revised: 1 to derive the first equality.

Original: Because
Revised: Because

Original: [Equation]
Revised: [Equation]

Original: + α E ( s,a ) ∼ d
Revised: + α E ( s,a ) ∼ d

Original: I [ w ∗ µ ∗ ,ν ∗ ( s, a )]
Revised: I [ w ∗ µ ∗ ,ν ∗ ( s, a )]

Original: = (1 − γ ) E s ∼ p 

Revised: = (1 − γ ) E s ∼ p

Original: [ ν ∗ ( s )]
Revised: [ ν ∗ ( s )]

Original: + E ( s,a,s ′ ) ∼ d ∗ [1] + α E ( s,a ) ∼ d ∗ [1] =
Revised: + E ( s,a,s ′ ) ∼ d ∗ [1] + α E ( s,a ) ∼ d ∗ [1] =

Original: (1 − γ )
Revised: (1 − γ )

Original: E s ∼ p 

Revised: E s ∼ p

Original: [ ν ∗ ( s )]
Revised: [ ν ∗ ( s )]

Original: + 1 + α, we can conclude that L FD ( µ ∗ + C, ν ∗ + C ′ ) =
Revised: + 1 + α, we can conclude that L FD ( µ ∗ + C, ν ∗ + C ′ ) =

Original: min
Revised: min

Original: µ,ν L FD ( µ, ν ) , i.e., ( µ ∗ + C, ν ∗ + C ′ ) ∈arg
Revised: µ,ν L FD ( µ, ν ) , i.e., ( µ ∗ + C, ν ∗ + C ′ ) ∈arg

Original: min µ,ν L FD ( µ, ν ) .
Revised: min µ,ν L FD ( µ, ν ) .

Original: It means,
Revised: It means,

Original: [Equation]
Revised: [Equation]

Original: Remark that the proposed objective (51) is stable and the aforementioned proposition allows us touse self-normalized weighted importance sampling to extract policy.
Revised: Remark that the proposed objective (51) is stable and the aforementioned proposition allows us touse self-normalized weighted importance sampling to extract policy.

Original: However, as we discussed in
Revised: However, as we discussed in

Original: Section 3.5, using µ is main bottleneck for the overall optimization and optimizing ˜ L ( ν ) shows betterperformance in practice.
Revised: Section 3.5, using µ is main bottleneck for the overall optimization and optimizing ˜ L ( ν ) shows betterperformance in practice.

Original: G Pseudocode of LobsDICEG.1 Tabular LobsDICE
Revised: G Pseudocode of LobsDICEG.1 Tabular LobsDICE

Original: For tabular MDPs, we first construct a maximum-likelihood estimation (MLE)
Revised: For tabular MDPs, we first construct a maximum-likelihood estimation (MLE)

Original: MDP
Revised: MDP

Original: ˆ M =⟨ S, A, ˆ T, ˆ p 0 , γ ⟩ using an offline dataset D I .
Revised: ˆ M =⟨ S, A, ˆ T, ˆ p 0 , γ ⟩ using an offline dataset D I .

Original: Then, tabular LobsDICE solves (16) on the MLE
Revised: Then, tabular LobsDICE solves (16) on the MLE

Original: MDP ˆ M .
Revised: MDP ˆ M .

Original: In tabular case, note that (cid:98) d
Revised: In tabular case, note that (cid:98) d

Original: I ( s, a ) , (cid:98) d
Revised: I ( s, a ) , (cid:98) d

Original: I ( s, s ′ ) , and (cid:98) d E ( s, s ′ ) are explicitly accessible bynormalized counts of ( s, a ) and ( s, s ′ ) in the datasets.
Revised: I ( s, s ′ ) , and (cid:98) d E ( s, s ′ ) are explicitly accessible bynormalized counts of ( s, a ) and ( s, s ′ ) in the datasets.

Original: In addition, (cid:98) r ( s, s ′ )
Revised: In addition, (cid:98) r ( s, s ′ )

Original: = (cid:98)
Revised: = (cid:98)

Original: d E ( s,s ′ ) (cid:98) d
Revised: d E ( s,s ′ ) (cid:98) d

Original: I ( s,s ′ ) is the logratio between two empirical distributions.
Revised: I ( s,s ′ ) is the logratio between two empirical distributions.

Original: Also, ν ∈ R | S | is represented as a | S | -dimensional vector,and µ ∈ R | S |×| S | is represented as a | S | 2 -dimensional vector, and we solve the following convexminimization problem for ( µ, ν ) :
Revised: Also, ν ∈ R | S | is represented as a | S | -dimensional vector,and µ ∈ R | S |×| S | is represented as a | S | 2 -dimensional vector, and we solve the following convexminimization problem for ( µ, ν ) :

Original: [Equation]
Revised: [Equation]

Original: This process is summarized in Algorithm 1.
Revised: This process is summarized in Algorithm 1.

Original: G.2 Practical LobsDICE (with function approximation)
Revised: G.2 Practical LobsDICE (with function approximation)

Original: To deal with continuous or large MDPs, we represent our optimization variable ν and discriminator cas neural networks, parameterized by θ and ϕ respectively:
Revised: To deal with continuous or large MDPs, we represent our optimization variable ν and discriminator cas neural networks, parameterized by θ and ϕ respectively:

Original: ν θ : S → R is an MLP that takes a stateas an input and outputs a scalar value, and c ϕ : S × S → [0 , 1] is defined similarly.
Revised: ν θ : S → R is an MLP that takes a stateas an input and outputs a scalar value, and c ϕ : S × S → [0 , 1] is defined similarly.

Original: For the policyπ ψ , we use a tanh-squashed Gaussian policy, where the parameters of a Gaussian distribution (i.e.mean and covariance) are output by the neural network.
Revised: For the policyπ ψ , we use a tanh-squashed Gaussian policy, where the parameters of a Gaussian distribution (i.e.mean and covariance) are output by the neural network.

Original: The parameters of c ϕ are trained by:
Revised: The parameters of c ϕ are trained by:

Original: [Equation]
Revised: [Equation]

Original: which is analogous to (13).
Revised: which is analogous to (13).

Original: The parameters of ν θ are trained by:
Revised: The parameters of ν θ are trained by:

Original: [Equation]
Revised: [Equation]

Original: where ˆ A θ ( s, a, s ′ ) =
Revised: where ˆ A θ ( s, a, s ′ ) =

Original: r ϕ ( s, s ′ )
Revised: r ϕ ( s, s ′ )

Original: + γν θ ( s ′ )
Revised: + γν θ ( s ′ )

Original: − ν θ ( s ) and r ϕ ( s, s ′ )
Revised: − ν θ ( s ) and r ϕ ( s, s ′ )

Original: = − log (cid:16) 1 c ϕ ( s,s ′ ) − 1 (cid:17) , and this isanalogous to (23).
Revised: = − log (cid:16) 1 c ϕ ( s,s ′ ) − 1 (cid:17) , and this isanalogous to (23).

Original: Due to the logarithm outside the expectation in the second term, the mini-batch approximation would introduce additional bias in gradient estimation, but we found that using thebiased gradient estimate worked well in practice with moderately large batch size (e.g. 256).
Revised: Due to the logarithm outside the expectation in the second term, the mini-batch approximation would introduce additional bias in gradient estimation, but we found that using thebiased gradient estimate worked well in practice with moderately large batch size (e.g. 256).

Original: Finally,the parameters of π ψ are trained by:
Revised: Finally,the parameters of π ψ are trained by:

Original: [Equation]
Revised: [Equation]

Original: which is analogous to (24).
Revised: which is analogous to (24).

Original: Here, the self-normalized importance sampling may introduce an additional bias with mini-batch approximations, but it is well known that the self-normalized importancesampling provides a consistent estimator [31].
Revised: Here, the self-normalized importance sampling may introduce an additional bias with mini-batch approximations, but it is well known that the self-normalized importancesampling provides a consistent estimator [31].

Original: Also, it worked well in practice with moderately largebatch size (e.g. 256) in our experiments.
Revised: Also, it worked well in practice with moderately largebatch size (e.g. 256) in our experiments.

Original: We optimize the parameters ( ϕ, θ, ψ ) jointly in practice.
Revised: We optimize the parameters ( ϕ, θ, ψ ) jointly in practice.

Original: Weare not using a target network at all.
Revised: Weare not using a target network at all.

Original: The overall training process is summarized in Algorithm 2.
Revised: The overall training process is summarized in Algorithm 2.

Original: Algorithm 2 LobsDICE (with function approximation)
Revised: Algorithm 2 LobsDICE (with function approximation)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: ∇ ψ J π ( ψ ) (Eq. (55)) 8: end for
Revised: ∇ ψ J π ( ψ ) (Eq. (55)) 8: end for

Original: Output: The imitation policy π ψ .
Revised: Output: The imitation policy π ψ .

Original: H Additional Experiments
Revised: H Additional Experiments

Original: [Figure]
Revised: [Figure]

Original: H.1 Subsampled expert demonstrations
Revised: H.1 Subsampled expert demonstrations

Original: [Figure]
Revised: [Figure]

Original: Existing works on imitation learning [11] conduct experiments where the expert demonstrations aresubsampled.
Revised: Existing works on imitation learning [11] conduct experiments where the expert demonstrations aresubsampled.

Original: The following figure presents the result when given the subsampled expert demonstrations.
Revised: The following figure presents the result when given the subsampled expert demonstrations.

Original: LobsDICE still significantly outperforms the baseline algorithms even with the subsampledexpert demonstrations 3 .
Revised: LobsDICE still significantly outperforms the baseline algorithms even with the subsampledexpert demonstrations 3 .

Original: [Figure]
Revised: [Figure]

Original: 710 H.2 Error of inverse dynamics model in MuJoCo domains
Revised: 710 H.2 Error of inverse dynamics model in MuJoCo domains

Original: In principle, IDM trained with arbitrary demonstrations should be able to predict the expert’s missingactions accurately in deterministic MDPs.
Revised: In principle, IDM trained with arbitrary demonstrations should be able to predict the expert’s missingactions accurately in deterministic MDPs.

Original: However, in Section 5.2, we observed that the empiricalperformance of BCO and DemoDICEfO tends to degrade as the number of imperfect demonstrationsincreases (see Figure 2).
Revised: However, in Section 5.2, we observed that the empiricalperformance of BCO and DemoDICEfO tends to degrade as the number of imperfect demonstrationsincreases (see Figure 2).

Original: This is due to the use of function approximation for IDM: given thatthe expressive power of a function approximator is limited, the more data unrelated to the expertdemonstrations (i
Revised: This is due to the use of function approximation for IDM: given thatthe expressive power of a function approximator is limited, the more data unrelated to the expertdemonstrations (i

Original: [Figure]
Revised: [Figure]

Original: e. medium-v2 and random-v2 ) is used for training, the more the predictionaccuracy for the expert data would be adversely affected.
Revised: e. medium-v2 and random-v2 ) is used for training, the more the predictionaccuracy for the expert data would be adversely affected.

Original: Figure 6 shows that the mean squarederror of IDM for expert demonstrations D E increases as the number of non-expert demonstrationsincreases
Revised: Figure 6 shows that the mean squarederror of IDM for expert demonstrations D E increases as the number of non-expert demonstrationsincreases

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H.3 Fewer expert demonstrations
Revised: H.3 Fewer expert demonstrations

Original: In this section, we provide additional experiments with fewer expert demonstrations.
Revised: In this section, we provide additional experiments with fewer expert demonstrations.

Original: Specifically, wetested LobsDICE when the number of state-only expert demonstrations is 5, 3, and 1, respectively.
Revised: Specifically, wetested LobsDICE when the number of state-only expert demonstrations is 5, 3, and 1, respectively.

Original: The Figure H.2 shows that LobsDICE (#exp=3) and LobsDICE (#exp=1) still perform robustly.
Revised: The Figure H.2 shows that LobsDICE (#exp=3) and LobsDICE (#exp=1) still perform well.

Original: [Figure]
Revised: [Figure]

Original: I Experimental Details
Revised: I Experimental Details

Original: I.1 Experimental details for random MDPs
Revised: I.1 Experimental details for random MDPs

Original: Random MDP generation For MDP M = ⟨ S, A, T, R, γ, p 0 ⟩ , we first set | S | = 20 , | A | = 4 , γ =0 .
Revised: Random MDP generation For MDP M = ⟨ S, A, T, R, γ, p 0 ⟩ , we first set | S | = 20 , | A | = 4 , γ =0 .

Original: 95 , p 0 ( s ) =
Revised: 95 , p 0 ( s ) =

Original: 1 for a fixed s
Revised: 1 for a fixed s

Original: = s 0 .
Revised: = s 0 .

Original: For each ( s, a ) , we samples four different state { s ′ 1 , s ′ 2 , s ′ 3 , s ′ 4 } .
Revised: For each ( s, a ) , we samples four different state { s ′ 1 , s ′ 2 , s ′ 3 , s ′ 4 } .

Original: Then, we set the transition probability ( T ( s ′ 1 | s, a ) , T ( s ′ 2 | s, a ) , T ( s ′ 3 | s, a ) , T ( s ′ 4 | s, a )) =
Revised: Then, we set the transition probability ( T ( s ′ 1 | s, a ) , T ( s ′ 2 | s, a ) , T ( s ′ 3 | s, a ) , T ( s ′ 4 | s, a )) =

Original: (1 − β ) X
Revised: (1 − β ) X

Original: +βY , where X ∼ Categorical (0 . 25 , 0 .
Revised: +βY , where X ∼ Categorical (0 . 25 , 0 .

Original: 25 , 0 . 25) and Y ∼ Dir (1 , 1 , 1 , 1) .
Revised: 25 , 0 . 25) and Y ∼ Dir (1 , 1 , 1 , 1) .

Original: β ∈ [0 , 1] is thehyperparameter to control transition stochasticity:
Revised: β ∈ [0 , 1] is thehyperparameter to control transition stochasticity:

Original: when β = 0 , the transition probability becomesone-hot vector, i.e., deterministic MDP.
Revised: when β = 0 , the transition probability becomesone-hot vector, i.e., deterministic MDP.

Original: In contrast, when β = 1 , the transition of MDP becomesstochastic.
Revised: In contrast, when β = 1 , the transition of MDP becomesstochastic.

Original: Finally, the reward of 1 is only given to a state that minimizes the optimal value at initialstate s 0 ; other states have zero rewards.
Revised: Finally, the reward of 1 is only given to a state that minimizes the optimal value at initialstate s 0 ; other states have zero rewards.

Original: Offline dataset generation For each random MDP M , we generate state-only expertdemonstrations by executing a (softmax) expert policy and state-action imperfect demonstrations by a uniform random policy.
Revised: Offline dataset generation For each random MDP M , we generate state-only expertdemonstrations by executing a (softmax) expert policy and state-action imperfect demonstrations by a uniform random policy.

Original: We perform experiments for a varying number ofexpert demonstrations N E ∈ { 10 , 100 , 1000 , 1000 } and imperfect demonstrations N I ∈{ 1 , 3 , 10 , 30 , 100 , 300 , 1000 , 3000 , 10000 } .
Revised: We perform experiments for a varying number ofexpert demonstrations N E ∈ { 10 , 100 , 1000 , 1000 } and imperfect demonstrations N I ∈{ 1 , 3 , 10 , 30 , 100 , 300 , 1000 , 3000 , 10000 } .

Original: Hyperparameters We compare our tabular LobsDICE with BC, BCO, DemoDICEfO, and OPOLO.
Revised: Hyperparameters We compare our tabular LobsDICE with BC, BCO, DemoDICEfO, and OPOLO.

Original: For KL-regularization hyperparameters of DemoDICEfO and LobsDICE, we use α = 0 . 1 .
Revised: For KL-regularization hyperparameters of DemoDICEfO and LobsDICE, we use α = 0 . 1 .

Original: I.2 Experimental details for MuJoCo control tasks
Revised: I.2 Experimental details for MuJoCo control tasks

Original: Hyperparameters
Revised: Hyperparameters

Original: [Table]
Revised: [Table]

Original: For fair comparison, we use the same learning rate to train actors of BC, BCO, DemoDICEfO
Revised: For fair comparison, we use the same learning rate to train actors of BC, BCO, DemoDICEfO

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: LobsDICE.
Revised: LobsDICE.

Original: We implement our network architectures for BC, BCO, DemoDICEfO
Revised: We implement our network architectures for BC, BCO, DemoDICEfO

Original: [Table]
Revised: [Table]

Original: and LobsDICEbased on the implementation of OptiDICE 4 .
Revised: and LobsDICEbased on the implementation of OptiDICE 4 .

Original: For OPOLO, we use its official implmentation 5 .
Revised: For OPOLO, we use its official implmentation 5 .

Original: Wehave tried hyperparameter tuning for OPOLO but never obtained a successful learning curve, showing numerical instability due to using out-of-distribution action values.
Revised: Wehave tried hyperparameter tuning for OPOLO but never obtained a successful learning curve, showing numerical instability due to using out-of-distribution action values.

Original: Vanilla online off-policyalgorithms commonly fail in the offline learning setting.
Revised: Vanilla online off-policyalgorithms commonly fail in the offline learning setting.

Original: Therefore, we report the results of officialimplementation without any modification to network architectures or hyperparameters.
Revised: Therefore, we report the results of officialimplementation without any modification to network architectures or hyperparameters.

Original: For stablediscriminator learning, we use gradient penalty regularization on the r ( s, a ) and r ( s, s ′ ) functions,which was proposed in [10] to enforce 1-Lipschitz constraint.
Revised: For stablediscriminator learning, we use gradient penalty regularization on the r ( s, a ) and r ( s, s ′ ) functions,which was proposed in [10] to enforce 1-Lipschitz constraint.

Original: To stabilize critic training, we addgradient L2-norm to the critic loss for the regularization.
Revised: To stabilize critic training, we addgradient L2-norm to the critic loss for the regularization.

Original: In addition, we use every state s in D I to define D 0 , i.e., D 

Revised: In addition, we use every state s in D I to define D 0 , i.e., D

Original: =
Revised: =

Original: { s | ( s, a, s ′ )
Revised: { s | ( s, a, s ′ )

Original: ∈
Revised: ∈

Original: D I } , following ValueDICE [17].
Revised: D I } , following ValueDICE [17].

Original: Detailed hyperparameterconfigurations used for our main experiments are summarized in Table 1.
Revised: Detailed hyperparameterconfigurations used for our main experiments are summarized in Table 1.

Original: Evaluation metric For each environment, the normalized score is measured by 100 ×score − random score expert score − random score , where the expert score and random score are average returns of trajectories in expert-v2 and random-v2 , respectively.
Revised: Evaluation metric For each environment, the normalized score is measured by 100 ×score − random score expert score − random score , where the expert score and random score are average returns of trajectories in expert-v2 and random-v2 , respectively.

Original: J Generalization to γ = 1
Revised: J Generalization to γ = 1

Original: In this section, we generalize LobsDICE for discounted MDPs ( γ < 1 ) to undiscounted MDPs( γ = 1 ).
Revised: In this section, we generalize LobsDICE for discounted MDPs ( γ < 1 ) to undiscounted MDPs( γ = 1 ).

Original: Suppose that the stationary distributions d and ¯ d satisfy the Bellman flow constraints (7) andmarginalization constraints (8), respectively.
Revised: Suppose that the stationary distributions d and ¯ d satisfy the Bellman flow constraints (7) andmarginalization constraints (8), respectively.

Original: When γ = 1 , for any constant c ≥ 0 , cd and c ¯ d alsosatisfy the corresponding constraints.
Revised: When γ = 1 , for any constant c ≥ 0 , cd and c ¯ d alsosatisfy the corresponding constraints.

Original: It means that the original constrained optimization problem ((6)-(8)) for the stationary distributionsd and ¯ d is an ill-posed problem.
Revised: It means that the original constrained optimization problem ((6)-(8)) for the stationary distributionsd and ¯ d is an ill-posed problem.

Original: We tackle this ill-posedness by adding additional normalizationconstraint (cid:80) s,a d ( s, a ) = 1 to (6-8):
Revised: We tackle this ill-posedness by adding additional normalizationconstraint (cid:80) s,a d ( s, a ) = 1 to (6-8):

Original: [Equation]
Revised: [Equation]

Original: Using a derivation similar to that from (9) to (11), we obtain the following min-max optimization forw, ¯ w, µ, ν , and λ :
Revised: Using a derivation similar to that from (9) to (11), we obtain the following min-max optimization forw, ¯ w, µ, ν , and λ :

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R is the Lagrange multiplier for the normalization constraint (cid:80) s,a d ( s, a ) = 1 .
Revised: where λ ∈ R is the Lagrange multiplier for the normalization constraint (cid:80) s,a d ( s, a ) = 1 .

Original: Similar to
Revised: Similar to

Original: Proposition 3.1, we can derive a closed-form solution to the inner maximization of (56):w µ,ν,λ ( s, a ) =
Revised: Proposition 3.1, we can derive a closed-form solution to the inner maximization of (56):w µ,ν,λ ( s, a ) =

Original: exp (cid:0) 1 α e µ,ν,λ ( s, a ) − 1 (cid:1) and ¯ w µ ( s, s ′ ) = exp( r ( s, s ′ )
Revised: exp (cid:0) 1 α e µ,ν,λ ( s, a ) − 1 (cid:1) and ¯ w µ ( s, s ′ ) = exp( r ( s, s ′ )

Original: + µ ( s, s ′ )
Revised: + µ ( s, s ′ )

Original: − 1) .
Revised: − 1) .

Original: Then, we can reduce the nested min-max optimization of (56) to a single minimization by pluggingthe closed-form solution ( w µ,ν,λ , ¯ w µ ) into L ( w, ¯ w, µ, ν, λ ) :
Revised: Then, we can reduce the nested min-max optimization of (56) to a single minimization by pluggingthe closed-form solution ( w µ,ν,λ , ¯ w µ ) into L ( w, ¯ w, µ, ν, λ ) :

Original: [Equation]
Revised: [Equation]

Original: In practice, we estimate L ( w µ,ν,λ , ¯ w µ , µ, ν, λ )) using samples from distribution d I .
Revised: In practice, we estimate L ( w µ,ν,λ , ¯ w µ , µ, ν, λ )) using samples from distribution d I .

Original: To derive asample-based objective, we use analogous derivation in Section 3.5.
Revised: To derive asample-based objective, we use analogous derivation in Section 3.5.

Original: Sample-based objective can berepresented as
Revised: Sample-based objective can berepresented as

Original: [Equation]
Revised: [Equation]

Original: where ˆ e µ,ν,λ ( s, a, s ′ )
Revised: where ˆ e µ,ν,λ ( s, a, s ′ )

Original: = − µ ( s, s ′ )
Revised: = − µ ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) − λ .
Revised: − ν ( s ) − λ .

Original: Then, the closed-form solution to theminimization (57) with respect to µ is:
Revised: Then, the closed-form solution to theminimization (57) with respect to µ is:

Original: [Equation]
Revised: [Equation]

Original: Using the above solution, we obtain the following minimization problem:min
Revised: Using the above solution, we obtain the following minimization problem:min

Original: [Equation]
Revised: [Equation]

Original: 59) where (cid:98)
Revised: 59) where (cid:98)

Original: A
Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( s, a, s ′ ) :
Revised: ( s, a, s ′ ) :

Original: = r ( s, s ′ )
Revised: = r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) − λ .
Revised: − ν ( s ) − λ .

Original: Finally, similar to Proposition 3.4, we canobtain the following objective with the same minimum as (59):
Revised: Finally, similar to Proposition 3.4, we canobtain the following objective with the same minimum as (59):

Original: [Equation]
Revised: [Equation]

Original: Interestingly, the resulting objective is the same as the original objective (cid:101) L ( (cid:101) ν ) of (23).
Revised: Interestingly, the resulting objective is the same as the original objective (cid:101) L ( (cid:101) ν ) of (23).

Original: Based on thistheoretical result, we compare LobsDICE for γ = 1 , denoted by LobsDICE ( γ = 1 ), with LobsDICEfor γ = 0 .
Revised: Based on thistheoretical result, we compare LobsDICE for γ = 1 , denoted by LobsDICE ( γ = 1 ), with LobsDICEfor γ = 0 .

Original: 99 , denoted by LobsDICE ( γ = 0 . 99 ).
Revised: 99 , denoted by LobsDICE ( γ = 0 . 99 ).

Original: In Figure I.2, LobsDICE ( γ = 1 ) shows goodperformance [Figure] but LobsDICE ( γ = 0 . 99 ) is more stable and  better performance in practice[Figure]
Revised: In Figure I.2, LobsDICE ( γ = 1 ) shows goodperformance, but LobsDICE ( γ = 0 . 99 ) is more stable and achieves better performance in practice

Original: [Figure]
Revised: [Figure]

Original: K Computation Resources
Revised: K Computation Resources

Original: We used 10 servers equipped with the following specification:
Revised: We used 10 servers equipped with the following specification:

Original: • CPU: Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz.
Revised: • CPU: Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz.

Original: • Memory: 32 GB.
Revised: • Memory: 32 GB.

Original: • GPU: TITAN V.
Revised: • GPU: TITAN V.

Original: L Limitation
Revised: L Limitation

Original: Although we demonstrated that LobsDICE successfully recovered the expert’s behavior in theexperiments, it requires the assumption that support of D I covers D E .
Revised: Although we demonstrated that LobsDICE successfully recovered the expert’s behavior in theexperiments, it requires the assumption that support of D I covers D E .

Original: Relaxing this assumptionremains as a future work.
Revised: Relaxing this assumptionremains as a future work.

Original: M
Revised: M

Original: License
Revised: License

Original: D4RL [6] is licensed under the Apache 2.

Revised: D4RL [6] is licensed under the Apache 2.

Original: N Social Impact
Revised: N Social Impact

Original: This work contributes to an algorithmic foundation for imitation learning from observation in anoffline setting.
Revised: This work contributes to an algorithmic foundation for imitation learning from observation in anoffline setting.

Original: Given that state-only expert demonstrations are much easier to be collected than theaction-labeled expert demonstrations and that the imperfect demonstrations of arbitrary optimalityis also easy to be collected, our method has a potential to be widely adopted in many real-worldapplications.
Revised: Given that state-only expert demonstrations are much easier to be collected than theaction-labeled expert demonstrations and that the imperfect demonstrations of arbitrary optimalityis also easy to be collected, our method has a potential to be widely adopted in many real-worldapplications.

Original: On the other hand, this work could adversely affect employment by contributing to theautomation of tasks having done by human experts (e.g. factory automation, autonomous driving).
Revised: On the other hand, this work could adversely affect employment by contributing to theautomation of tasks having done by human experts (e.g. factory automation, autonomous driving).