Original: LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation
Revised: LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation

Original: Abstract
Revised: Abstract

Original: We consider the problem of learning from observation (LfO), in which the agent aims to mimic the expert’s behavior from the state-only demonstrations by experts.
Revised: We consider the problem of learning from observation (LfO), in which the agent aims to mimic the expert’s behavior from the state-only demonstrations by experts.

Original: We additionally assume that the agent cannot interact with the environment but has access to the action-labeled transition data collected by some agents with unknown qualities.
Revised: We additionally assume that the agent cannot interact with the environment but has access to the action-labeled transition data collected by some agents with unknown qualities.

Original: This offline setting for LfO is appealing in many real-world scenarios where the ground-truth expert actions are inaccessible and the arbitrary environment interactions are costly or risky.
Revised: This offline setting for LfO is appealing in many real-world scenarios where the ground-truth expert actions are inaccessible and the arbitrary environment interactions are costly or risky.

Original: In this paper, we present LobsDICE, an offline LfO algorithm that learns to imitate the expert policy via optimization in the space of stationary distributions.
Revised: In this paper, we present LobsDICE, an offline LfO algorithm that learns to imitate the expert policy via optimization in the space of stationary distributions.

Original: Our algorithm solves a single convex minimization problem, which minimizes the divergence between the two statetransition distributions induced by the expert and the agent policy.
Revised: Our algorithm solves a single convex minimization problem, which minimizes the divergence between the two statetransition distributions induced by the expert and the agent policy.

Original: Through an extensive set of offline LfO tasks, we show that LobsDICE outperforms strong baseline methods.
Revised: Through an extensive set of offline LfO tasks, we show that LobsDICE outperforms strong baseline methods.

Original:  Introduction
Revised:  Introduction

Original: The ability to learn from experience is one of the core aspects of an intelligent agent.
Revised: The ability to learn from experience is one of the core aspects of an intelligent agent.

Original: Reinforcement learning (RL) [36] provides a framework to acquire such an intelligent behavior autonomously through interactions with the environment while receiving reward feedback.
Revised: Reinforcement learning (RL) [36] provides a framework to acquire such an intelligent behavior autonomously through interactions with the environment while receiving reward feedback.

Original: However, the practical applicability of RL to real-world domains has been limited for two reasons.
Revised: However, the practical applicability of RL to real-world domains has been limited for two reasons.

Original: First, designing a suitable reward function for complex tasks can be extremely difficult.
Revised: First, designing a suitable reward function for complex tasks can be extremely difficult.

Original: The RL agent learns behaviors incentivized by the reward function rather than the ones intended, which can be nontrivial to specify in terms of reward.
Revised: The RL agent learns behaviors incentivized by the reward function rather than the ones intended, which can be nontrivial to specify in terms of reward.

Original: Second, the need for online interaction with the environment during the RL training loop has hindered its adoption in many real-world domains, where environment interactions are costly or risky.
Revised: Second, the need for online interaction with the environment during the RL training loop has hindered its adoption in many real-world domains, where environment interactions are costly or risky.

Original: Imitation learning (IL) [1, 33, 34] circumvents the difficulty of reward design in RL by leveraging demonstrations given by experts, where the goal is to mimic the expert’s behavior.
Revised: Imitation learning (IL) [1, 33, 34] circumvents the difficulty of reward design in RL by leveraging demonstrations given by experts, where the goal is to mimic the expert’s behavior.

Original: However, the standard IL requires the expert demonstrations to contain not only the state information (e.g. robot joint angles) but also the precise action (e.g. robot joint torques) executed by the expert at each time step.
Revised: However, the standard IL requires the expert demonstrations to contain not only the state information (e.g. robot joint angles) but also the precise action (e.g. robot joint torques) executed by the expert at each time step.

Original: This demand for explicit action labels is in contrast to the way human imitates (e.g. learning by watching videos) and precludes leveraging a massive amount of data in which the action label is missing.
Revised: This demand for explicit action labels is in contrast to the way human imitates (e.g. learning by watching videos) and precludes leveraging a massive amount of data in which the action label is missing.

Original: Therefore, developing an imitation learning algorithm that can learn from observing the
Revised: Therefore, developing an imitation learning algorithm that can learn from observing the

Original: [Table]
Revised: [Table]

Original: experts’ state-only trajectories is a promising direction for creating a practical, autonomous intelligent agent.
Revised: experts’ state-only trajectories is a promising direction for creating a practical, autonomous intelligent agent.

Original: Learning from observation (LfO) [2, 14, 26, 39, 45] concerns this particular learning scenario and has been gaining interest in recent years.
Revised: Learning from observation (LfO) [2, 14, 26, 39, 45] concerns this particular learning scenario and has been gaining interest in recent years.

Original: In this paper, we are particularly interested in solving the LfO problem in an offline setting:
Revised: In this paper, we are particularly interested in solving the LfO problem in an offline setting:

Original: Given the state-only demonstrations by experts and abundant state-action demonstrations by imperfect agents of arbitrary levels of optimality, we aim to find a policy that follows the sequence of expert’s states without further interaction with the environment.
Revised: Given the state-only demonstrations by experts and abundant state-action demonstrations by imperfect agents of arbitrary levels of optimality, we aim to find a policy that follows the sequence of expert’s states without further interaction with the environment.

Original: This problem setting is appealing in many practical situations where environment interactions are costly or risky.
Revised: This problem setting is appealing in many practical situations where environment interactions are costly or risky.

Original: Yet, it cannot be straightforwardly tackled by existing approaches.
Revised: Yet, it cannot be straightforwardly tackled by existing approaches.

Original: Most of the existing LfO methods are on-policy algorithms [25, 35, 39, 40], which is not directly applicable to the offline setting.
Revised: Most of the existing LfO methods are on-policy algorithms [25, 35, 39, 40], which is not directly applicable to the offline setting.

Original: One of the few exceptions is BCO [38], which performs behavior cloning on the inferred action via an inverse dynamics model.
Revised: One of the few exceptions is BCO [38], which performs behavior cloning on the inferred action via an inverse dynamics model.

Original: This method could be suboptimal unless the inverse dynamics disagreement is always zero, i.e. the underlying environment dynamics is deterministic and injective.
Revised: This method could be suboptimal unless the inverse dynamics disagreement is always zero, i.e. the underlying environment dynamics is deterministic and injective.

Original: Another one is OPOLO [45], an off-policy LfO algorithm, but it relies on nested min-max optimization as well as out-of-distribution (OOD) action values, which can be unstable especially in the offline setting.
Revised: Another one is OPOLO [45], an off-policy LfO algorithm, but it relies on nested min-max optimization as well as out-of-distribution (OOD) action values, which can be unstable especially in the offline setting.

Original: IQ-Learn [8] solves an Inverse RL and can be applied to the offline LfO by learning a state-only reward function.
Revised: IQ-Learn [8] solves an Inverse RL and can be applied to the offline LfO by learning a state-only reward function.

Original: However, it suffers from numerical instability due to using OOD action values in the offline setting.
Revised: However, it suffers from numerical instability due to using OOD action values in the offline setting.

Original: Lastly, RCE [5], an example-based control algorithm, can in principle be applied to the offline LfO setting by providing the expert trajectories as successful example states; however, its empirical performance is known to be limited without online data collection.
Revised: Lastly, RCE [5], an example-based control algorithm, can in principle be applied to the offline LfO setting by providing the expert trajectories as successful example states; however, its empirical performance is known to be limited without online data collection.

Original: We present an offline LfO algorithm that minimizes the divergence between state-transition distributions induced by the expert and the learned policy, without requiring an inverse dynamics model.
Revised: We present an offline LfO algorithm that minimizes the divergence between state-transition distributions induced by the expert and the learned policy, without requiring an inverse dynamics model.

Original: Our algorithm, offline Learning from OBServation via stationary DIstribution Correction
Revised: Our algorithm, offline Learning from OBServation via stationary DIstribution Correction

Original: Estimation (LobsDICE), essentially optimizes in the space of state-action stationary distributions and state-transition stationary distributions, rather than in the space of policies.
Revised: Estimation (LobsDICE), essentially optimizes in the space of state-action stationary distributions and state-transition stationary distributions, rather than in the space of policies.

Original: We show that our formulation can be reduced to a single convex minimization problem that can be solved efficiently in practice, unlike existing imitation learning (from observation) algorithms that rely on nested min-max optimization.
Revised: We show that our formulation can be reduced to a single convex minimization problem that can be solved efficiently in practice, unlike existing imitation learning (from observation) algorithms that rely on nested min-max optimization.

Original: In the experiments, we demonstrate that LobsDICE can successfully recover the state visitations by the expert, outperforming strong baseline methods.
Revised: In the experiments, we demonstrate that LobsDICE can successfully recover the state visitations by the expert, outperforming strong baseline methods.

Original:  Preliminaries 2.1 Markov decision process
Revised:  Preliminaries 2.1 Markov decision process

Original: We consider an environment modeled as a Markov Decision Process (MDP), defined by M = ⟨ S, A, T, R, p 0 , γ ⟩ [36], where S is the set of states, A is the set of actions, T : S
Revised: We consider an environment modeled as a Markov Decision Process (MDP), defined by M = ⟨ S, A, T, R, p 0 , γ ⟩ [36], where S is the set of states, A is the set of actions, T : S

Original: ×
Revised: ×

Original: A → ∆( S ) is the transition probability, R : S ×
Revised: A → ∆( S ) is the transition probability, R : S ×

Original: A → R is the reward function, p 0 ∈ ∆( S ) is the distribution of the initial state, and γ ∈ (0 , 1) is the discount factor.
Revised: A → R is the reward function, p 0 ∈ ∆( S ) is the distribution of the initial state, and γ ∈ (0 , 1) is the discount factor.

Original: The policy π : S → ∆( A ) is a mapping from states to distribution over actions.
Revised: The policy π : S → ∆( A ) is a mapping from states to distribution over actions.

Original: For the given policy π , its state-action stationary distribution d π ( s, a ) and state-transition stationary ¯ d π ( s, s ′ ) are defined as:
Revised: For the given policy π , its state-action stationary distribution d π ( s, a ) and state-transition stationary ¯ d π ( s, s ′ ) are defined as:

Original: [Equation]
Revised: [Equation]

Original: where s 0 ∼ p 0 , a t ∼ π ( ·| s t ) , and s t
Revised: where s 0 ∼ p 0 , a t ∼ π ( ·| s t ) , and s t

Original: +1 ∼ T ( ·| s t , a t ) for all timesteps t ≥ 0 .
Revised: +1 ∼ T ( ·| s t , a t ) for all timesteps t ≥ 0 .

Original: For brevity, the bar notation ¯( · ) will be used to denote the distributions for ( s, s ′ ) , e.g. ¯ d π ( s, s ′ ) .
Revised: For brevity, the bar notation ¯( · ) will be used to denote the distributions for ( s, s ′ ) , e.g. ¯ d π ( s, s ′ ) .

Original: We assume offline LfO setting, where direct, online interactions with the environment are not allowed, and the policy should be optimized solely from a pre-collected dataset.
Revised: We assume offline LfO setting, where direct, online interactions with the environment are not allowed, and the policy should be optimized solely from a pre-collected dataset.

Original: We denote the dataset of state-only demonstrations collected by experts as D E
Revised: We denote the dataset of state-only demonstrations collected by experts as D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: i } N E i =1 and the dataset of stateaction demonstrations by some imperfect agents as D I = { ( s, a, s ′ )
Revised: i } N E i =1 and the dataset of stateaction demonstrations by some imperfect agents as D I = { ( s, a, s ′ )

Original: i } N I
Revised: i } N I

Original: i =1 .
Revised: i =1 .

Original: That is, we do not have information about the actions taken by the expert, but instead, we have additional action-labeled transition data collected by some other agents with unknown levels of optimality.
Revised: That is, we do not have information about the actions taken by the expert, but instead, we have additional action-labeled transition data collected by some other agents with unknown levels of optimality.

Original: We denote the corresponding distributions of the datasets D E and D I by ¯ d E and d I , respectively.
Revised: We denote the corresponding distributions of the datasets D E and D I by ¯ d E and d I , respectively.

Original: For brevity, we will abuse notation d I to represent ( s, a ) ∼ d
Revised: For brevity, we will abuse notation d I to represent ( s, a ) ∼ d

Original: I , ( s, a, s ′ ) ∼ d I , and ( s, s ′ )
Revised: I , ( s, a, s ′ ) ∼ d I , and ( s, s ′ )

Original: ∼ ¯ d I unless ambiguous.
Revised: ∼ ¯ d I unless ambiguous.

Original: 2.2 Imitation learning and learning from observation
Revised: 2.2 Imitation learning and learning from observation

Original: Imitation learning (IL) aims to mimic the expert policy from its state-action demonstrations.
Revised: Imitation learning (IL) aims to mimic the expert policy from its state-action demonstrations.

Original: IL can be naturally formulated as a distribution matching problem that minimizes the divergence between state-action stationary distributions induced by the expert and the target policy [11, 13].
Revised: IL can be naturally formulated as a distribution matching problem that minimizes the divergence between state-action stationary distributions induced by the expert and the target policy [11, 13].

Original: For example, one can consider minimizing KL-divergence [17]:
Revised: For example, one can consider minimizing KL-divergence [17]:

Original: [Equation]
Revised: [Equation]

Original: However, the standard IL requires action labels in the expert demonstrations, which may be a too strong requirement for various practical situations.
Revised: However, the standard IL requires action labels in the expert demonstrations, which may be a too strong requirement for various practical situations.

Original: Learning from observation (LfO) relaxes the requirement on action labels, and aims to imitate the expert’s behavior only from the state observations.
Revised: Learning from observation (LfO) relaxes the requirement on action labels, and aims to imitate the expert’s behavior only from the state observations.

Original: Since the expert’s action information is missing in the demonstrations, the distribution matching for state-action stationary distribution is no longer readily applicable.
Revised: Since the expert’s action information is missing in the demonstrations, the distribution matching for state-action stationary distribution is no longer readily applicable.

Original: Therefore, LfO is reformulated as another distribution matching problem that minimizes the divergence between state-transition stationary distributions induced by the expert and the target policy [39, 40, 45] † :
Revised: Therefore, LfO is reformulated as another distribution matching problem that minimizes the divergence between state-transition stationary distributions induced by the expert and the target policy [39, 40, 45] 1 :

Original: [Equation]
Revised: [Equation]

Original: Still, optimizing Eq.
Revised: Still, optimizing Eq.

Original: (2) in a purely offline manner is challenging since naively estimating the expectation would require the knowledge of T ( ·| s, a ) for the OOD action a ∼ π ( s ) by the target policy, which is inaccessible in the offline LfO setting; see Section 9.8 in [45] for more discussions.
Revised: (2) in a purely offline manner is challenging since naively estimating the expectation would require the knowledge of T ( ·| s, a ) for the OOD action a ∼ π ( s ) by the target policy, which is inaccessible in the offline LfO setting; see Section 9.8 in [45] for more discussions.

Original: OPOLO [45], an off-policy LfO algorithm, mitigates this challenge by minimizing the following upper bound of the divergence D KL ( ¯ d π ( s, s ′ )
Revised: OPOLO [45], an off-policy LfO algorithm, mitigates this challenge by minimizing the following upper bound of the divergence D KL ( ¯ d π ( s, s ′ )

Original: ∥ ¯ d E ( s, s ′ )) :
Revised: ∥ ¯ d E ( s, s ′ )) :

Original: [Equation]
Revised: [Equation]

Original: and applying DualDICE trick [29] to the RHS of (3).
Revised: and applying DualDICE trick [29] to the RHS of (3).

Original: The upper bound gap is given by the inverse dynamics disagreement between the target policy and the imperfect demonstrator:
Revised: The upper bound gap is given by the inverse dynamics disagreement between the target policy and the imperfect demonstrator:

Original: [Equation]
Revised: [Equation]

Original: which usually gets larger as the stochasticity of the environment transition increases.
Revised: which usually gets larger as the stochasticity of the environment transition increases.

Original:  LobsDICE
Revised:  LobsDICE

Original: We present offline Learning from OBServation via stationary DIstribution Correction Estimation (LobsDICE), a principled offline LfO algorithm that further extends the recent progress made by the
Revised: We present offline Learning from OBServation via stationary DIstribution Correction Estimation (LobsDICE), a principled offline LfO algorithm that further extends the recent progress made by the

Original: DIstribution Correction Estimation (DICE) methods for offline RL.
Revised: DIstribution Correction Estimation (DICE) methods for offline RL.

Original: LobsDICE essentially optimizes the stationary distributions of the target policy to match the expert’s state visitations.
Revised: LobsDICE essentially optimizes the stationary distributions of the target policy to match the expert’s state visitations.

Original: In the context of offline RL, the policy constraint principle (i.e. prevent deviating too much from the data support) is one of the common approaches to avoid severe performance degradation [7, 12, 18, 22, 30].
Revised: In the context of offline RL, the policy constraint principle (i.e. prevent deviating too much from the data support) is one of the common approaches to avoid severe performance degradation [7, 12, 18, 22, 30].

Original: In the same manner, we use KL divergence minimization between ¯ d π ( s, s ′ ) and ¯ d E ( s, s ′ ) with additional KL regularization on the deviation from d I :
Revised: In the same manner, we use KL divergence minimization between ¯ d π ( s, s ′ ) and ¯ d E ( s, s ′ ) with additional KL regularization on the deviation from d I :

Original: [Equation]
Revised: [Equation]

Original: Here the hyperparameter α > 0 balances between encouraging state-transition matching and preventing distribution shift from the distribution of imperfect demonstrations.
Revised: Here the hyperparameter α > 0 balances between encouraging state-transition matching and preventing distribution shift from the distribution of imperfect demonstrations.

Original: All the proofs can be found in Appendix F.
Revised: All the proofs can be found in Appendix F.

Original: 3.1 Lagrange dual formulation
Revised: 3.1 Lagrange dual formulation

Original: The derivation of our algorithm starts by rewriting the (regularized) distribution matching problem (5) in terms of directly optimizing stationary distribution, rather than policy:
Revised: The derivation of our algorithm starts by rewriting the (regularized) distribution matching problem (5) in terms of directly optimizing stationary distribution, rather than policy:

Original: [Equation]
Revised: [Equation]

Original: The Bellman flow constraint (7) ensures d ( s, a ) to be a valid state-action stationary distribution of some policy, where d ( s, a ) can be interpreted as a normalized occupancy measure of ( s, a ) .
Revised: The Bellman flow constraint (7) ensures d ( s, a ) to be a valid state-action stationary distribution of some policy, where d ( s, a ) can be interpreted as a normalized occupancy measure of ( s, a ) .

Original: The marginalization constraint (8) enforces ¯ d ( s, s ′ ) to be the state-transition stationary distribution that is directly induced by d ( s, a ) .
Revised: The marginalization constraint (8) enforces ¯ d ( s, s ′ ) to be the state-transition stationary distribution that is directly induced by d ( s, a ) .

Original: In essence, the constrained optimization problem (6-8) seeks the stationary distributions of an optimal policy, which best matches the state-transition trajectories of the expert.
Revised: In essence, the constrained optimization problem (6-8) seeks the stationary distributions of an optimal policy, which best matches the state-transition trajectories of the expert.

Original: Once we have computed the optimal solution ( d ∗ , ¯ d ∗ ) , its corresponding optimal policy can also be obtained by normalizing d ∗ for each state [32]: π ∗ ( a | s ) =
Revised: Once we have computed the optimal solution ( d ∗ , ¯ d ∗ ) , its corresponding optimal policy can also be obtained by normalizing d ∗ for each state [32]: π ∗ ( a | s ) =

Original: [Equation]
Revised: [Equation]

Original: Note that DemoDICE [15] considers a similar optimization problem to ours, but it deals with the offline IL (i.e. state-action stationary distribution matching), whereas we consider the offline LfO (e.g state-transition stationary distribution matching).
Revised: Note that DemoDICE [15] considers a similar optimization problem to ours, but it deals with the offline IL (i.e. state-action stationary distribution matching), whereas we consider the offline LfO (e.g state-transition stationary distribution matching).

Original: Accordingly, the optimization variable ¯ d ( s, s ′ ) and the marginalization constraint (8) are newly added in our formulation.
Revised: Accordingly, the optimization variable ¯ d ( s, s ′ ) and the marginalization constraint (8) are newly added in our formulation.

Original: We then consider the Lagrangian for the constrained optimization (6-8):
Revised: We then consider the Lagrangian for the constrained optimization (6-8):

Original: [Equation]
Revised: [Equation]

Original: where ν ( s ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (7)
Revised: where ν ( s ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (7)

Original: [Equation]
Revised: [Equation]

Original: and µ ( s, s ′ ) ∈ R are the the Lagrange multipliers for the marginalization constraint (8).
Revised: and µ ( s, s ′ ) ∈ R are the the Lagrange multipliers for the marginalization constraint (8).

Original: Note that the Lagrangian (9) cannot be naively optimized in an offline manner since it requires evaluation of T ( s ′ | s, a ) for ( s, a ) ∼ d , which is not accessible in the offline LfO setting.
Revised: Note that the Lagrangian (9) cannot be naively optimized in an offline manner since it requires evaluation of T ( s ′ | s, a ) for ( s, a ) ∼ d , which is not accessible in the offline LfO setting.

Original: Therefore, we rearrange the terms in (9) to eliminate the direct dependence on d and ¯ d , introducing new optimization variables w and ¯ w that denote stationary distribution correction ratios for ( s, a ) and ( s, s ′ ) , respectively:
Revised: Therefore, we rearrange the terms in (9) to eliminate the direct dependence on d and ¯ d , introducing new optimization variables w and ¯ w that denote stationary distribution correction ratios for ( s, a ) and ( s, s ′ ) , respectively:

Original: [Equation]
Revised: [Equation]

Original: Similar to the assumption of full coverage which is fairly standard across a broad set of recent offline
Revised: Similar to the assumption of full coverage which is fairly standard across a broad set of recent offline

Original: RL approaches [19, 22, 27], we make a milder assumption that ¯ d I ( s, s ′ ) > 0 whenever ¯ d E ( s, s ′ ) > 0 .
Revised: RL approaches [19, 22, 27], we make a milder assumption that ¯ d I ( s, s ′ ) > 0 whenever ¯ d E ( s, s ′ ) > 0 .

Original: This assumption is necessary to recover the expert’s behavior successfully.
Revised: This assumption is necessary to recover the expert’s behavior successfully.

Original: We introduce the log ratio r ( s, s ′ )
Revised: We introduce the log ratio r ( s, s ′ )

Original: =
Revised: =

Original: log ¯ d E ( s,s ′ )
Revised: log ¯ d E ( s,s ′ )

Original: ¯ d I ( s,s ′ ) in (10) to take an expectation under ¯ d I (instead of ¯ d E ), which is assumed to have a broader support than ¯ d E .
Revised: ¯ d I ( s,s ′ ) in (10) to take an expectation under ¯ d I (instead of ¯ d E ), which is assumed to have a broader support than ¯ d E .

Original: This log ratio r ( s, s ′ ) can be easily estimated using a pretrained discriminator for two datasets D E and D I , which will be explained in detail in the following section.
Revised: This log ratio r ( s, s ′ ) can be easily estimated using a pretrained discriminator for two datasets D E and D I , which will be explained in detail in the following section.

Original: In summary, LobsDICE aims to solve the minimax optimization,
Revised: In summary, LobsDICE aims to solve the minimax optimization,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The optimal solution ( w ∗ , ¯ w ∗ ) of (12) represents stationary distribution corrections of an optimal policy π ∗ : w ∗ ( s, a )
Revised: The optimal solution ( w ∗ , ¯ w ∗ ) of (12) represents stationary distribution corrections of an optimal policy π ∗ : w ∗ ( s, a )

Original: =
Revised: =

Original: d π ∗ ( s,a ) d
Revised: d π ∗ ( s,a ) d

Original: I ( s,a ) and ¯ w ∗ ( s, s ′ ) =
Revised: I ( s,a ) and ¯ w ∗ ( s, s ′ ) =

Original: ¯ d π ∗ ( s,s ′ )
Revised: ¯ d π ∗ ( s,s ′ )

Original: ¯ d I ( s,s ′ ) .
Revised: ¯ d I ( s,s ′ ) .

Original: 3.2 Log ratio estimation via a pretrained discriminator
Revised: 3.2 Log ratio estimation via a pretrained discriminator

Original: To optimize (11), an estimate of the log ratio r ( s, s ′ )
Revised: To optimize (11), an estimate of the log ratio r ( s, s ′ )

Original: =
Revised: =

Original: log ¯ d E ( s,s ′ )
Revised: log ¯ d E ( s,s ′ )

Original: ¯ d I ( s,s ′ ) is required.
Revised: ¯ d I ( s,s ′ ) is required.

Original: The log ratio estimation is straightforward for tabular MDPs since we can use empirical distributions from the datasets to estimate ¯ d E ( s, s ′ ) and ¯ d I ( s, s ′ ) .
Revised: The log ratio estimation is straightforward for tabular MDPs since we can use empirical distributions from the datasets to estimate ¯ d E ( s, s ′ ) and ¯ d I ( s, s ′ ) .

Original: For continuous MDPs, we train a discriminator c :
Revised: For continuous MDPs, we train a discriminator c :

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: ×

Original: [Equation]
Revised: [Equation]

Original: → [0 , 1] by solving the following maximization problem [9, 45]:
Revised: → [0 , 1] by solving the following maximization problem [9, 45]:

Original: [Equation]
Revised: [Equation]

Original: It is easy to show that the optimal discriminator satisfies c ∗ ( s, s ′ ) =
Revised: It is easy to show that the optimal discriminator satisfies c ∗ ( s, s ′ ) =

Original: [Equation]
Revised: [Equation]

Original: Thus, r ( s, s
Revised: Thus, r ( s, s

Original: [Equation]
Revised: [Equation]

Original: ) can be derived from the optimal discriminator c ∗ as,
Revised: ) can be derived from the optimal discriminator c ∗ as,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3.3 Minimax to min: a closed-form solution
Revised: 3.3 Minimax to min: a closed-form solution

Original: Exploiting the strict convexity of x log x , we can derive a closed-form solution to the inner maximization for ( w, ¯ w ) in (11).
Revised: Exploiting the strict convexity of x log x , we can derive a closed-form solution to the inner maximization for ( w, ¯ w ) in (11).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.
Revised: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: L ( w,
Revised: L ( w,

Original: [Equation]
Revised: [Equation]

Original: w, µ, ν ) , is given by:
Revised: w, µ, ν ) , is given by:

Original: [Equation]
Revised: [Equation]

Original: ¯
Revised: ¯

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Based on the above result, we reduce the nested min-max optimization of (11) to a single minimization by plugging the closed-form solution ( w µ,ν , ¯ w µ ) into L ( w, ¯ w, µ, ν ) : min
Revised: Based on the above result, we reduce the nested min-max optimization of (11) to a single minimization by plugging the closed-form solution ( w µ,ν , ¯ w µ ) into L ( w, ¯ w, µ, ν ) : min

Original: [Equation]
Revised: [Equation]

Original: We can even show that L ( w µ,ν , ¯ w µ , µ, ν ) is a convex function of µ and ν .
Revised: We can even show that L ( w µ,ν , ¯ w µ , µ, ν ) is a convex function of µ and ν .

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .
Revised: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .

Original: In short, by operating in the space of stationary distributions, offline LfO can, in principle, be resolved by solving a convex minimization problem.
Revised: In short, by operating in the space of stationary distributions, offline LfO can, in principle, be resolved by solving a convex minimization problem.

Original: This is in contrast to the existing LfO algorithms, which typically involve either an adversarial training that optimizes the policy and the discriminator [14, 39, 40] or a nested min-max optimization for the policy and the critic [45].
Revised: This is in contrast to the existing LfO algorithms, which typically involve either an adversarial training that optimizes the policy and the discriminator [14, 39, 40] or a nested min-max optimization for the policy and the critic [45].

Original: 3.4 Policy extraction
Revised: 3.4 Policy extraction

Original: So far, we have derived an algorithm that essentially solves the state-transition distribution matching problem via convex minimization.
Revised: So far, we have derived an algorithm that essentially solves the state-transition distribution matching problem via convex minimization.

Original: However, we obtain ( µ ∗ , ν ∗ ) as the solution of (16), instead of the optimal policy π ∗ .
Revised: However, we obtain ( µ ∗ , ν ∗ ) as the solution of (16), instead of the optimal policy π ∗ .

Original: The remaining problem is to extract the optimal policy from ( µ ∗ , ν ∗ ) .
Revised: The remaining problem is to extract the optimal policy from ( µ ∗ , ν ∗ ) .

Original: The first step is to see that we can obtain the state-action stationary distribution correction w µ ∗ ,ν ∗ ( s, a ) =
Revised: The first step is to see that we can obtain the state-action stationary distribution correction w µ ∗ ,ν ∗ ( s, a ) =

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ) of the optimal policy from the closed-form solution (15).
Revised: ) of the optimal policy from the closed-form solution (15).

Original: Among many possibilities to extract the policy from the state-action distribution correction, we adopt weighted behavior cloning (WBC):
Revised: Among many possibilities to extract the policy from the state-action distribution correction, we adopt weighted behavior cloning (WBC):

Original: [Equation]
Revised: [Equation]

Original: (17) which aims to maximize the predicted probability of actions chosen by optimal policy π ∗ .
Revised: (17) which aims to maximize the predicted probability of actions chosen by optimal policy π ∗ .

Original: This is done by BC on the offline dataset D I where each sample ( s, a ) is weighted by w µ ∗ ,ν ∗ ( s, a ) .
Revised: This is done by BC on the offline dataset D I where each sample ( s, a ) is weighted by w µ ∗ ,ν ∗ ( s, a ) .

Original: For tabular MDPs, we can formally show that WBC extracts an optimal policy π ∗ (Appendix C).
Revised: For tabular MDPs, we can formally show that WBC extracts an optimal policy π ∗ (Appendix C).

Original: 3.5 Practical algorithm with sample-based approximation
Revised: 3.5 Practical algorithm with sample-based approximation

Original: In practice, we estimate L ( w, ¯ w, µ, ν ) in (11) using samples from distribution d I .
Revised: In practice, we estimate L ( w, ¯ w, µ, ν ) in (11) using samples from distribution d I .

Original: Let ˆE x ∈ D
Revised: Let ˆE x ∈ D

Original: [ f ( x )] :=
Revised: [ f ( x )] :=

Original: | (cid:80) x ∈ D f ( x ) be a Monte-Carlo estimate of E x ∼ p
Revised: | (cid:80) x ∈ D f ( x ) be a Monte-Carlo estimate of E x ∼ p

Original: [ f ( x )] where D = { x i } | D | i =1 ∼ p .
Revised: [ f ( x )] where D = { x i } | D | i =1 ∼ p .

Original: We denote each sample ( s, a, s ′ ) in D I as x for brevity.
Revised: We denote each sample ( s, a, s ′ ) in D I as x for brevity.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˆ e µ,ν ( s, a, s ′ ) :
Revised: where ˆ e µ,ν ( s, a, s ′ ) :

Original: = − µ ( s, s ′ )
Revised: = − µ ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) is a single-sample estimate of e µ,ν ( s, a ) .
Revised: − ν ( s ) is a single-sample estimate of e µ,ν ( s, a ) .

Original: Note that this sample-based objective function (cid:98) L ( w, ¯ w, µ, ν ) can be estimated only from samples in the offline dataset D I and is an unbiased estimator of L ( w, ¯ w, µ, ν ) as long as every sample in D I was collected by interacting with the underlying MDP.
Revised: Note that this sample-based objective function (cid:98) L ( w, ¯ w, µ, ν ) can be estimated only from samples in the offline dataset D I and is an unbiased estimator of L ( w, ¯ w, µ, ν ) as long as every sample in D I was collected by interacting with the underlying MDP.

Original: To the best of our knowledge, this is the first result to directly solve the state-transition distribution matching problem in a fully offline manner.
Revised: To the best of our knowledge, this is the first result to directly solve the state-transition distribution matching problem in a fully offline manner.

Original: In contrast, OPOLO [45] relies on the (potentially loose) upper bound of the objective in (3).
Revised: In contrast, OPOLO [45] relies on the (potentially loose) upper bound of the objective in (3).

Original: To reduce the minimax optimization to a single minimization, we apply the non-parametric closedform solution for each sample x
Revised: To reduce the minimax optimization to a single minimization, we apply the non-parametric closedform solution for each sample x

Original: = ( s, a, s ′ ) in D
Revised: = ( s, a, s ′ ) in D

Original: I :
Revised: I :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which is analogous to (15).
Revised: which is analogous to (15).

Original: Plugging this result into (18) yields a sample-based objective function for minimization † :
Revised: Plugging this result into (18) yields a sample-based objective function for minimization 2 :

Original: [Equation]
Revised: [Equation]

Original: Still, the variable µ ∈ R S × S is much higher dimensional than ν ∈ R S .
Revised: Still, the variable µ ∈ R S × S is much higher dimensional than ν ∈ R S .

Original: So, it works as the main bottleneck for the overall optimization.
Revised: So, it works as the main bottleneck for the overall optimization.

Original: Fortunately, we can further simplify (20) by eliminating its dependence on µ via exploiting an additional closed-form solution.
Revised: Fortunately, we can further simplify (20) by eliminating its dependence on µ via exploiting an additional closed-form solution.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e. µ ν = arg min µ (cid:98) L ( µ, ν ) , is
Revised: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e. µ ν = arg min µ (cid:98) L ( µ, ν ) , is

Original: [Equation]
Revised: [Equation]

Original: Using the above result in (20), we obtain the following minimization problem:
Revised: Using the above result in (20), we obtain the following minimization problem:

Original: [Equation]
Revised: [Equation]

Original: 22) where (cid:98) A ν ( s, a, s ′ ) :
Revised: 22) where (cid:98) A ν ( s, a, s ′ ) :

Original: = r ( s, s ′ )
Revised: = r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) .
Revised: − ν ( s ) .

Original: The remaining issue is that optimizing (22) is not practical because exp( · ) often causes numerical instability and gradient explosion.
Revised: The remaining issue is that optimizing (22) is not practical because exp( · ) often causes numerical instability and gradient explosion.

Original: To address this, we use a numerically-stable alternative of (22):
Revised: To address this, we use a numerically-stable alternative of (22):

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: Let (cid:101) L ( (cid:101) ν ) be the function:
Revised: Let (cid:101) L ( (cid:101) ν ) be the function:

Original: [Equation]
Revised: [Equation]

Original: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )
Revised: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )

Original: = min (cid:101) ν (cid:101)
Revised: = min (cid:101) ν (cid:101)

Original: L ( (cid:101) ν ) holds.
Revised: L ( (cid:101) ν ) holds.

Original: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .
Revised: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .

Original: In order see why minimizing (cid:101) L ( (cid:101) ν ) no longer suffers from numerical instability, note that the gradient ∇ x log E x ∼ p [exp( h ( x ))]
Revised: In order see why minimizing (cid:101) L ( (cid:101) ν ) no longer suffers from numerical instability, note that the gradient ∇ x log E x ∼ p [exp( h ( x ))]

Original: =
Revised: =

Original: E x ∼ p [ exp( h ( x ))
Revised: E x ∼ p [ exp( h ( x ))

Original: E ¯ x ∼ p [exp( h (¯ x ))] ∇ x h ( x )] normalizes exp( · ) by softmax and thus tames large numerical values.
Revised: E ¯ x ∼ p [exp( h (¯ x ))] ∇ x h ( x )] normalizes exp( · ) by softmax and thus tames large numerical values.

Original: Finally, we can show the following connection between (cid:98)
Revised: Finally, we can show the following connection between (cid:98)

Original: ν ∗ and (cid:101)
Revised: ν ∗ and (cid:101)

Original: ν ∗ :
Revised: ν ∗ :

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: Let (cid:98) V and (cid:101)
Revised: Let (cid:98) V and (cid:101)

Original: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.
Revised: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.

Original: Then, (cid:101) V = { (cid:98) ν
Revised: Then, (cid:101) V = { (cid:98) ν

Original: ∗ + C | (cid:98) ν ∗ ∈ (cid:98) V , C ∈ R } holds.
Revised: ∗ + C | (cid:98) ν ∗ ∈ (cid:98) V , C ∈ R } holds.

Original: Proposition 3.5 implies that an unnormalized stationary distribution corrections of an optimal policy would be obtained from (cid:101)
Revised: Proposition 3.5 implies that an unnormalized stationary distribution corrections of an optimal policy would be obtained from (cid:101)

Original: ν ∗ ∈ (cid:101) V .
Revised: ν ∗ ∈ (cid:101) V .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (by Proposition 3.5)
Revised: (by Proposition 3.5)

Original: Policy extraction We must take caution when using (cid:101) w (cid:101) ν ∗ ( x ) since it is an unnormalized density ratio, i.e. E x ∼ d I [ (cid:101) w (cid:101) ν ∗ ( x )]
Revised: Policy extraction We must take caution when using (cid:101) w (cid:101) ν ∗ ( x ) since it is an unnormalized density ratio, i.e. E x ∼ d I [ (cid:101) w (cid:101) ν ∗ ( x )]

Original: ̸ = 1 .
Revised: ̸ = 1 .

Original: Therefore, to extract a policy, we perform weighted BC using self-normalized importance sampling [31]:
Revised: Therefore, to extract a policy, we perform weighted BC using self-normalized importance sampling [31]:

Original: [Equation]
Revised: [Equation]

Original: (24) which completes the derivation of the practical version of LobsDICE.
Revised: (24) which completes the derivation of the practical version of LobsDICE.

Original: To sum up, LobsDICE solves (cid:101)
Revised: To sum up, LobsDICE solves (cid:101)

Original: ν ∗ = arg min (cid:101) ν (cid:101) L ( (cid:101)
Revised: ν ∗ = arg min (cid:101) ν (cid:101) L ( (cid:101)

Original: ν ) of (23) via gradient descent, and extracts a policy via self-normalized weighted
Revised: ν ) of (23) via gradient descent, and extracts a policy via self-normalized weighted

Original: BC of (24).
Revised: BC of (24).

Original: Pseudocode for LobsDICE can be found in Appendix G.2.
Revised: Pseudocode for LobsDICE can be found in Appendix G.2.

Original:  Related Work
Revised:  Related Work

Original: Learning from observation (LfO)
Revised: Learning from observation (LfO)

Original: Recent approaches for LfO are mostly on-policy [25, 26, 35, 39, 40] algorithms and are not directly applicable to the offline LfO setting considered in this work.
Revised: Recent approaches for LfO are mostly on-policy [25, 26, 35, 39, 40] algorithms and are not directly applicable to the offline LfO setting considered in this work.

Original: MobILE [14] is a model-based LfO algorithm, but it encourages uncertainty for online exploration, which is not suitable for the offline setting.
Revised: MobILE [14] is a model-based LfO algorithm, but it encourages uncertainty for online exploration, which is not suitable for the offline setting.

Original: BCO [38] uses an inverse dynamics model (IDM) to infer the missing expert actions and performs BC on the generated expert’s state-action dataset.
Revised: BCO [38] uses an inverse dynamics model (IDM) to infer the missing expert actions and performs BC on the generated expert’s state-action dataset.

Original: In addition to common issues by vanilla BC, BCO is not guaranteed to recover the expert’s behavior in general.
Revised: In addition to common issues by vanilla BC, BCO is not guaranteed to recover the expert’s behavior in general.

Original: OPOLO [45] is a principled off-policy LfO algorithm, but it solves a nested optimization and requires evaluation on OOD action values during training, which suffers from numerical instability in the offline setting.
Revised: OPOLO [45] is a principled off-policy LfO algorithm, but it solves a nested optimization and requires evaluation on OOD action values during training, which suffers from numerical instability in the offline setting.

Original: IQ-Learn [8] solves online and offline IL problems while avoiding adversarial training by learning a single Q-function.
Revised: IQ-Learn [8] solves online and offline IL problems while avoiding adversarial training by learning a single Q-function.

Original: However, it also suffers from numerical instability in the offline setting by overestimating Q due to using OOD action values.
Revised: However, it also suffers from numerical instability in the offline setting by overestimating Q due to using OOD action values.

Original: RCE [5] aims to solve example-based control tasks in which a collection of example success states is assumed to be provided instead of entire expert trajectories.
Revised: RCE [5] aims to solve example-based control tasks in which a collection of example success states is assumed to be provided instead of entire expert trajectories.

Original: While RCE can be applied to LfO in principle, it tends to stay in a few expert states that are easy to reach in the offline setting, as discussed in Section C.2 of [5].
Revised: While RCE can be applied to LfO in principle, it tends to stay in a few expert states that are easy to reach in the offline setting, as discussed in Section C.2 of [5].

Original: Stationary DIstribution Correction Estimation (DICE) DICE-family algorithms perform stationary distribution estimation, and many of them have been proposed for off-policy evaluation [4, 29, 42– 44].
Revised: Stationary DIstribution Correction Estimation (DICE) DICE-family algorithms perform stationary distribution estimation, and many of them have been proposed for off-policy evaluation [4, 29, 42– 44].

Original: Other lines of works consider reinforcement learning [22, 23, 30], offline policy selection [41].
Revised: Other lines of works consider reinforcement learning [22, 23, 30], offline policy selection [41].

Original: ValueDICE [17] and OPOLO [45] derive off-policy IL and LfO objectives using DICE.
Revised: ValueDICE [17] and OPOLO [45] derive off-policy IL and LfO objectives using DICE.

Original: However, they suffer from numerical instability in the offline setting due to the nested min-max optimization and
Revised: However, they suffer from numerical instability in the offline setting due to the nested min-max optimization and

Original: OOD action evaluation.
Revised: OOD action evaluation.

Original: DemoDICE [15] is an offline IL algorithm that directly optimizes stationary distribution as ours and reduces to solving a convex minimization.
Revised: DemoDICE [15] is an offline IL algorithm that directly optimizes stationary distribution as ours and reduces to solving a convex minimization.

Original: Yet, it requires expert action labels and is not directly applicable to LfO. Concurrent to our work, SMODICE [28] is an offline
Revised: Yet, it requires expert action labels and is not directly applicable to LfO. Concurrent to our work, SMODICE [28] is an offline

Original: LfO method, aiming to match stationary state distributions by minimizing an objective similar to
Revised: LfO method, aiming to match stationary state distributions by minimizing an objective similar to

Original: OPOLO [45] in (3).
Revised: OPOLO [45] in (3).

Original: Thus it also relies on potentially loose upper bound of the divergence.
Revised: Thus it also relies on potentially loose upper bound of the divergence.

Original: In addition, matching the state distributions may not be sufficient to recover the expert’s behavior, on which we provide detailed discussion in Appendix A and Remark B.1.
Revised: In addition, matching the state distributions may not be sufficient to recover the expert’s behavior, on which we provide detailed discussion in Appendix A and Remark B.1.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate LobsDICE on both tabular and continuous MDPs.
Revised: In this section, we evaluate LobsDICE on both tabular and continuous MDPs.

Original: We use four baseline methods in tabular MDPs: BC on imperfect demonstrations, BCO [38], and OPOLO [45].
Revised: We use four baseline methods in tabular MDPs: BC on imperfect demonstrations, BCO [38], and OPOLO [45].

Original: Additionally, we designed a strong baseline DemoDICEfO, which extends the state-of-the-art offline
Revised: Additionally, we designed a strong baseline DemoDICEfO, which extends the state-of-the-art offline

Original: IL algorithm, DemoDICE [15].
Revised: IL algorithm, DemoDICE [15].

Original: DemoDICEfO trains an inverse dynamics model, uses it to fill the missing actions in the expert demonstrations, and then runs DemoDICE using the approximate expert demonstrations and the imperfect demonstrations.
Revised: DemoDICEfO trains an inverse dynamics model, uses it to fill the missing actions in the expert demonstrations, and then runs DemoDICE using the approximate expert demonstrations and the imperfect demonstrations.

Original: For continuous control tasks, we use two additional baselines: IQ-Learn [8] and RCE [5].
Revised: For continuous control tasks, we use two additional baselines: IQ-Learn [8] and RCE [5].

Original: 5.1 Random MDPs
Revised: 5.1 Random MDPs

Original: We first evaluate LobsDICE and baseline algorithms on randomly generated finite MDPs using a varying number of expert/imperfect trajectories and different degrees of stochasticity in the environment .
Revised: We first evaluate LobsDICE and baseline algorithms on randomly generated finite MDPs using a varying number of expert/imperfect trajectories and different degrees of stochasticity in the environment .

Original: We follow the experimental protocol in previous offline RL works [20–22] but with additional control on the stochasticity of transition probabilities.
Revised: We follow the experimental protocol in previous offline RL works [20–22] but with additional control on the stochasticity of transition probabilities.

Original: We conduct repeated experiments for 1K runs.
Revised: We conduct repeated experiments for 1K runs.

Original: For each run, (1) a random MDP is generated, (2) expert trajectories and imperfect trajectories are collected, and (3) each offline LfO algorithm is tested on the collected offline dataset.
Revised: For each run, (1) a random MDP is generated, (2) expert trajectories and imperfect trajectories are collected, and (3) each offline LfO algorithm is tested on the collected offline dataset.

Original: We evaluate
Revised: We evaluate

Original: [Figure]
Revised: [Figure]

Original: the performance of each algorithm by measuring the total variation distance between state-transition stationary distributions by the expert policy and the learned policy, i.e. D TV ( ¯ d π ( s, s ′ )
Revised: the performance of each algorithm by measuring the total variation distance between state-transition stationary distributions by the expert policy and the learned policy, i.e. D TV ( ¯ d π ( s, s ′ )

Original: ∥ ¯ d π E ( s, s ′ )) .
Revised: ∥ ¯ d π E ( s, s ′ )) .

Original: For the tabular MDP experiments, we adopt tabular methods but not function approximation.
Revised: For the tabular MDP experiments, we adopt tabular methods but not function approximation.

Original: Our tabular LobsDICE optimizes (16) on the empirical MDP model constructed from the action-labeled dataset D I while extracting the policy through (17).
Revised: Our tabular LobsDICE optimizes (16) on the empirical MDP model constructed from the action-labeled dataset D I while extracting the policy through (17).

Original: The pseudocode for tabular LobsDICE can be found in Appendix G.1.
Revised: The pseudocode for tabular LobsDICE can be found in Appendix G.1.

Original: The detailed experimental setup such as random MDP generation and offline dataset generation can be found in Appendix I.1.
Revised: The detailed experimental setup such as random MDP generation and offline dataset generation can be found in Appendix I.1.

Original: Results Figure 1 presents the results in random MDP experiments, where β is the hyperparameter that controls the stochasticity of the underlying MDP.
Revised: Results Figure 1 presents the results in random MDP experiments, where β is the hyperparameter that controls the stochasticity of the underlying MDP.

Original: The first row corresponds to the case when β = 0 .
Revised: The first row corresponds to the case when β = 0 .

Original: 01 (nearly deterministic MDP).
Revised: 01 (nearly deterministic MDP).

Original: In this situation, the inverse dynamics disagreement will be close to zero, i.e. D KL ( d π 1 ( a | s, s ′ )
Revised: In this situation, the inverse dynamics disagreement will be close to zero, i.e. D KL ( d π 1 ( a | s, s ′ )

Original: ||
Revised: ||

Original: d π 2 ( a | s, s ′ ))
Revised: d π 2 ( a | s, s ′ ))

Original: ≈ 0 for any two policies π 1 and π 2 .
Revised: ≈ 0 for any two policies π 1 and π 2 .

Original: Thus, the algorithms whose performance directly relies on the IDM’s accuracy (i.e. BCO and DemoDICEfO) even perform very well since it is very easy to learn a perfect inverse dynamics model in this scenario.
Revised: Thus, the algorithms whose performance directly relies on the IDM’s accuracy (i.e. BCO and DemoDICEfO) even perform very well since it is very easy to learn a perfect inverse dynamics model in this scenario.

Original: OPOLO’s upper bound gap (4) will also be close to zero, thus OPOLO directly minimizes the divergence of state-transition distributions.
Revised: OPOLO’s upper bound gap (4) will also be close to zero, thus OPOLO directly minimizes the divergence of state-transition distributions.

Original: As a result, there is no performance gap among different algorithms, except for BC whose performance is determined by the quality of imperfect demonstrations.
Revised: As a result, there is no performance gap among different algorithms, except for BC whose performance is determined by the quality of imperfect demonstrations.

Original: Also, the performance of all algorithms (except for BC) improves as more data is given, which is natural.
Revised: Also, the performance of all algorithms (except for BC) improves as more data is given, which is natural.

Original: The second row and the third row in Figure 1 presents the result when β = 0 .
Revised: The second row and the third row in Figure 1 presents the result when β = 0 .

Original: 1 (weakly stochastic
Revised: 1 (weakly stochastic

Original: MDP) and β = 1 . 

Revised: MDP) and β = 1 .

Original: (highly stochastic MDP) respectively.
Revised: (highly stochastic MDP) respectively.

Original: In the stochastic MDPs, the IDM trained by the imperfect demonstrations faces a challenge in predicting the expert’s actions accurately (more challenging as β gets larger).
Revised: In the stochastic MDPs, the IDM trained by the imperfect demonstrations faces a challenge in predicting the expert’s actions accurately (more challenging as β gets larger).

Original: As a consequence, BCO gets suboptimal and its suboptimality cannot be improved even if more data is given.
Revised: As a consequence, BCO gets suboptimal and its suboptimality cannot be improved even if more data is given.

Original: DemoDICEfO performs better than BCO since it additionally considers the distributional shift by considering state distribution matching.
Revised: DemoDICEfO performs better than BCO since it additionally considers the distributional shift by considering state distribution matching.

Original: However, it is still suboptimal due to its nature that directly depends on the quality of inferred action by the learned
Revised: However, it is still suboptimal due to its nature that directly depends on the quality of inferred action by the learned

Original: IDM.
Revised: IDM.

Original: OPOLO does not rely on the learned IDM and outperforms both BCO and DemoDICEfO.
Revised: OPOLO does not rely on the learned IDM and outperforms both BCO and DemoDICEfO.

Original: Still, OPOLO can be inherently suboptimal due to its nature of optimizing the upper bound unless the underlying transition dynamics are deterministic and injective.
Revised: Still, OPOLO can be inherently suboptimal due to its nature of optimizing the upper bound unless the underlying transition dynamics are deterministic and injective.

Original: This upper bound gap (4) is not controllable by the algorithm and implies that OPOLO can be suboptimal even given an infinite
Revised: This upper bound gap (4) is not controllable by the algorithm and implies that OPOLO can be suboptimal even given an infinite

Original: [Figure]
Revised: [Figure]

Original: amount of data with sufficient dataset coverage, which can be seen in the rightmost figures.
Revised: amount of data with sufficient dataset coverage, which can be seen in the rightmost figures.

Original: Finally, our tabular LobsDICE using (16) essentially solves the exact state-transition distribution matching problem (as α → 0 ).
Revised: Finally, our tabular LobsDICE using (16) essentially solves the exact state-transition distribution matching problem (as α → 0 ).

Original: LobsDICE is the only offline LfO algorithm that can asymptotically recovers the expert’s state demonstrations even though the underlying MDP is stochastic.
Revised: LobsDICE is the only offline LfO algorithm that can asymptotically recovers the expert’s state demonstrations even though the underlying MDP is stochastic.

Original: 5.2 Continuous control tasks (Gym-MuJoCo)
Revised: 5.2 Continuous control tasks (Gym-MuJoCo)

Original: We present the empirical performance of LobsDICE and baselines on MuJoCo [37] continuous control tasks using the OpenAI Gym [3] framework.
Revised: We present the empirical performance of LobsDICE and baselines on MuJoCo [37] continuous control tasks using the OpenAI Gym [3] framework.

Original: We utilize the D4RL dataset [6] for offline
Revised: We utilize the D4RL dataset [6] for offline

Original: LfO tasks in four MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant.
Revised: LfO tasks in four MuJoCo environments: Hopper, Walker2d, HalfCheetah, and Ant.

Original: Implementation details for LobsDICE and baseline algorithms such as hyperparameters and evaluation metric are provided in Appendix I.2.
Revised: Implementation details for LobsDICE and baseline algorithms such as hyperparameters and evaluation metric are provided in Appendix I.2.

Original: Task setup For each MuJoCo environments, we employ expert-v2 , medium-v2 , and random-v from D4RL datasets [6].
Revised: Task setup For each MuJoCo environments, we employ expert-v2 , medium-v2 , and random-v from D4RL datasets [6].

Original: Across all environments, we consider three tasks, each of which uses different imperfect demonstrations while sharing the same expert observations.
Revised: Across all environments, we consider three tasks, each of which uses different imperfect demonstrations while sharing the same expert observations.

Original: First, we construct the stateonly expert demonstration D E
Revised: First, we construct the stateonly expert demonstration D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: i } N E i =
Revised: i } N E i =

Original: 1 using the first 5 trajectories in expert-v2 .
Revised: 1 using the first 5 trajectories in expert-v2 .

Original: Then, we use trajectories in expert-v2 , medium-v2 , and random-v2 to construct imperfect demonstrations with different ratios.
Revised: Then, we use trajectories in expert-v2 , medium-v2 , and random-v2 to construct imperfect demonstrations with different ratios.

Original: We denote the composition of imperfect demonstrations as ( X, Y, Z ) in the title of each subplot in Figure 2, which means that the imperfect dataset consists of X trajectories from expert-v2 , Y trajectories from medium-v2 , and Z trajectories from random-v2 .
Revised: We denote the composition of imperfect demonstrations as ( X, Y, Z ) in the title of each subplot in Figure 2, which means that the imperfect dataset consists of X trajectories from expert-v2 , Y trajectories from medium-v2 , and Z trajectories from random-v2 .

Original: Results Figure 2 summarizes that the empirical results of LobsDICE and baselines on continuous control tasks.
Revised: Results Figure 2 summarizes that the empirical results of LobsDICE and baselines on continuous control tasks.

Original: We first remark that LobsDICE (blue) significantly outperforms OPOLO (green) in all tasks across all domains, although both LobsDICE and OPOLO are DICE-based algorithms.
Revised: We first remark that LobsDICE (blue) significantly outperforms OPOLO (green) in all tasks across all domains, although both LobsDICE and OPOLO are DICE-based algorithms.

Original: The failure of OPOLO comes from its numerical instability due to its dependence on nested optimization and using OOD action values during training.
Revised: The failure of OPOLO comes from its numerical instability due to its dependence on nested optimization and using OOD action values during training.

Original: In contrast, LobsDICE solves a single minimization (23) while it does not involve any evaluation on OOD actions, thus it is optimized stably.
Revised: In contrast, LobsDICE solves a single minimization (23) while it does not involve any evaluation on OOD actions, thus it is optimized stably.

Original: IQ-Learn (purple) also suffers from numerical instability due to the usage of OOD action values during training, showing poor performance similar to OPOLO.
Revised: IQ-Learn (purple) also suffers from numerical instability due to the usage of OOD action values during training, showing poor performance similar to OPOLO.

Original: RCE (brown) tends to stay in a few states that are easy to reach in the offline setting, rather than following the entire expert trajectories.
Revised: RCE (brown) tends to stay in a few states that are easy to reach in the offline setting, rather than following the entire expert trajectories.

Original: Naive BC on imperfect demonstrations (black) is inherently suboptimal since it does not consider distribution-matching with the expert’s observation at all.
Revised: Naive BC on imperfect demonstrations (black) is inherently suboptimal since it does not consider distribution-matching with the expert’s observation at all.

Original: While BCO (orange) exploits the expert’s demonstrations with the inferred actions by the IDM, its policy learning is done only on the very scarce expert dataset (i.e. 5 trajectories), which makes the algorithm perform not well.
Revised: While BCO (orange) exploits the expert’s demonstrations with the inferred actions by the IDM, its policy learning is done only on the very scarce expert dataset (i.e. 5 trajectories), which makes the algorithm perform not well.

Original: DemoDICEfO (red) exploits both expert demonstrations (where the missing actions are filled with the IDM) and the abundant imperfect demonstrations, but its performance is affected by the quality of the learned IDM.
Revised: DemoDICEfO (red) exploits both expert demonstrations (where the missing actions are filled with the IDM) and the abundant imperfect demonstrations, but its performance is affected by the quality of the learned IDM.

Original: We empirically observe that the IDM error (on the true expert data) increases as the proportion of the non-expert data (i.e. medium-v2 and random-v2 ) increases, resulting in performance degradation of both
Revised: We empirically observe that the IDM error (on the true expert data) increases as the proportion of the non-expert data (i.e. medium-v2 and random-v2 ) increases, resulting in performance degradation of both

Original: BCO and DemoDICEfO.
Revised: BCO and DemoDICEfO.

Original: Finally, LobsDICE is the only algorithm that was able to fully recover the expert’s performance regardless of the increase of non-expert data in the imperfect demonstrations, significantly outperforming baseline algorithms.
Revised: Finally, LobsDICE is the only algorithm that was able to fully recover the expert’s performance regardless of the increase of non-expert data in the imperfect demonstrations, significantly outperforming baseline algorithms.

Original: This result highlights the effectiveness of our method that solves a state-transition stationary matching problem in a principled manner.
Revised: This result highlights the effectiveness of our method that solves a state-transition stationary matching problem in a principled manner.

Original: We provide additional experiments in Appendix H.
Revised: We provide additional experiments in Appendix H.

Original:  Conclusion
Revised:  Conclusion

Original: We presented LobsDICE, an algorithm for offline learning from observations (LfO), which successfully achieves state-of-the-art performance on various tabular and continuous tasks.
Revised: We presented LobsDICE, an algorithm for offline learning from observations (LfO), which successfully achieves state-of-the-art performance on various tabular and continuous tasks.

Original: We formulated the offline LfO as a state-transition stationary distribution matching problem, where the stationary distribution is optimized via convex minimization.
Revised: We formulated the offline LfO as a state-transition stationary distribution matching problem, where the stationary distribution is optimized via convex minimization.

Original: Experimental results demonstrated that LobsDICE achieves promising performance in both tabular and continuous offline LfO tasks.
Revised: Experimental results demonstrated that LobsDICE achieves promising performance in both tabular and continuous offline LfO tasks.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by National Research Foundation (NRF) of Korea (NRF2019R1A2C1087634), Field-oriented Technology Development Project for Customs Administration through National Research Foundation (NRF) of Korea funded by the Ministry of Science & ICT and Korea Customs Service (NRF-2021M3I1A1097938), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural
Revised: This work was supported by National Research Foundation (NRF) of Korea (NRF2019R1A2C1087634), Field-oriented Technology Development Project for Customs Administration through National Research Foundation (NRF) of Korea funded by the Ministry of Science & ICT and Korea Customs Service (NRF-2021M3I1A1097938), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural

Original: Language Processing; No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning
Revised: Language Processing; No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning

Original: Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, Artificial
Revised: Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, Artificial

Original: Intelligence Graduate School Program (KAIST); No.2021-0-02068, Artificial Intelligence Innovation Hub), and Electronics and Telecommunications Research Institute (ETRI) grant funded by the
Revised: Intelligence Graduate School Program (KAIST); No.2021-0-02068, Artificial Intelligence Innovation Hub), and Electronics and Telecommunications Research Institute (ETRI) grant funded by the

Original: Korean government (22ZS1100, Core Technology Research for Self-Improving Integrated Artificial
Revised: Korean government (22ZS1100, Core Technology Research for Self-Improving Integrated Artificial

Original: Intelligence System).
Revised: Intelligence System).

Original: Hongseok Yang was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R1A5A1059921) and also by the Institute for Basic Science (IBS-R029-C1).
Revised: Hongseok Yang was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R1A5A1059921) and also by the Institute for Basic Science (IBS-R029-C1).

Original: Kee-Eung
Revised: Kee-Eung

Original: Kim was supported by KAIST-NAVER Hypercreative AI Center.
Revised: Kim was supported by KAIST-NAVER Hypercreative AI Center.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See abstract and introduction (Section 1) (b) Did you describe the limitations of your work?
Revised: [Yes] See abstract and introduction (Section 1) (b) Did you describe the limitations of your work?

Original: [Yes] Due to space limit, we discuss it in the supplementary material (Section L) (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] Due to space limit, we discuss it in the supplementary material (Section L) (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Due to space limit, we discuss it in the supplementary material (Section N).
Revised: [Yes] Due to space limit, we discuss it in the supplementary material (Section N).

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] In appropriate places.
Revised: [Yes] In appropriate places.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Due to space limit, we provide proofs in the supplementary material.
Revised: [Yes] Due to space limit, we provide proofs in the supplementary material.

Original: See Section F 3.
Revised: See Section F 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We submit the code as supplementary material.
Revised: [Yes] We submit the code as supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We specify them in the supplementary material.
Revised: [Yes] We specify them in the supplementary material.

Original: See Section I (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: See Section I (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report them in each figure.
Revised: [Yes] We report them in each figure.

Original: See Figure 1 and 2.
Revised: See Figure 1 and 2.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We specify them in the supplementary material.
Revised: [Yes] We specify them in the supplementary material.

Original: See Section K 4.
Revised: See Section K 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite Random MDP protocols and D4RL dataset.
Revised: [Yes] We cite Random MDP protocols and D4RL dataset.

Original: See Section 5.
Revised: See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] D4RL dataset is licensed under the Apache 2.0.
Revised: [Yes] D4RL dataset is licensed under the Apache 2.0.

Original: See Section M. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: See Section M. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] All the datasets we used are public.
Revised: [N/A] All the datasets we used are public.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: None of datasets contain identifiable contents.
Revised: None of datasets contain identifiable contents.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: However, if we perform
Revised: However, if we perform

Original: ) -matching even
Revised: ) -matching even

Original: 1 can be obtained as a resulting which is opposite of to fact that
Revised: 1 can be obtained as a resulting which is opposite of to fact that

Original: 2 share the same state stationary distribution of a uniform distribution:
Revised: 2 share the same state stationary distribution of a uniform distribution:

Original: d π ( s ) = 1 | S | ∀ s .
Revised: d π ( s ) = 1 | S | ∀ s .

Original: Thus, both π 1 and π must optimal of min π D KL ( d
Revised: Thus, both π 1 and π must optimal of min π D KL ( d

Original: ( s ) ||
Revised: ( s ) ||

Original: d E ( s )) .
Revised: d E ( s )) .

Original: In contrast, the d ( s, s
Revised: In contrast, the d ( s, s

Original: ) -matching has a unique solution of π 2 , exploiting the directionality information (
Revised: ) -matching has a unique solution of π 2 , exploiting the directionality information (

Original: i e. moving counterclockwise) of the expert’s trajectories.
Revised: i e. moving counterclockwise) of the expert’s trajectories.

Original: B LobsDICE for State Occupancy Matching
Revised: B LobsDICE for State Occupancy Matching

Original: While we have provided a counter-example where d ( s ) -matching can fail to recover the expert’s behavior in Appendix A, we still can derive an offline LfO algorithm for d ( s ) -matching.
Revised: While we have provided a counter-example where d ( s ) -matching can fail to recover the expert’s behavior in Appendix A, we still can derive an offline LfO algorithm for d ( s ) -matching.

Original: For the given policy π , its state stationary distribution is defined as:
Revised: For the given policy π , its state stationary distribution is defined as:

Original: [Equation]
Revised: [Equation]

Original: Then, similar to the state-transition occupancy matching objective (5), we formulate a state occupancy matching problem as follows:
Revised: Then, similar to the state-transition occupancy matching objective (5), we formulate a state occupancy matching problem as follows:

Original: [Equation]
Revised: [Equation]

Original: where the hyperparameter α > 0 balances between encouraging state matching and preventing distribution shift from the distribution of imperfect demonstrations.
Revised: where the hyperparameter α > 0 balances between encouraging state matching and preventing distribution shift from the distribution of imperfect demonstrations.

Original: This objective can be reformulated in terms of optimizing stationary distribution:
Revised: This objective can be reformulated in terms of optimizing stationary distribution:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, we consider the Lagrangian for the constrained optimization (26-28):
Revised: Then, we consider the Lagrangian for the constrained optimization (26-28):

Original: [Equation]
Revised: [Equation]

Original: where ν ( s ′ ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (27), and µ ( s ) ∈ R are the Lagrange multipliers for the marginalization constraints (28).
Revised: where ν ( s ′ ) ∈ R are the Lagrange multipliers for the Bellman flow constraints (27), and µ ( s ) ∈ R are the Lagrange multipliers for the marginalization constraints (28).

Original: To make the optimization tractable in the offline setting, we rearrange the terms in (29), introducing new optimization variables w and ¯ w that denote stationary distribution correction ratios for ( s, a ) and s respectively.
Revised: To make the optimization tractable in the offline setting, we rearrange the terms in (29), introducing new optimization variables w and ¯ w that denote stationary distribution correction ratios for ( s, a ) and s respectively.

Original: [Equation]
Revised: [Equation]

Original: We introduce the log ratio r ( s ) = log d E ( s )
Revised: We introduce the log ratio r ( s ) = log d E ( s )

Original: d I ( s ) to make expectation for d I instead of d E .
Revised: d I ( s ) to make expectation for d I instead of d E .

Original: The log ratio r ( s ) can be estimated by using a pretrained discriminator c : S → [0 , 1]
Revised: The log ratio r ( s ) can be estimated by using a pretrained discriminator c : S → [0 , 1]

Original: [Equation]
Revised: [Equation]

Original: where c is trained by:
Revised: where c is trained by:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: it is proven that the optimal discriminator c ∗ satisfies
Revised: it is proven that the optimal discriminator c ∗ satisfies

Original: [Equation]
Revised: [Equation]

Original: Therefore, r ( s ) can be estimated by:
Revised: Therefore, r ( s ) can be estimated by:

Original: [Equation]
Revised: [Equation]

Original: Similar to Proposition 3.
Revised: Similar to Proposition 3.

Original: [Equation]
Revised: [Equation]

Original: we can easily derive the closed-form solution to the inner maximization in (30):
Revised: we can easily derive the closed-form solution to the inner maximization in (30):

Original: [Equation]
Revised: [Equation]

Original: Finally, by plugging these closed-form solutions into (30)
Revised: Finally, by plugging these closed-form solutions into (30)

Original: [Equation]
Revised: [Equation]

Original: the nested min-max optimization of (30) is reduced to a single minimization:
Revised: the nested min-max optimization of (30) is reduced to a single minimization:

Original: [Equation]
Revised: [Equation]

Original: which is a convex function of µ and ν .
Revised: which is a convex function of µ and ν .

Original: Finally, once we obtain the optimal solution ( µ ∗ , ν ∗ ) of (31), we have w
Revised: Finally, once we obtain the optimal solution ( µ ∗ , ν ∗ ) of (31), we have w

Original: [Equation]
Revised: [Equation]

Original: ,ν
Revised: ,ν

Original: [Equation]
Revised: [Equation]

Original: s, a
Revised: s, a

Original: [Equation]
Revised: [Equation]

Original: Then, we can extract an optimal policy via weighted BC:
Revised: Then, we can extract an optimal policy via weighted BC:

Original: [Equation]
Revised: [Equation]

Original: Remark B
Revised: Remark B

Original: [Equation]
Revised: [Equation]

Original: To the best of our knowledge, (31) is the first result to directly solve the state distribution matching problem in an offline setting.
Revised: To the best of our knowledge, (31) is the first result to directly solve the state distribution matching problem in an offline setting.

Original: Although SMODICE [28], a concurrent work with ours, also aims to solve offline LfO in terms of optimizing stationary distribution, it essentially minimizes the (potentially loose) upper bound ,
Revised: Although SMODICE [28], a concurrent work with ours, also aims to solve offline LfO in terms of optimizing stationary distribution, it essentially minimizes the (potentially loose) upper bound ,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The upper bound gap (33) − (35) is given by D KL ( π ( a | s )
Revised: The upper bound gap (33) − (35) is given by D KL ( π ( a | s )

Original: ||
Revised: ||

Original: π I ( a | s )) .
Revised: π I ( a | s )) .

Original: Unlike OPOLO of (4), this upper bound gap does not vanish even when the transition is deterministic.
Revised: Unlike OPOLO of (4), this upper bound gap does not vanish even when the transition is deterministic.

Original: Consequently, SMODICE may not be able to precisely recover the expert’s state visitations even given an infinite amount of data due to the upper bound gap unless d I is collected purely by expert.
Revised: Consequently, SMODICE may not be able to precisely recover the expert’s state visitations even given an infinite amount of data due to the upper bound gap unless d I is collected purely by expert.

Original: In contrast, we are directly minimizing the divergence: solving (31) is exactly equivalent to solving (25).
Revised: In contrast, we are directly minimizing the divergence: solving (31) is exactly equivalent to solving (25).

Original: As can be seen in
Revised: As can be seen in

Original: Figure 4, SMODICE optimizing (33) fails to recover the expert’s behavior due to the upper bound gap that is irreducible even for deterministic MDPs.
Revised: Figure 4, SMODICE optimizing (33) fails to recover the expert’s behavior due to the upper bound gap that is irreducible even for deterministic MDPs.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ¯
Revised: ¯

Original: [Figure]
Revised: [Figure]

Original: π k ¯
Revised: π k ¯

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: As the level of stochasticity increases, baselines fall into suboptimal, even the number of state-only expert demonstrations and imperfect demonstrations increases, while LobsDICE goes to optimal.
Revised: As the level of stochasticity increases, baselines fall into suboptimal, even the number of state-only expert demonstrations and imperfect demonstrations increases, while LobsDICE goes to optimal.

Original: For each algorithm, we measure the performance using total variation between state-transition stationary distributions of expert and learned policy.
Revised: For each algorithm, we measure the performance using total variation between state-transition stationary distributions of expert and learned policy.

Original: We plot the mean and standard error of total variations
Revised: We plot the mean and standard error of total variations

Original: TV( ¯ d π ( s, s ′ ) , ¯ d π E ( s, s ′ )) over 1000 random seeds.
Revised: TV( ¯ d π ( s, s ′ ) , ¯ d π E ( s, s ′ )) over 1000 random seeds.

Original: C Validity of Weighted Behavior Cloning for Policy Extraction
Revised: C Validity of Weighted Behavior Cloning for Policy Extraction

Original: For tabular MDPs, we can show that the weighted behavior cloning of (17)
Revised: For tabular MDPs, we can show that the weighted behavior cloning of (17)

Original: [Equation]
Revised: [Equation]

Original: (17) exactly yields an optimal policy π ∗ of the original optimization problem (5).
Revised: (17) exactly yields an optimal policy π ∗ of the original optimization problem (5).

Original: First, the optimization problem (5) and the constrained optimization problem (6-8) are equivalent with the following relationship between π ∗ and d ∗ :
Revised: First, the optimization problem (5) and the constrained optimization problem (6-8) are equivalent with the following relationship between π ∗ and d ∗ :

Original: [Equation]
Revised: [Equation]

Original: (36) where d ∗ is the optimal solution of both (6-8) and (10).
Revised: (36) where d ∗ is the optimal solution of both (6-8) and (10).

Original: Then, w µ ∗ ,ν ∗ ( s, a ) is computed by (15) with ( µ ∗ , ν ∗ ) , the optimal solution of both (16) and (11).
Revised: Then, w µ ∗ ,ν ∗ ( s, a ) is computed by (15) with ( µ ∗ , ν ∗ ) , the optimal solution of both (16) and (11).

Original: Consequently, w µ ∗ ,ν ∗ ( s, a ) =
Revised: Consequently, w µ ∗ ,ν ∗ ( s, a ) =

Original: d ∗ ( s,a ) d I ( s,a ) holds, and thus the weighted behavior cloning of (17) essentially performs the following:
Revised: d ∗ ( s,a ) d I ( s,a ) holds, and thus the weighted behavior cloning of (17) essentially performs the following:

Original: [Equation]
Revised: [Equation]

Original: Now, consider the Lagrangian for the constrained optimization (37-38):
Revised: Now, consider the Lagrangian for the constrained optimization (37-38):

Original: [Equation]
Revised: [Equation]

Original: where λ ( s ) is the Lagrange multiplier for the constraint (38).
Revised: where λ ( s ) is the Lagrange multiplier for the constraint (38).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: considering the constraint (cid:80) a x ∗ ( a | s )
Revised: considering the constraint (cid:80) a x ∗ ( a | s )

Original: = 1 ∀ s
Revised: = 1 ∀ s

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: 41) which is identical to (36).
Revised: 41) which is identical to (36).

Original: In summary, the weighted behavior cloning performs the process of extracting the optimal policy π ∗ from w µ ∗ ,ν ∗ .
Revised: In summary, the weighted behavior cloning performs the process of extracting the optimal policy π ∗ from w µ ∗ ,ν ∗ .

Original: D Challenges of Extending DemoDICE to Offline Learning from Observation
Revised: D Challenges of Extending DemoDICE to Offline Learning from Observation

Original: DemoDICE [15] is an algorithm that optimizes the state-action stationary distribution d
Revised: DemoDICE [15] is an algorithm that optimizes the state-action stationary distribution d

Original: ( s, a )
Revised: ( s, a )

Original: [Equation]
Revised: [Equation]

Original: where its naive application to ( s, s ′ )
Revised: where its naive application to ( s, s ′ )

Original: -distribution matching problem yields:
Revised: -distribution matching problem yields:

Original: [Equation]
Revised: [Equation]

Original: However, estimating E ( s,
Revised: However, estimating E ( s,

Original: [Equation]
Revised: [Equation]

Original: (cid:105) is intractable in the offline setting due to its inclusion of marginalization over a inside the log( · ) .
Revised: (cid:105) is intractable in the offline setting due to its inclusion of marginalization over a inside the log( · ) .

Original: This challenge can be addressed by using the upper bound proposed by OPOLO [45]:
Revised: This challenge can be addressed by using the upper bound proposed by OPOLO [45]:

Original: [Equation]
Revised: [Equation]

Original: Replacing D KL ( ¯ d π ( s, s ′ )
Revised: Replacing D KL ( ¯ d π ( s, s ′ )

Original: ||
Revised: ||

Original: d E ( s, s ′ )) with the (potentially loose) upper bound in the objective function yields the following tractable (but biased) optimization problem:
Revised: d E ( s, s ′ )) with the (potentially loose) upper bound in the objective function yields the following tractable (but biased) optimization problem:

Original: [Equation]
Revised: [Equation]

Original: In contrast, we address the intractability challenge by introducing an additional optimization variable ¯ d ( s, s ′ ) along with the marginalization constraint (8).
Revised: In contrast, we address the intractability challenge by introducing an additional optimization variable ¯ d ( s, s ′ ) along with the marginalization constraint (8).

Original: This allows for tractable offline optimization by (11,16) without introducing bias, which is our novel contribution.
Revised: This allows for tractable offline optimization by (11,16) without introducing bias, which is our novel contribution.

Original: E More Discussions on LobsDICE with Finite Samples
Revised: E More Discussions on LobsDICE with Finite Samples

Original: In this section, we provide additional discussions on LobsDICE with finite samples.
Revised: In this section, we provide additional discussions on LobsDICE with finite samples.

Original: E.1 LobsDICE with sample-based approximation: distribution matching on the MLE MDP
Revised: E.1 LobsDICE with sample-based approximation: distribution matching on the MLE MDP

Original: We first show that LobsDICE that optimizes (18) is equivalent to performing optimization of (6-8) on the MLE MDP constructed by D I .
Revised: We first show that LobsDICE that optimizes (18) is equivalent to performing optimization of (6-8) on the MLE MDP constructed by D I .

Original: Let ˆ M \ R = ⟨ S, A, ˆ T, (cid:98) p 0 , γ ⟩ be the MLE MDP constructed using D I .
Revised: Let ˆ M \ R = ⟨ S, A, ˆ T, (cid:98) p 0 , γ ⟩ be the MLE MDP constructed using D I .

Original: Then, constrained optimization (6-8) for the MLE MDP ˆ M \ R can be formulated as:
Revised: Then, constrained optimization (6-8) for the MLE MDP ˆ M \ R can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) p 0 , (cid:98) d E
Revised: where (cid:98) p 0 , (cid:98) d E

Original: [Equation]
Revised: [Equation]

Original: and (cid:98) d I are empirical distributions of D 0 , D E
Revised: and (cid:98) d I are empirical distributions of D 0 , D E

Original: [Equation]
Revised: [Equation]

Original: and D I , respectively.
Revised: and D I , respectively.

Original: We consider
Revised: We consider

Original: Lagrangian for the above constrained optimization:
Revised: Lagrangian for the above constrained optimization:

Original: [Equation]
Revised: [Equation]

Original: Note that this is identical to (18).
Revised: Note that this is identical to (18).

Original: We can also derive the closed-form solution ( w Iµ,ν , ¯ w Iµ ) to the max w, ¯ w ≥
Revised: We can also derive the closed-form solution ( w Iµ,ν , ¯ w Iµ ) to the max w, ¯ w ≥

Original: [Equation]
Revised: [Equation]

Original: L I ( w, ¯ w, µ, ν ) :
Revised: L I ( w, ¯ w, µ, ν ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that this is different from (19) in that (49) is the closed-form solution for each ( s, a ) and ( s, s ′ ) , while (19) is the non-parametric closed-form solution for each sample x ∈ D I .
Revised: Note that this is different from (19) in that (49) is the closed-form solution for each ( s, a ) and ( s, s ′ ) , while (19) is the non-parametric closed-form solution for each sample x ∈ D I .

Original: Exploiting (49) does not introduce an additional bias, but requires evaluation of the expectation within the exp( · ) , i.e. transition model is needed.
Revised: Exploiting (49) does not introduce an additional bias, but requires evaluation of the expectation within the exp( · ) , i.e. transition model is needed.

Original: The last step is to plug the solution ( w Iµ,ν , ¯ w Iµ ) in (49) into L I ( w, ¯ w, µ, ν ) :
Revised: The last step is to plug the solution ( w Iµ,ν , ¯ w Iµ ) in (49) into L I ( w, ¯ w, µ, ν ) :

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: this can be rewritten as:
Revised: this can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: which is the objective function for our tabular LobsDICE (52)
Revised: which is the objective function for our tabular LobsDICE (52)

Original: [Equation]
Revised: [Equation]

Original: E.2 When will LobsDICE reduce to BCO?
Revised: E.2 When will LobsDICE reduce to BCO?

Original: LobsDICE reduces to BCO when D I is collected by an expert policy and D E is a dataset identical to
Revised: LobsDICE reduces to BCO when D I is collected by an expert policy and D E is a dataset identical to

Original: D I except that action is missing, i.e., D E
Revised: D I except that action is missing, i.e., D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: | ( s, a, s ′ )
Revised: | ( s, a, s ′ )

Original: ∈
Revised: ∈

Original: D I } .
Revised: D I } .

Original: In this situation, note that (cid:98) d
Revised: In this situation, note that (cid:98) d

Original: I ( s, s ′ ) and (cid:98) d
Revised: I ( s, s ′ ) and (cid:98) d

Original: I ( s, a ) are valid stationary distributions on the MLE MDP constructed by D I , since they satisfy both the Bellman flow constraint (47) and the marginalization constraint (48) on the MLE MDP.
Revised: I ( s, a ) are valid stationary distributions on the MLE MDP constructed by D I , since they satisfy both the Bellman flow constraint (47) and the marginalization constraint (48) on the MLE MDP.

Original: The (cid:98) d I ( s, s ′ ) and (cid:98) d
Revised: The (cid:98) d I ( s, s ′ ) and (cid:98) d

Original: I ( s, a ) are also the optimal solution of (46-48), making the KL divergences to 0:
Revised: I ( s, a ) are also the optimal solution of (46-48), making the KL divergences to 0:

Original: [Equation]
Revised: [Equation]

Original: Finally, the policy obtained by LobsDICE will simply reduce to BC on D I by noting that:
Revised: Finally, the policy obtained by LobsDICE will simply reduce to BC on D I by noting that:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, one can easily show that BC on D I is identical to BCO (i.e. BC on D E where the missing actions are inferred by the IDM trained by D I ), which concludes that LobsDICE is equivalent to BCO in this special case.
Revised: Furthermore, one can easily show that BC on D I is identical to BCO (i.e. BC on D E where the missing actions are inferred by the IDM trained by D I ), which concludes that LobsDICE is equivalent to BCO in this special case.

Original: However, D I will include demonstrations collected by non-expert policies in general, which makes the LobsDICE’s solution different from (and usually better than) the BCO’s solution in general cases.
Revised: However, D I will include demonstrations collected by non-expert policies in general, which makes the LobsDICE’s solution different from (and usually better than) the BCO’s solution in general cases.

Original: F Theoretical Analysis F.1 Closed-form solutions
Revised: F Theoretical Analysis F.1 Closed-form solutions

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.
Revised: For any ( µ, ν ) , the closed-form solution to the inner maximization of (11) , i.e.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = arg max
Revised: = arg max

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: w, µ, ν ) , is given by:
Revised: w, µ, ν ) , is given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let e µ,ν ( s, a, s ′ )
Revised: Let e µ,ν ( s, a, s ′ )

Original: = − µ ( s, s ′ )
Revised: = − µ ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) .
Revised: − ν ( s ) .

Original: Then, e µ,ν ( s, a ) =
Revised: Then, e µ,ν ( s, a ) =

Original: E s ′ ∼ T ( ·| s,a ) [ e µ,ν ( s, a, s ′ )] .
Revised: E s ′ ∼ T ( ·| s,a ) [ e µ,ν ( s, a, s ′ )] .

Original: For ( s, a ) with d I ( s, a ) > 0 , we can compute the derivative ∂
Revised: For ( s, a ) with d I ( s, a ) > 0 , we can compute the derivative ∂

Original: L ∂w ( s,a ) of L w.r.t.
Revised: L ∂w ( s,a ) of L w.r.t.

Original: w ( s, a ) as follows:
Revised: w ( s, a ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Similar to the aforementioned derivation, when ¯ d I ( s, s ′ ) > 0 , we can derive the derivative of L w.r
Revised: Similar to the aforementioned derivation, when ¯ d I ( s, s ′ ) > 0 , we can derive the derivative of L w.r

Original: [Equation]
Revised: [Equation]

Original: t.
Revised: t.

Original: ¯ w ( s, s ′ ) :
Revised: ¯ w ( s, s ′ ) :

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .
Revised: L ( w µ,ν , ¯ w µ , µ, ν ) is convex with respect to µ and ν .

Original: Proof.
Revised: Proof.

Original: Using the fact that exp( · ) is a convex function, we can easily prove the convexity of
Revised: Using the fact that exp( · ) is a convex function, we can easily prove the convexity of

Original: L ( w µ,ν , ¯ w µ,ν , µ, ν ) .
Revised: L ( w µ,ν , ¯ w µ,ν , µ, ν ) .

Original: For brevity, let L ( µ, ν ) :
Revised: For brevity, let L ( µ, ν ) :

Original: = L ( w µ,ν , ¯ w µ,ν , µ, ν ) .
Revised: = L ( w µ,ν , ¯ w µ,ν , µ, ν ) .

Original: Then, for given ( µ 1 , ν 1 ) , ( µ 2 , ν 2 ) and t ∈ [0 , 1] ,
Revised: Then, for given ( µ 1 , ν 1 ) , ( µ 2 , ν 2 ) and t ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: 2 ( s )]
Revised: 2 ( s )]

Original: + E d
Revised: + E d

Original: I
Revised: I

Original: [exp( r ( s, s ′ )
Revised: [exp( r ( s, s ′ )

Original: + tµ 1 ( s, s ′ )
Revised: + tµ 1 ( s, s ′ )

Original: + (1 − t ) µ 2 ( s
Revised: + (1 − t ) µ 2 ( s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: − 1)]
Revised: − 1)]

Original: + α
Revised: + α

Original: [Equation]
Revised: [Equation]

Original: d I (cid:104) exp (cid:16)
Revised: d I (cid:104) exp (cid:16)

Original: [Equation]
Revised: [Equation]

Original: E s ′ (cid:2) − tµ 1 ( s
Revised: E s ′ (cid:2) − tµ 1 ( s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: − (1 − t ) µ 2 ( s
Revised: − (1 − t ) µ 2 ( s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: + γtν 1 ( s ′ )
Revised: + γtν 1 ( s ′ )

Original: + γ
Revised: + γ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ν 2 ( s ′ )
Revised: ν 2 ( s ′ )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For the inequalities in the above formulation
Revised: For the inequalities in the above formulation

Original: [Equation]
Revised: [Equation]

Original: we use the fact that E d I [exp( · )] is a instance of convex functions.
Revised: we use the fact that E d I [exp( · )] is a instance of convex functions.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e. µ ν = arg min µ (cid:98) L ( µ, ν ) , is
Revised: For any ν , the closed-form solution to the minimization (20) with respect to µ , i.e. µ ν = arg min µ (cid:98) L ( µ, ν ) , is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For simplicity, let (cid:98) L ( µ, ν ) :
Revised: For simplicity, let (cid:98) L ( µ, ν ) :

Original: = (cid:98) L ( w µ,ν , ¯ w µ,ν , µ, ν ) and x :=
Revised: = (cid:98) L ( w µ,ν , ¯ w µ,ν , µ, ν ) and x :=

Original: ( s, a, s ′ ) .
Revised: ( s, a, s ′ ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: F.2 Surrogate Objective
Revised: F.2 Surrogate Objective

Original: We first show a property of ˜ L that will be used to prove propositions:
Revised: We first show a property of ˜ L that will be used to prove propositions:

Original: Lemma F.1.
Revised: Lemma F.1.

Original: For given function ν : S → R ,
Revised: For given function ν : S → R ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any C ∈ R , ˜ L ( ν + C )
Revised: For any C ∈ R , ˜ L ( ν + C )

Original: [Equation]
Revised: [Equation]

Original: E ( s,a,s ′ ) ∼ d I (cid:104) exp (cid:16) 1 

Revised: E ( s,a,s ′ ) ∼ d I (cid:104) exp (cid:16) 1

Original: + α ˆ
Revised: + α ˆ

Original: A
Revised: A

Original: ν
Revised: ν

Original: + C ( s, a, s ′ ) (cid:17)(cid:105) =
Revised: + C ( s, a, s ′ ) (cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )
Revised: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )

Original: + γ ( ν ( s ′ )
Revised: + γ ( ν ( s ′ )

Original: + C ) − ( ν ( s ) + C ) (cid:1)(cid:17)(cid:105) =
Revised: + C ) − ( ν ( s ) + C ) (cid:1)(cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )
Revised: ∼ d I (cid:104) exp (cid:16) 1 1+ α (cid:0) r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) (cid:1)(cid:17) exp (cid:16) ( γ − 1) C 1+ α (cid:17)(cid:105) =
Revised: − ν ( s ) (cid:1)(cid:17) exp (cid:16) ( γ − 1) C 1+ α (cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )
Revised: [ ν ( s ) + C ] + (1 + α ) log E ( s,a,s ′ )

Original: ∼ d I (cid:104) exp (cid:16) 1 1+ α ( r ( s, s ′ )
Revised: ∼ d I (cid:104) exp (cid:16) 1 1+ α ( r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ))
Revised: − ν ( s ))

Original: (cid:17)(cid:105) + ( γ − 1) C
Revised: (cid:17)(cid:105) + ( γ − 1) C

Original: = (1 − γ )
Revised: = (1 − γ )

Original: E s ∼ p 

Revised: E s ∼ p

Original: [ ν ( s )]
Revised: [ ν ( s )]

Original: + (1 + α ) log E ( s,a,s ′ ) ∼ d I (cid:104) exp (cid:16) 1 1+ α
Revised: + (1 + α ) log E ( s,a,s ′ ) ∼ d I (cid:104) exp (cid:16) 1 1+ α

Original: r ( s, s ′ )
Revised: r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ))
Revised: − ν ( s ))

Original: (cid:17)(cid:105) =
Revised: (cid:17)(cid:105) =

Original: ˜ L ( ν ) .
Revised: ˜ L ( ν ) .

Original: Now, we prove Proposition 3.4 and Proposition 3.5:
Revised: Now, we prove Proposition 3.4 and Proposition 3.5:

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: Let (cid:101) L ( (cid:101) ν ) be the function:
Revised: Let (cid:101) L ( (cid:101) ν ) be the function:

Original: [Equation]
Revised: [Equation]

Original: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )
Revised: Then, min (cid:98) ν (cid:98) L ( (cid:98) ν )

Original: = min (cid:101) ν (cid:101)
Revised: = min (cid:101) ν (cid:101)

Original: L ( (cid:101) ν ) holds.
Revised: L ( (cid:101) ν ) holds.

Original: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .
Revised: Also, (cid:101) L ( (cid:101) ν ) is convex with respect to (cid:101) ν .

Original: Proof.
Revised: Proof.

Original: First of all, from the fact that log( x + 1) ≤ x for all x > − 1 , we can easily conclude that min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) .
Revised: First of all, from the fact that log( x + 1) ≤ x for all x > − 1 , we can easily conclude that min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) .

Original: Now, we will show that min ν ˆ L ( ν ) ≤ min ν ˜ L ( ν ) .
Revised: Now, we will show that min ν ˆ L ( ν ) ≤ min ν ˜ L ( ν ) .

Original: For given ν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:
Revised: For given ν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Then, from the following two equations:
Revised: Then, from the following two equations:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: , s ′ )
Revised: , s ′ )

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: α
Revised: α

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s ′
Revised: s ′

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: we can conclude that
Revised: we can conclude that

Original: [Equation]
Revised: [Equation]

Original: From the Lemma F.1, we obtain
Revised: From the Lemma F.1, we obtain

Original: [Equation]
Revised: [Equation]

Original: We show that min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) and min ν ˆ L ( ν )
Revised: We show that min ν ˜ L ( ν ) ≤ min ν ˆ L ( ν ) and min ν ˆ L ( ν )

Original: ≤ min ν ˜ L ( ν ) , so min ν ˆ L ( ν ) =
Revised: ≤ min ν ˜ L ( ν ) , so min ν ˆ L ( ν ) =

Original: min ν ˜ L ( ν ) .
Revised: min ν ˜ L ( ν ) .

Original: Finally, similar to the proof steps for Proposition 3.2, we can easily show that ˆ L FD ( ν ) is convex w.r.t.
Revised: Finally, similar to the proof steps for Proposition 3.2, we can easily show that ˆ L FD ( ν ) is convex w.r.t.

Original: ν (Remark that log-sum-exp is a convex function).
Revised: ν (Remark that log-sum-exp is a convex function).

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: Let (cid:98) V and (cid:101)
Revised: Let (cid:98) V and (cid:101)

Original: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.
Revised: V be the sets arg min (cid:98) ν (cid:98) L ( (cid:98) ν ) and arg min (cid:101) ν (cid:101) L ( (cid:101) ν ) , respectively.

Original: Then, (cid:101) V = { (cid:98) ν
Revised: Then, (cid:101) V = { (cid:98) ν

Original: ∗ + C | (cid:98) ν ∗ ∈ (cid:98) V , C ∈ R } holds.
Revised: ∗ + C | (cid:98) ν ∗ ∈ (cid:98) V , C ∈ R } holds.

Original: Proof.
Revised: Proof.

Original: We will prove this proposition by showing ˜ V ⊆ { ν ∗ + C ′ | ν ∗ ∈
Revised: We will prove this proposition by showing ˜ V ⊆ { ν ∗ + C ′ | ν ∗ ∈

Original: ˆ V , C ∈ R } and ˜ V ⊇ { ν ∗ + C ′ | ν ∗ ∈
Revised: ˆ V , C ∈ R } and ˜ V ⊇ { ν ∗ + C ′ | ν ∗ ∈

Original: ˆ V , C ∈ R } .
Revised: ˆ V , C ∈ R } .

Original: ( ⊆ )
Revised: ( ⊆ )

Original: For given ν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:
Revised: For given ν ∗ ∈ arg min ν ˜ L ( ν ) , we define a constant C as follows:

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Then, by the proof steps of Proposition 3.4, we obtain
Revised: Then, by the proof steps of Proposition 3.4, we obtain

Original: [Equation]
Revised: [Equation]

Original: It means ( ν ∗ + C ) ∈
Revised: It means ( ν ∗ + C ) ∈

Original: arg min
Revised: arg min

Original: ν ˆ L ( ν ) and thus,
Revised: ν ˆ L ( ν ) and thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( ⊇ ) Let ˆ ν ∗ ∈ arg min
Revised: ( ⊇ ) Let ˆ ν ∗ ∈ arg min

Original: ν ˆ L ( ν ) .
Revised: ν ˆ L ( ν ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Let C ∈ R .
Revised: Let C ∈ R .

Original: Then, we derive the following equation:
Revised: Then, we derive the following equation:

Original: [Equation]
Revised: [Equation]

Original: Here, we apply Lemma F.1 to derive the first equality.
Revised: Here, we apply Lemma F.1 to derive the first equality.

Original: Because min
Revised: Because min

Original: [Equation]
Revised: [Equation]

Original: we can conclude that L FD (ˆ ν ∗ + C ) =
Revised: we can conclude that L FD (ˆ ν ∗ + C ) =

Original: min
Revised: min

Original: ν
Revised: ν

Original: L FD ( ν ) , i
Revised: L FD ( ν ) , i

Original: [Equation]
Revised: [Equation]

Original: e., (ˆ ν ∗ + C ) ∈ arg min ν ˜ L ( ν ) .
Revised: e., (ˆ ν ∗ + C ) ∈ arg min ν ˜ L ( ν ) .

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: F.3 Fenchel dual formulation
Revised: F.3 Fenchel dual formulation

Original: Let δ C ( x ) :
Revised: Let δ C ( x ) :

Original: = (cid:26)x ∈ C ∞ otherwise .
Revised: = (cid:26)x ∈ C ∞ otherwise .

Original: Then we can provide following proposition:
Revised: Then we can provide following proposition:

Original: Proposition F.2.
Revised: Proposition F.2.

Original: We can rewrite the optimization problem (6-8) as
Revised: We can rewrite the optimization problem (6-8) as

Original: [Equation]
Revised: [Equation]

Original: Then, the dual problem of (50) is given by
Revised: Then, the dual problem of (50) is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first define following three functions
Revised: We first define following three functions

Original: [Equation]
Revised: [Equation]

Original: and conjugate functions,
Revised: and conjugate functions,

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: the dual formulation of the primal (50) can be derived as follows:
Revised: the dual formulation of the primal (50) can be derived as follows:

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: d )
Revised: d )

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: (cid:21)
Revised: (cid:21)

Original: [Equation]
Revised: [Equation]

Original: αD
Revised: αD

Original: [Equation]
Revised: [Equation]

Original: ( d
Revised: ( d

Original: [Equation]
Revised: [Equation]

Original: d I ) = max d ≥ 0 − f (cid:0)
Revised: d I ) = max d ≥ 0 − f (cid:0)

Original: [Equation]
Revised: [Equation]

Original: ( γ T − B ) ∗ d (cid:1)
Revised: ( γ T − B ) ∗ d (cid:1)

Original: [Equation]
Revised: [Equation]

Original: E ( s
Revised: E ( s

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: s ′
Revised: s ′

Original: [Equation]
Revised: [Equation]

Original: T ( ·
Revised: T ( ·

Original: [Equation]
Revised: [Equation]

Original: a ) (cid:20) − log
Revised: a ) (cid:20) − log

Original: [Equation]
Revised: [Equation]

Original: ( ¯ T
Revised: ( ¯ T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: log d
Revised: log d

Original: [Equation]
Revised: [Equation]

Original: ( ¯ T
Revised: ( ¯ T

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s ′ )
Revised: s ′ )

Original: (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: [Equation]

Original: (cid:21) − αD KL ( d ∥ d I ) = max d ≥ 0 − f
Revised: (cid:21) − αD KL ( d ∥ d I ) = max d ≥ 0 − f

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: r )
Revised: r )

Original: [Equation]
Revised: [Equation]

Original: αD KL ( d ∥ d I ) =
Revised: αD KL ( d ∥ d I ) =

Original: max d
Revised: max d

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: − (cid:26) max
Revised: − (cid:26) max

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: µ ⟩ (cid:27) − αD
Revised: µ ⟩ (cid:27) − αD

Original: [Equation]
Revised: [Equation]

Original: ( d ∥ d
Revised: ( d ∥ d

Original: [Equation]
Revised: [Equation]

Original: ) = max
Revised: ) = max

Original: [Equation]
Revised: [Equation]

Original: ≥ 0 min
Revised: ≥ 0 min

Original: [Equation]
Revised: [Equation]

Original: − f
Revised: − f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: r ) − ⟨ ¯
Revised: r ) − ⟨ ¯

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: − αD
Revised: − αD

Original: [Equation]
Revised: [Equation]

Original: ( d ∥ d I ) =
Revised: ( d ∥ d I ) =

Original: max d ≥ 0 min
Revised: max d ≥ 0 min

Original: [Equation]
Revised: [Equation]

Original: − f
Revised: − f

Original: [Equation]
Revised: [Equation]

Original: ( γ T
Revised: ( γ T

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: d (cid:1)
Revised: d (cid:1)

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: r ) − ⟨ d
Revised: r ) − ⟨ d

Original: [Equation]
Revised: [Equation]

Original: ¯ T µ ⟩ − αD KL ( d ∥ d I ) = max d ≥ 0 min
Revised: ¯ T µ ⟩ − αD KL ( d ∥ d I ) = max d ≥ 0 min

Original: [Equation]
Revised: [Equation]

Original: − f (cid:0)
Revised: − f (cid:0)

Original: [Equation]
Revised: [Equation]

Original: ( γ T − B )
Revised: ( γ T − B )

Original: [Equation]
Revised: [Equation]

Original: d (cid:1) + g ∗ ( µ
Revised: d (cid:1) + g ∗ ( µ

Original: [Equation]
Revised: [Equation]

Original: r ) − h
Revised: r ) − h

Original: ( d
Revised: ( d

Original: [Equation]
Revised: [Equation]

Original: µ )
Revised: µ )

Original: = max d ≥ 0 min µ −
Revised: = max d ≥ 0 min µ −

Original: [Equation]
Revised: [Equation]

Original: max ν ⟨−
Revised: max ν ⟨−

Original: [Equation]
Revised: [Equation]

Original: γ T
Revised: γ T

Original: [Equation]
Revised: [Equation]

Original: B )
Revised: B )

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: ν ⟩ −
Revised: ν ⟩ −

Original: [Equation]
Revised: [Equation]

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we can reorder the maximin to minimax and therefore
Revised: we can reorder the maximin to minimax and therefore

Original: [Equation]
Revised: [Equation]

Original: we can derive the
Revised: we can derive the

Original: [Equation]
Revised: [Equation]

Original: We can rewrite the last term as min
Revised: We can rewrite the last term as min

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Lemma F.3.
Revised: Lemma F.3.

Original: For given function µ : S × S → R and ν : S → R ,
Revised: For given function µ : S × S → R and ν : S → R ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Finally, we can show that the relation between arg min µ,ν L and arg min µ,ν L FD :
Revised: Finally, we can show that the relation between arg min µ,ν L and arg min µ,ν L FD :

Original: Proposition F.4.
Revised: Proposition F.4.

Original: Let V and V FD be the set of optimal solutions ( µ ∗ , ν ∗ ) of arg min µ,ν L ( w ν,µ , ¯ w ν,µ , µ, ν ) and arg min µ,ν L FD ( µ, ν ) , respectively.
Revised: Let V and V FD be the set of optimal solutions ( µ ∗ , ν ∗ ) of arg min µ,ν L ( w ν,µ , ¯ w ν,µ , µ, ν ) and arg min µ,ν L FD ( µ, ν ) , respectively.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Proof.
Revised: Proof.

Original: For brevity, we will denote L ( w ν,µ , ¯ w ν,µ , µ, ν ) as L ( µ, ν ) .
Revised: For brevity, we will denote L ( w ν,µ , ¯ w ν,µ , µ, ν ) as L ( µ, ν ) .

Original: ( ⊆ )
Revised: ( ⊆ )

Original: For given (ˆ µ ∗ , ˆ ν ∗ ) ∈ arg min µ,ν L FD ( µ, ν ) , we define two constants as follows:
Revised: For given (ˆ µ ∗ , ˆ ν ∗ ) ∈ arg min µ,ν L FD ( µ, ν ) , we define two constants as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, from the following two equations:
Revised: Then, from the following two equations:

Original: [Equation]
Revised: [Equation]

Original: [ˆ ν ∗ ( s ) + C ′ ]
Revised: [ˆ ν ∗ ( s ) + C ′ ]

Original: + E ( s,a,s ′ )
Revised: + E ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + C
Revised: + C

Original: + r ( s, s ′ ) − 1)]
Revised: + r ( s, s ′ ) − 1)]

Original: + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E s ′
Revised: + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: + γC
Revised: + γC

Original: ′ − ˆ µ ∗ ( s, s ′ )
Revised: ′ − ˆ µ ∗ ( s, s ′ )

Original: − C ] − ˆ ν ∗ ( s ′ )
Revised: − C ] − ˆ ν ∗ ( s ′ )

Original: − C ′ (cid:1) − 1 (cid:17)(cid:105) =
Revised: − C ′ (cid:1) − 1 (cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s )]
Revised: [ˆ ν ∗ ( s )]

Original: + (1 − γ ) C ′
Revised: + (1 − γ ) C ′

Original: + E ( s,a,s ′ )
Revised: + E ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + r ( s, s ′ ) − 1)] exp( C ) + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E
Revised: + r ( s, s ′ ) − 1)] exp( C ) + α E ( s,a ) ∼ d I (cid:104) exp (cid:16) 1 α (cid:0) E

Original: s ′
Revised: s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: − ˆ µ ∗ ( s, s ′ )]
Revised: − ˆ µ ∗ ( s, s ′ )]

Original: − ˆ ν ∗ ( s ′ )
Revised: − ˆ ν ∗ ( s ′ )

Original: (cid:1) − 1 (cid:17)(cid:105) exp (cid:16) 1 α (cid:0) − (1 − γ ) C ′ − C
Revised: (cid:1) − 1 (cid:17)(cid:105) exp (cid:16) 1 α (cid:0) − (1 − γ ) C ′ − C

Original: (cid:1)(cid:17) =
Revised: (cid:1)(cid:17) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s )]
Revised: [ˆ ν ∗ ( s )]

Original: + (1 − γ ) C ′
Revised: + (1 − γ ) C ′

Original: +
Revised: +

Original: + α, and
Revised: + α, and

Original: L FD
Revised: L FD

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ))] +
Revised: ))] +

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) −
Revised: ) −

Original: [Equation]
Revised: [Equation]

Original: ] −
Revised: ] −

Original: [Equation]
Revised: [Equation]

Original: ν
Revised: ν

Original: [Equation]
Revised: [Equation]

Original: ( s ′ )
Revised: ( s ′ )

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ′ (cid:1)(cid:17)(cid:105) =
Revised: ′ (cid:1)(cid:17)(cid:105) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s )]
Revised: [ˆ ν ∗ ( s )]

Original: + (1 − γ ) C ′ + log E
Revised: + (1 − γ ) C ′ + log E

Original: ( s,a,s ′ )
Revised: ( s,a,s ′ )

Original: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )
Revised: ∼ d I [exp(ˆ µ ∗ ( s, s ′ )

Original: + r ( s, s ′ ))]
Revised: + r ( s, s ′ ))]

Original: + C
Revised: + C

Original: + α log E ( s,a ) ∼ d I (cid:104) exp (cid:16) 

Revised: + α log E ( s,a ) ∼ d I (cid:104) exp (cid:16)

Original: α (cid:0) E s ′
Revised: α (cid:0) E s ′

Original: [ γ ˆ ν ∗ ( s ′ )
Revised: [ γ ˆ ν ∗ ( s ′ )

Original: − ˆ µ ∗ ( s, s ′ )]
Revised: − ˆ µ ∗ ( s, s ′ )]

Original: − ˆ ν ∗ ( s ′ ) (cid:1)(cid:17)(cid:105) + ( γC ′ − C − C ′ ) =
Revised: − ˆ ν ∗ ( s ′ ) (cid:1)(cid:17)(cid:105) + ( γC ′ − C − C ′ ) =

Original: (1 − γ ) E s ∼ p 

Revised: (1 − γ ) E s ∼ p

Original: [ˆ ν ∗ ( s ) + C ′ ]
Revised: [ˆ ν ∗ ( s ) + C ′ ]

Original: + 1 + α, we can conclude that
Revised: + 1 + α, we can conclude that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: It means (ˆ µ
Revised: It means (ˆ µ

Original: [Equation]
Revised: [Equation]

Original: + C, ˆ ν
Revised: + C, ˆ ν

Original: [Equation]
Revised: [Equation]

Original: + C ′ ) ∈ arg min µ,ν L ( µ, ν ) and thus,
Revised: + C ′ ) ∈ arg min µ,ν L ( µ, ν ) and thus,

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: ( ⊇ ) Let ( µ ∗ , ν ∗ ) ∈ arg min µ,ν L ( µ, ν ) .
Revised: ( ⊇ ) Let ( µ ∗ , ν ∗ ) ∈ arg min µ,ν L ( µ, ν ) .

Original: Then, d ∗ ( s, a )
Revised: Then, d ∗ ( s, a )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let C ∈ R and C ′ ∈ R .
Revised: Let C ∈ R and C ′ ∈ R .

Original: Then, we derive the following equation:
Revised: Then, we derive the following equation:

Original: [Equation]
Revised: [Equation]

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we apply Lemma F.1 to derive the first equality.
Revised: we apply Lemma F.1 to derive the first equality.

Original: Because min
Revised: Because min

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: ,ν ∗ ( s, a )]
Revised: ,ν ∗ ( s, a )]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∼
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: 0 [ ν
Revised: 0 [ ν

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∗
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: ] +
Revised: ] +

Original: [Equation]
Revised: [Equation]

Original: ,a
Revised: ,a

Original: [Equation]
Revised: [Equation]

Original: we can conclude that L FD ( µ ∗ + C, ν ∗ + C ′ ) =
Revised: we can conclude that L FD ( µ ∗ + C, ν ∗ + C ′ ) =

Original: min
Revised: min

Original: µ,ν L FD ( µ, ν ) , i.e., ( µ ∗ + C, ν ∗ + C ′ ) ∈ arg min
Revised: µ,ν L FD ( µ, ν ) , i.e., ( µ ∗ + C, ν ∗ + C ′ ) ∈ arg min

Original: µ,ν L FD ( µ, ν ) .
Revised: µ,ν L FD ( µ, ν ) .

Original: It means,
Revised: It means,

Original: [Equation]
Revised: [Equation]

Original: Remark that the proposed objective (51) is stable and the aforementioned proposition allows us to use self-normalized weighted importance sampling to extract policy.
Revised: Remark that the proposed objective (51) is stable and the aforementioned proposition allows us to use self-normalized weighted importance sampling to extract policy.

Original: However, as we discussed in
Revised: However, as we discussed in

Original: Section 3.5, using µ is main bottleneck for the overall optimization and optimizing ˜ L ( ν ) shows better performance in practice.
Revised: Section 3.5, using µ is main bottleneck for the overall optimization and optimizing ˜ L ( ν ) shows better performance in practice.

Original: G Pseudocode of LobsDICE G.1 Tabular LobsDICE
Revised: G Pseudocode of LobsDICE G.1 Tabular LobsDICE

Original: For tabular MDPs, we first construct a maximum-likelihood estimation (MLE)
Revised: For tabular MDPs, we first construct a maximum-likelihood estimation (MLE)

Original: MDP
Revised: MDP

Original: ˆ M = ⟨ S, A, ˆ T, ˆ p 0 , γ ⟩ using an offline dataset D I .
Revised: ˆ M = ⟨ S, A, ˆ T, ˆ p 0 , γ ⟩ using an offline dataset D I .

Original: Then, tabular LobsDICE solves (16) on the MLE
Revised: Then, tabular LobsDICE solves (16) on the MLE

Original: MDP ˆ M .
Revised: MDP ˆ M .

Original: In tabular case, note that (cid:98) d
Revised: In tabular case, note that (cid:98) d

Original: I ( s, a ) , (cid:98) d
Revised: I ( s, a ) , (cid:98) d

Original: I ( s, s ′ ) , and (cid:98) d E ( s, s ′ ) are explicitly accessible by normalized counts of ( s, a ) and ( s, s ′ ) in the datasets.
Revised: I ( s, s ′ ) , and (cid:98) d E ( s, s ′ ) are explicitly accessible by normalized counts of ( s, a ) and ( s, s ′ ) in the datasets.

Original: In addition, (cid:98) r ( s, s ′ )
Revised: In addition, (cid:98) r ( s, s ′ )

Original: = (cid:98)
Revised: = (cid:98)

Original: d E ( s,s ′ ) (cid:98) d
Revised: d E ( s,s ′ ) (cid:98) d

Original: I ( s,s ′ ) is the log ratio between two empirical distributions.
Revised: I ( s,s ′ ) is the log ratio between two empirical distributions.

Original: Also, ν ∈ R | S | is represented as a | S | -dimensional vector, and µ ∈ R | S |×| S | is represented as a | S | 2 -dimensional vector, and we solve the following convex minimization problem for ( µ, ν ) :
Revised: Also, ν ∈ R | S | is represented as a | S | -dimensional vector, and µ ∈ R | S |×| S | is represented as a | S | 2 -dimensional vector, and we solve the following convex minimization problem for ( µ, ν ) :

Original: [Equation]
Revised: [Equation]

Original: This process is summarized in Algorithm 1.
Revised: This process is summarized in Algorithm 1.

Original: Input: state-only demonstrations by experts D E
Revised: Input: state-only demonstrations by experts D E

Original: = { ( s, s ′ )
Revised: = { ( s, s ′ )

Original: i } N E i =1 , state-action demonstrations by some imperfect agents D I =
Revised: i } N E i =1 , state-action demonstrations by some imperfect agents D I =

Original: ( s a, s ′ )
Revised: ( s a, s ′ )

Original: i } N I
Revised: i } N I

Original: i =1 , a learning rate η .
Revised: i =1 , a learning rate η .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∗
Revised: ∗

Original: (cid:16) 1 α (cid:0) (cid:80) s ′
Revised: (cid:16) 1 α (cid:0) (cid:80) s ′

Original: ˆ T ( s ′
Revised: ˆ T ( s ′

Original: s
Revised: s

Original: −
Revised: −

Original: s
Revised: s

Original: s ′
Revised: s ′

Original: (cid:98) d
Revised: (cid:98) d

Original: I ( s,a ) w ∗ ( s,a ) (cid:80) a ′ (cid:98) d
Revised: I ( s,a ) w ∗ ( s,a ) (cid:80) a ′ (cid:98) d

Original: I ( s,a ′ )
Revised: I ( s,a ′ )

Original: w ∗ ( s,a ′ )
Revised: w ∗ ( s,a ′ )

Original: s
Revised: s

Original: G.2 Practical LobsDICE (with function approximation)
Revised: G.2 Practical LobsDICE (with function approximation)

Original: To deal with continuous or large MDPs, we represent our optimization variable ν and discriminator c as neural networks, parameterized by θ and ϕ respectively:
Revised: To deal with continuous or large MDPs, we represent our optimization variable ν and discriminator c as neural networks, parameterized by θ and ϕ respectively:

Original: ν θ : S → R is an MLP that takes a state as an input and outputs a scalar value, and c ϕ : S × S → [0 , 1] is defined similarly.
Revised: ν θ : S → R is an MLP that takes a state as an input and outputs a scalar value, and c ϕ : S × S → [0 , 1] is defined similarly.

Original: For the policy π ψ , we use a tanh-squashed Gaussian policy, where the parameters of a Gaussian distribution (i.e. mean and covariance) are output by the neural network.
Revised: For the policy π ψ , we use a tanh-squashed Gaussian policy, where the parameters of a Gaussian distribution (i.e. mean and covariance) are output by the neural network.

Original: The parameters of c ϕ are trained by:
Revised: The parameters of c ϕ are trained by:

Original: [Equation]
Revised: [Equation]

Original: which is analogous to (13).
Revised: which is analogous to (13).

Original: The parameters of ν θ are trained by:
Revised: The parameters of ν θ are trained by:

Original: [Equation]
Revised: [Equation]

Original: where ˆ A θ ( s, a, s ′ ) =
Revised: where ˆ A θ ( s, a, s ′ ) =

Original: r ϕ ( s, s ′ )
Revised: r ϕ ( s, s ′ )

Original: + γν θ ( s ′ )
Revised: + γν θ ( s ′ )

Original: − ν θ ( s ) and r ϕ ( s, s ′ )
Revised: − ν θ ( s ) and r ϕ ( s, s ′ )

Original: = − log (cid:16) 1 c ϕ ( s,s ′ ) − 1 (cid:17) , and this is analogous to (23).
Revised: = − log (cid:16) 1 c ϕ ( s,s ′ ) − 1 (cid:17) , and this is analogous to (23).

Original: Due to the logarithm outside the expectation in the second term, the mini-batch approximation would introduce additional bias in gradient estimation, but we found that using the biased gradient estimate worked well in practice with moderately large batch size (e.g. 256).
Revised: Due to the logarithm outside the expectation in the second term, the mini-batch approximation would introduce additional bias in gradient estimation, but we found that using the biased gradient estimate worked well in practice with moderately large batch size (e.g. 256).

Original: Finally, the parameters of π ψ are trained by:
Revised: Finally, the parameters of π ψ are trained by:

Original: [Equation]
Revised: [Equation]

Original: which is analogous to (24).
Revised: which is analogous to (24).

Original: Here, the self-normalized importance sampling may introduce an additional bias with mini-batch approximations, but it is well known that the self-normalized importance sampling provides a consistent estimator [31].
Revised: Here, the self-normalized importance sampling may introduce an additional bias with mini-batch approximations, but it is well known that the self-normalized importance sampling provides a consistent estimator [31].

Original: Also, it worked well in practice with moderately large batch size (e.g. 256) in our experiments.
Revised: Also, it worked well in practice with moderately large batch size (e.g. 256) in our experiments.

Original: We optimize the parameters ( ϕ, θ, ψ ) jointly in practice.
Revised: We optimize the parameters ( ϕ, θ, ψ ) jointly in practice.

Original: We are not using a target network at all.
Revised: We are not using a target network at all.

Original: The overall training process is summarized in Algorithm 2.
Revised: The overall training process is summarized in Algorithm 2.

Original: Compute gradients and
Revised: Compute gradients and

Original: [Equation]
Revised: [Equation]

Original: ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: ϕ
Revised: ϕ

Original: [Equation]
Revised: [Equation]

Original: (Eq.
Revised: (Eq.

Original: [Equation]
Revised: [Equation]

Original: ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: ( θ ) (Eq.
Revised: ( θ ) (Eq.

Original: 7: ψ ←
Revised: 7: ψ ←

Original: [Equation]
Revised: [Equation]

Original: η ∇
Revised: η ∇

Original: [Equation]
Revised: [Equation]

Original: ( ψ ) (Eq.
Revised: ( ψ ) (Eq.

Original: H Additional Experiments H.1 Subsampled expert demonstrations
Revised: H Additional Experiments H.1 Subsampled expert demonstrations

Original: Existing works on imitation learning [11] conduct experiments where the expert demonstrations are subsampled.
Revised: Existing works on imitation learning [11] conduct experiments where the expert demonstrations are subsampled.

Original: The following figure presents the result when given the subsampled expert demonstrations.
Revised: The following figure presents the result when given the subsampled expert demonstrations.

Original: LobsDICE still significantly outperforms the baseline algorithms even with the subsampled expert demonstrations † .
Revised: LobsDICE still significantly outperforms the baseline algorithms even with the subsampled expert demonstrations 3 .

Original: [Figure]
Revised: [Figure]

Original: H.2 Error of inverse dynamics model in MuJoCo domains
Revised: H.2 Error of inverse dynamics model in MuJoCo domains

Original: In principle
Revised: In principle

Original: [Figure]
Revised: [Figure]

Original: IDM trained with arbitrary demonstrations should be able to predict the expert’s missing actions accurately in deterministic MDPs.
Revised: IDM trained with arbitrary demonstrations should be able to predict the expert’s missing actions accurately in deterministic MDPs.

Original: However, in Section 5.
Revised: However, in Section 5.

Original: [Figure]
Revised: [Figure]

Original: we observed that the empirical performance of BCO and DemoDICEfO tends to degrade as the number of imperfect demonstrations increases (see Figure 2).
Revised: we observed that the empirical performance of BCO and DemoDICEfO tends to degrade as the number of imperfect demonstrations increases (see Figure 2).

Original: This is due to the use of function approximation for IDM: given that the expressive power of a function approximator is limited, the more data unrelated to the expert demonstrations (i
Revised: This is due to the use of function approximation for IDM: given that the expressive power of a function approximator is limited, the more data unrelated to the expert demonstrations (i

Original: [Figure]
Revised: [Figure]

Original: e. medium-v2 and random-v2 ) is used for training, the more the prediction accuracy for the expert data would be adversely affected.
Revised: e. medium-v2 and random-v2 ) is used for training, the more the prediction accuracy for the expert data would be adversely affected.

Original: Figure 6 shows that the mean squared error of IDM for expert demonstrations D E increases as the number of non-expert demonstrations increases
Revised: Figure 6 shows that the mean squared error of IDM for expert demonstrations D E increases as the number of non-expert demonstrations increases

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H.3 Fewer expert demonstrations
Revised: H.3 Fewer expert demonstrations

Original: In this section, we provide additional experiments with fewer expert demonstrations.
Revised: In this section, we provide additional experiments with fewer expert demonstrations.

Original: Specifically, we tested LobsDICE when the number of state-only expert demonstrations is 5, 3, and 1, respectively.
Revised: Specifically, we tested LobsDICE when the number of state-only expert demonstrations is 5, 3, and 1, respectively.

Original: The Figure 7 shows that LobsDICE (#exp=3) and LobsDICE (#exp=1) still perform well.
Revised: The Figure 7 shows that LobsDICE (#exp=3) and LobsDICE (#exp=1) still perform well.

Original: [Figure]
Revised: [Figure]

Original: z
Revised: z

Original: [Figure]
Revised: [Figure]

Original: I Experimental Details I.1 Experimental details for random MDPs
Revised: I Experimental Details I.1 Experimental details for random MDPs

Original: Random MDP generation For MDP M = ⟨ S, A, T, R, γ, p 0 ⟩ , we first set | S | = 20 , | A | = 4 , γ = 0 .
Revised: Random MDP generation For MDP M = ⟨ S, A, T, R, γ, p 0 ⟩ , we first set | S | = 20 , | A | = 4 , γ = 0 .

Original: 95 , p 0 ( s ) =
Revised: 95 , p 0 ( s ) =

Original: 1 for a fixed s
Revised: 1 for a fixed s

Original: = s 0 .
Revised: = s 0 .

Original: For each ( s, a ) , we samples four different state { s ′ 1 , s ′ 2 , s ′ 3 , s ′ 4 } .
Revised: For each ( s, a ) , we samples four different state { s ′ 1 , s ′ 2 , s ′ 3 , s ′ 4 } .

Original: Then, we set the transition probability ( T ( s ′ 1 | s, a ) , T ( s ′ 2 | s, a ) , T ( s ′ 3 | s, a ) , T ( s ′ 4 | s, a )) =
Revised: Then, we set the transition probability ( T ( s ′ 1 | s, a ) , T ( s ′ 2 | s, a ) , T ( s ′ 3 | s, a ) , T ( s ′ 4 | s, a )) =

Original: (1 − β ) X
Revised: (1 − β ) X

Original: + βY , where X ∼ Categorical (0 . 25 , 0 .
Revised: + βY , where X ∼ Categorical (0 . 25 , 0 .

Original: 25 , 0 . 25) and Y ∼ Dir (1 , 1 , 1 , 1) .
Revised: 25 , 0 . 25) and Y ∼ Dir (1 , 1 , 1 , 1) .

Original: β ∈ [0 , 1] is the hyperparameter to control transition stochasticity:
Revised: β ∈ [0 , 1] is the hyperparameter to control transition stochasticity:

Original: when β = 0 , the transition probability becomes one-hot vector, i.e., deterministic MDP.
Revised: when β = 0 , the transition probability becomes one-hot vector, i.e., deterministic MDP.

Original: In contrast, when β = 1 , the transition of MDP becomes stochastic.
Revised: In contrast, when β = 1 , the transition of MDP becomes stochastic.

Original: Finally, the reward of 1 is only given to a state that minimizes the optimal value at initial state s 0 ; other states have zero rewards.
Revised: Finally, the reward of 1 is only given to a state that minimizes the optimal value at initial state s 0 ; other states have zero rewards.

Original: Offline dataset generation For each random MDP M , we generate state-only expert demonstrations by executing a (softmax) expert policy and state-action imperfect demonstrations by a uniform random policy.
Revised: Offline dataset generation For each random MDP M , we generate state-only expert demonstrations by executing a (softmax) expert policy and state-action imperfect demonstrations by a uniform random policy.

Original: We perform experiments for a varying number of expert demonstrations N E ∈ { 10 , 100 , 1000 , 1000 } and imperfect demonstrations N I ∈ { 1 , 3 , 10 , 30 , 100 , 300 , 1000 , 3000 , 10000 } .
Revised: We perform experiments for a varying number of expert demonstrations N E ∈ { 10 , 100 , 1000 , 1000 } and imperfect demonstrations N I ∈ { 1 , 3 , 10 , 30 , 100 , 300 , 1000 , 3000 , 10000 } .

Original: Hyperparameters We compare our tabular LobsDICE with BC, BCO, DemoDICEfO, and OPOLO.
Revised: Hyperparameters We compare our tabular LobsDICE with BC, BCO, DemoDICEfO, and OPOLO.

Original: For KL-regularization hyperparameters of DemoDICEfO and LobsDICE, we use α = 0 . 1 .
Revised: For KL-regularization hyperparameters of DemoDICEfO and LobsDICE, we use α = 0 . 1 .

Original: I.2 Experimental details for MuJoCo control tasks
Revised: I.2 Experimental details for MuJoCo control tasks

Original: Hyperparameters
Revised: Hyperparameters

Original: BC
Revised: BC

Original: BCO
Revised: BCO

Original: DemoDICEfO
Revised: DemoDICEfO

Original: LobsDICE γ (discount factor) 0.
Revised: LobsDICE γ (discount factor) 0.

Original: [Table]
Revised: [Table]

Original: For fair comparison, we use the same learning rate to train actors of BC, BCO, DemoDICEfO
Revised: For fair comparison, we use the same learning rate to train actors of BC, BCO, DemoDICEfO

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: LobsDICE.
Revised: LobsDICE.

Original: We implement our network architectures for BC, BCO, DemoDICEfO
Revised: We implement our network architectures for BC, BCO, DemoDICEfO

Original: [Table]
Revised: [Table]

Original: and LobsDICE based on the implementation of OptiDICE † .
Revised: and LobsDICE based on the implementation of OptiDICE 4 .

Original: For OPOLO, we use its official implmentation † .
Revised: For OPOLO, we use its official implmentation 5 .

Original: We have tried hyperparameter tuning for OPOLO but never obtained a successful learning curve, showing numerical instability due to using out-of-distribution action values.
Revised: We have tried hyperparameter tuning for OPOLO but never obtained a successful learning curve, showing numerical instability due to using out-of-distribution action values.

Original: Vanilla online off-policy algorithms commonly fail in the offline learning setting.
Revised: Vanilla online off-policy algorithms commonly fail in the offline learning setting.

Original: Therefore, we report the results of official implementation without any modification to network architectures or hyperparameters.
Revised: Therefore, we report the results of official implementation without any modification to network architectures or hyperparameters.

Original: For stable discriminator learning, we use gradient penalty regularization on the r ( s, a ) and r ( s, s ′ ) functions, which was proposed in [10] to enforce 1-Lipschitz constraint.
Revised: For stable discriminator learning, we use gradient penalty regularization on the r ( s, a ) and r ( s, s ′ ) functions, which was proposed in [10] to enforce 1-Lipschitz constraint.

Original: To stabilize critic training, we add gradient L2-norm to the critic loss for the regularization.
Revised: To stabilize critic training, we add gradient L2-norm to the critic loss for the regularization.

Original: In addition, we use every state s in D I to define D 0 , i.e., D 

Revised: In addition, we use every state s in D I to define D 0 , i.e., D

Original: =
Revised: =

Original: { s | ( s, a, s ′ )
Revised: { s | ( s, a, s ′ )

Original: ∈
Revised: ∈

Original: D I } , following ValueDICE [17].
Revised: D I } , following ValueDICE [17].

Original: Detailed hyperparameter configurations used for our main experiments are summarized in Table 1.
Revised: Detailed hyperparameter configurations used for our main experiments are summarized in Table 1.

Original: Evaluation metric For each environment, the normalized score is measured by 100 × score − random score expert score − random score , where the expert score and random score are average returns of trajectories in expert-v2 and random-v2 , respectively.
Revised: Evaluation metric For each environment, the normalized score is measured by 100 × score − random score expert score − random score , where the expert score and random score are average returns of trajectories in expert-v2 and random-v2 , respectively.

Original: J Generalization to γ =
Revised: J Generalization to γ =

Original: In this section, we generalize LobsDICE for discounted MDPs ( γ < 1 ) to undiscounted MDPs ( γ = 1 ).
Revised: In this section, we generalize LobsDICE for discounted MDPs ( γ < 1 ) to undiscounted MDPs ( γ = 1 ).

Original: Suppose that the stationary distributions d and ¯ d satisfy the Bellman flow constraints (7) and marginalization constraints (8), respectively.
Revised: Suppose that the stationary distributions d and ¯ d satisfy the Bellman flow constraints (7) and marginalization constraints (8), respectively.

Original: When γ = 1 , for any constant c ≥ 0 , cd and c ¯ d also satisfy the corresponding constraints.
Revised: When γ = 1 , for any constant c ≥ 0 , cd and c ¯ d also satisfy the corresponding constraints.

Original: It means that the original constrained optimization problem (6-8) for the stationary distributions d and ¯ d is an ill-posed problem.
Revised: It means that the original constrained optimization problem (6-8) for the stationary distributions d and ¯ d is an ill-posed problem.

Original: We tackle this ill-posedness by adding additional normalization constraint (cid:80) s,a d ( s, a ) = 1 to (6-8):
Revised: We tackle this ill-posedness by adding additional normalization constraint (cid:80) s,a d ( s, a ) = 1 to (6-8):

Original: [Equation]
Revised: [Equation]

Original: Using a derivation similar to that from (9) to (11), we obtain the following min-max optimization for w, ¯ w, µ, ν , and λ :
Revised: Using a derivation similar to that from (9) to (11), we obtain the following min-max optimization for w, ¯ w, µ, ν , and λ :

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R is the Lagrange multiplier for the normalization constraint (cid:80) s,a d ( s, a ) = 1 .
Revised: where λ ∈ R is the Lagrange multiplier for the normalization constraint (cid:80) s,a d ( s, a ) = 1 .

Original: Similar to
Revised: Similar to

Original: Proposition 3.1, we can derive a closed-form solution to the inner maximization of (56): w µ,ν,λ ( s, a ) = exp
Revised: Proposition 3.1, we can derive a closed-form solution to the inner maximization of (56): w µ,ν,λ ( s, a ) = exp

Original: [Equation]
Revised: [Equation]

Original: 1 α e µ,ν,λ
Revised: 1 α e µ,ν,λ

Original: [Equation]
Revised: [Equation]

Original: s, a
Revised: s, a

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: and ¯ w µ ( s, s ′ )
Revised: and ¯ w µ ( s, s ′ )

Original: =
Revised: =

Original: exp( r ( s, s ′ )
Revised: exp( r ( s, s ′ )

Original: + µ ( s, s ′ )
Revised: + µ ( s, s ′ )

Original: − 1) .
Revised: − 1) .

Original: Then, we can reduce the nested min-max optimization of (56) to a single minimization by plugging the closed-form solution ( w µ,ν,λ , ¯ w µ ) into L ( w, ¯ w, µ, ν, λ ) :
Revised: Then, we can reduce the nested min-max optimization of (56) to a single minimization by plugging the closed-form solution ( w µ,ν,λ , ¯ w µ ) into L ( w, ¯ w, µ, ν, λ ) :

Original: [Equation]
Revised: [Equation]

Original: In practice, we estimate L ( w µ,ν,λ , ¯ w µ , µ, ν, λ )) using samples from distribution d I .
Revised: In practice, we estimate L ( w µ,ν,λ , ¯ w µ , µ, ν, λ )) using samples from distribution d I .

Original: To derive a sample-based objective, we use analogous derivation in Section 3.5.
Revised: To derive a sample-based objective, we use analogous derivation in Section 3.5.

Original: Sample-based objective can be represented as
Revised: Sample-based objective can be represented as

Original: [Equation]
Revised: [Equation]

Original: where ˆ e µ,ν,λ ( s, a, s ′ )
Revised: where ˆ e µ,ν,λ ( s, a, s ′ )

Original: = − µ ( s, s ′ )
Revised: = − µ ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) − λ .
Revised: − ν ( s ) − λ .

Original: Then, the closed-form solution to the minimization (57) with respect to µ is:
Revised: Then, the closed-form solution to the minimization (57) with respect to µ is:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the above solution, we obtain the following minimization problem:
Revised: Using the above solution, we obtain the following minimization problem:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:98) A ν,
Revised: where (cid:98) A ν,

Original: [Equation]
Revised: [Equation]

Original: ( s, a, s
Revised: ( s, a, s

Original: [Equation]
Revised: [Equation]

Original: = r ( s, s ′ )
Revised: = r ( s, s ′ )

Original: + γν ( s ′ )
Revised: + γν ( s ′ )

Original: − ν ( s ) − λ .
Revised: − ν ( s ) − λ .

Original: Finally, similar to Proposition 3.4, we can obtain the following objective with the same minimum as (59):
Revised: Finally, similar to Proposition 3.4, we can obtain the following objective with the same minimum as (59):

Original: [Equation]
Revised: [Equation]

Original: Interestingly, the resulting objective is the same as the original objective (cid:101) L ( (cid:101) ν ) of (23).
Revised: Interestingly, the resulting objective is the same as the original objective (cid:101) L ( (cid:101) ν ) of (23).

Original: Based on this theoretical result, we compare LobsDICE for γ = 1 , denoted by LobsDICE ( γ = 1 ), with LobsDICE for γ = 0 .
Revised: Based on this theoretical result, we compare LobsDICE for γ = 1 , denoted by LobsDICE ( γ = 1 ), with LobsDICE for γ = 0 .

Original: 99 , denoted by LobsDICE ( γ = 0 . 99 ).
Revised: 99 , denoted by LobsDICE ( γ = 0 . 99 ).

Original: In Figure 8, LobsDICE ( γ = 1 ) shows good performance, but LobsDICE ( γ = 0 . 99 ) is more stable and achieves better performance in practice
Revised: In Figure 8, LobsDICE ( γ = 1 ) shows good performance, but LobsDICE ( γ = 0 . 99 ) is more stable and achieves better performance in practice

Original: [Figure]
Revised: [Figure]

Original: K Computation Resources
Revised: K Computation Resources

Original: We used 10 servers equipped with the following specification:
Revised: We used 10 servers equipped with the following specification:

Original: • CPU: Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz.
Revised: • CPU: Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz.

Original: • Memory: 32 GB.
Revised: • Memory: 32 GB.

Original: • GPU: TITAN V.
Revised: • GPU: TITAN V.

Original: L Limitation
Revised: L Limitation

Original: Although we demonstrated that LobsDICE successfully recovered the expert’s behavior in the experiments
Revised: Although we demonstrated that LobsDICE successfully recovered the expert’s behavior in the experiments

Original: [Figure]
Revised: [Figure]

Original: it requires the assumption that support of D I covers D E .
Revised: it requires the assumption that support of D I covers D E .

Original: Relaxing this assumption remains as a future work
Revised: Relaxing this assumption remains as a future work

Original: [Figure]
Revised: [Figure]

Original: M License
Revised: M License

Original: D4RL [6] is licensed under the Apache 2.
Revised: D4RL [6] is licensed under the Apache 2.

Original: N Social Impact
Revised: N Social Impact

Original: This work contributes to an algorithmic foundation for imitation learning from observation in an offline setting.
Revised: This work contributes to an algorithmic foundation for imitation learning from observation in an offline setting.

Original: Given that state-only expert demonstrations are much easier to be collected than the action-labeled expert demonstrations and that the imperfect demonstrations of arbitrary optimality is also easy to be collected, our method has a potential to be widely adopted in many real-world applications.
Revised: Given that state-only expert demonstrations are much easier to be collected than the action-labeled expert demonstrations and that the imperfect demonstrations of arbitrary optimality is also easy to be collected, our method has a potential to be widely adopted in many real-world applications.

Original: On the other hand, this work could adversely affect employment by contributing to the automation of tasks having done by human experts (e.g. factory automation, autonomous driving).
Revised: On the other hand, this work could adversely affect employment by contributing to the automation of tasks having done by human experts (e.g. factory automation, autonomous driving).