Original: R
Revised: R

Original: EPARAMETERIZED
Revised: EPARAMETERIZED

Original: V ARIATIONAL D IVERGENCE M INIMIZATION FOR S
Revised: V ARIATIONAL D IVERGENCE M INIMIZATION FOR S

Original: [Table]
Revised: [Table]

Original: authors
Revised: authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior.
Revised: State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior.

Original: Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes.
Revised: Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes.

Original: Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in signiﬁcant differences in model performance.
Revised: Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in signiﬁcant differences in model performance.

Original: Choices including Wasserstein distance and various f -divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning (Ke et al., 2019).
Revised: Choices including Wasserstein distance and various f -divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning (Ke et al., 2019).

Original: Unfortunately, we ﬁnd that in practice this existing imitation-learning framework for using f -divergences suffers from numerical instabilities stemming from the combination of function approximation and policygradient reinforcement learning.
Revised: Unfortunately, we ﬁnd that in practice this existing imitation-learning framework for using f -divergences suffers from numerical instabilities stemming from the combination of function approximation and policygradient reinforcement learning.

Original: In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as f -divergence minimization before further extending the framework to handle the problem of imitation from observations only .
Revised: In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as f -divergence minimization before further extending the framework to handle the problem of imitation from observations only .

Original: Empirically, we demonstrate that our design choices for coupling imitation learning and f -divergences are critical to recovering successful imitation policies.
Revised: Empirically, we demonstrate that our design choices for coupling imitation learning and f -divergences are critical to recovering successful imitation policies.

Original: Moreover, we ﬁnd that with the appropriate choice of fdivergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continouscontrol tasks with low-dimensional observation spaces.
Revised: Moreover, we ﬁnd that with the appropriate choice of fdivergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continouscontrol tasks with low-dimensional observation spaces.

Original: With high-dimensional observations, we still observe a signiﬁcant gap with and without action labels, offering an interesting avenue for future work.
Revised: With high-dimensional observations, we still observe a signiﬁcant gap with and without action labels, offering an interesting avenue for future work.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imitation Learning (IL)
Revised: Imitation Learning (IL)

Original: (Osa et al., 2018) refers to a paradigm of reinforcement learning in which the learning agent has access to an optimal, reward-maximizing expert for the underlying environment.
Revised: (Osa et al., 2018) refers to a paradigm of reinforcement learning in which the learning agent has access to an optimal, reward-maximizing expert for the underlying environment.

Original: In most work, this access is provided through a dataset of trajectories where each observed state is annotated with the action prescribed by the expert policy.
Revised: In most work, this access is provided through a dataset of trajectories where each observed state is annotated with the action prescribed by the expert policy.

Original: This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning, since not all tasks of interest admit easily-speciﬁed reward functions.
Revised: This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning, since not all tasks of interest admit easily-speciﬁed reward functions.

Original: Additionally, not all environments are amenable to the prolonged and potentially unsafe exploration needed for reward-maximizing agents to arrive at satisfactory policies (Achiam et al., 2017; Chow et al., 2019).
Revised: Additionally, not all environments are amenable to the prolonged and potentially unsafe exploration needed for reward-maximizing agents to arrive at satisfactory policies (Achiam et al., 2017; Chow et al., 2019).

Original: While the traditional formulation of the IL problem assumes access to optimal expert action labels, the provision of such information can often be laborious (in the case of a real, human expert) or incur signiﬁcant ﬁnancial cost (such as using elaborate instrumentation to record expert actions).
Revised: While the traditional formulation of the IL problem assumes access to optimal expert action labels, the provision of such information can often be laborious (in the case of a real, human expert) or incur signiﬁcant ﬁnancial cost (such as using elaborate instrumentation to record expert actions).

Original: Additionally, this restrictive assumption removes a vast number of rich, observation-only data sources from consideration (Zhou et al., 2018).
Revised: Additionally, this restrictive assumption removes a vast number of rich, observation-only data sources from consideration (Zhou et al., 2018).

Original: To bypass these challenges, recent work (Liu et al., 2018;
Revised: To bypass these challenges, recent work (Liu et al., 2018;

Original: Torabi et al., 2018a;b; Edwards et al., 2019; Sun et al., 2019) has explored what is perhaps a more natural problem formulation in which an agent must recover an imitation policy from a dataset containing only expert observation sequences.
Revised: Torabi et al., 2018a;b; Edwards et al., 2019; Sun et al., 2019) has explored what is perhaps a more natural problem formulation in which an agent must recover an imitation policy from a dataset containing only expert observation sequences.

Original: While this Imitation Learning from Observations (ILfO) setting carries tremendous potential, such as enabling an agent to learn complex tasks from watching freely available videos on the Internet, it also is fraught with signiﬁcant additional challenges.
Revised: While this Imitation Learning from Observations (ILfO) setting carries tremendous potential, such as enabling an agent to learn complex tasks from watching freely available videos on the Internet, it also is fraught with signiﬁcant additional challenges.

Original: In this paper, we show how to incorporate recent advances in generative-adversarial training of deep neural networks to tackle imitation-learning problems and advance the state-of-the-art in ILfO.
Revised: In this paper, we show how to incorporate recent advances in generative-adversarial training of deep neural networks to tackle imitation-learning problems and advance the state-of-the-art in ILfO.

Original: With these considerations in mind, the overarching goal of this work is to enable sample-efﬁcient imitation from expert demonstrations, both with and without the provision of expert action labels.
Revised: With these considerations in mind, the overarching goal of this work is to enable sample-efﬁcient imitation from expert demonstrations, both with and without the provision of expert action labels.

Original: The rich literature on Generative
Revised: The rich literature on Generative

Original: Adversarial Networks (Goodfellow et al., 2014) has expanded in recent years to include alternative formulations of the underlying objective that yield qualitatively different solutions to the saddle-point optimization problem (Li et al., 2015;
Revised: Adversarial Networks (Goodfellow et al., 2014) has expanded in recent years to include alternative formulations of the underlying objective that yield qualitatively different solutions to the saddle-point optimization problem (Li et al., 2015;

Original: Dziugaite et al., 2015; Zhao et al., 2016; et al.
Revised: Dziugaite et al., 2015; Zhao et al., 2016; et al.

Original: et al et
Revised: et al et

Original: Of notable interest are the ﬁndings of Nowozin et al.
Revised: Of notable interest are the ﬁndings of Nowozin et al.

Original: (2016) who present Variational Divergence
Revised: (2016) who present Variational Divergence

Original: Minimization (VDM), a generalization of the generative-adversarial approach to arbitrary choices of distance measures between probability distributions drawn from the class of f -divergences (Ali & Silvey, 1966;
Revised: Minimization (VDM), a generalization of the generative-adversarial approach to arbitrary choices of distance measures between probability distributions drawn from the class of f -divergences (Ali & Silvey, 1966;

Original: Csisz ´ ar et al., 2004).
Revised: Csisz ´ ar et al., 2004).

Original: Applying
Revised: Applying

Original: VDM with varying choices of fdivergence, Nowozin et al.
Revised: VDM with varying choices of fdivergence, Nowozin et al.

Original: (2016) encounter learned synthetic distributions that can exhibit differences from one another while producing equally realistic samples.
Revised: (2016) encounter learned synthetic distributions that can exhibit differences from one another while producing equally realistic samples.

Original: Translating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradient reinforcement learning (Sutton et al., 2000).
Revised: Translating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradient reinforcement learning (Sutton et al., 2000).

Original: Existing work in combining adversarial IL and f -divergences (Ke et al., 2019), despite being well-motivated, fails to account for this difference; the end results (shown partially in Figure 1, where TV-VIM is the method of Ke et al.
Revised: Existing work in combining adversarial IL and f -divergences (Ke et al., 2019), despite being well-motivated, fails to account for this difference; the end results (shown partially in Figure 1, where TV-VIM is the method of Ke et al.

Original: (2019), and discussed further in later sections) are imitation-learning algorithms that scale poorly to environments with higher-dimensional observations.
Revised: (2019), and discussed further in later sections) are imitation-learning algorithms that scale poorly to environments with higher-dimensional observations.

Original: In this work, we assess the effect of the VDM principle and consideration of alternative fdivergences in the contexts of IL and ILfO.
Revised: In this work, we assess the effect of the VDM principle and consideration of alternative fdivergences in the contexts of IL and ILfO.

Original: We begin by reparameterizing the framework of Ke et al.
Revised: We begin by reparameterizing the framework of Ke et al.

Original: for the standard IL problem.
Revised: for the standard IL problem.

Original: Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of f -divergence.
Revised: Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of f -divergence.

Original: We then offer a single instantiation of our framework that, in practice, allows stable training of good policies across multiple choices of f -divergence.
Revised: We then offer a single instantiation of our framework that, in practice, allows stable training of good policies across multiple choices of f -divergence.

Original: An example is illustrated in Figure 1 where our methods (TV-VIM-sigmoid and TV-VIMO-sigmoid) result in signiﬁcantly superior policies.
Revised: An example is illustrated in Figure 1 where our methods (TV-VIM-sigmoid and TV-VIMO-sigmoid) result in signiﬁcantly superior policies.

Original: We go on to extend our framework to encapsulate the ILfO setting and examine the efﬁcacy of the resulting new algorithms across a range of continuous-control tasks in the MuJoCo (Todorov et al., 2012) domain.
Revised: We go on to extend our framework to encapsulate the ILfO setting and examine the efﬁcacy of the resulting new algorithms across a range of continuous-control tasks in the MuJoCo (Todorov et al., 2012) domain.

Original: Our empirical results validate our framework as a viable uniﬁcation of adversarial imitation methods under the VDM principle.
Revised: Our empirical results validate our framework as a viable uniﬁcation of adversarial imitation methods under the VDM principle.

Original: With the assistance of recent advances in stabilizing regularization for adversarial training (Mescheder et al., 2018), improvements in performance can be attained under an appropriate choice of f -divergence.
Revised: With the assistance of recent advances in stabilizing regularization for adversarial training (Mescheder et al., 2018), improvements in performance can be attained under an appropriate choice of f -divergence.

Original: However, there is still a signiﬁcant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional observations, leaving open directions for future work in developing improved ILfO algorithms.
Revised: However, there is still a signiﬁcant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional observations, leaving open directions for future work in developing improved ILfO algorithms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The algorithms presented in this work fall in with inverse reinforcement learning (IRL) (Ng et al.
Revised: The algorithms presented in this work fall in with inverse reinforcement learning (IRL) (Ng et al.

Original: Abbeel & Ng, 2004; Syed & Schapire, 2007; Ziebart et al., 2008; Finn et al., 2016; Ho & Ermon, 2016) approaches to IL.
Revised: Abbeel & Ng, 2004; Syed & Schapire, 2007; Ziebart et al., 2008; Finn et al., 2016; Ho & Ermon, 2016) approaches to IL.

Original: Early successes in this regime tend to rely on hand-engineered feature rep- resentations for success (Abbeel & Ng, 2004;
Revised: Early successes in this regime tend to rely on hand-engineered feature rep- resentations for success (Abbeel & Ng, 2004;

Original: Ziebart et al., 2008; Levine et al., 2011).
Revised: Ziebart et al., 2008; Levine et al., 2011).

Original: Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, high-dimensional observations found in real-world control problems (Finn et al., 2016; Ho & Ermon, 2016;
Revised: Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, high-dimensional observations found in real-world control problems (Finn et al., 2016; Ho & Ermon, 2016;

Original: Duan et al., 2017; Li et al., 2017; Fu et al., 2017; Kim & Park, 2018).
Revised: Duan et al., 2017; Li et al., 2017; Fu et al., 2017; Kim & Park, 2018).

Original: Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (Ho & Ermon, 2016;
Revised: Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (Ho & Ermon, 2016;

Original: Li et al., 2017; Fu et al., 2017; Kostrikov et al., 2018); at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) approach which produces high-ﬁdelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for modeling complex distributions over a high-dimensional support.
Revised: Li et al., 2017; Fu et al., 2017; Kostrikov et al., 2018); at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) approach which produces high-ﬁdelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for modeling complex distributions over a high-dimensional support.

Original: From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning (Sutton et al., 2000), allows the agent to shift its own behavior closer to that of the expert.
Revised: From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning (Sutton et al., 2000), allows the agent to shift its own behavior closer to that of the expert.

Original: From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert.
Revised: From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert.

Original: While a large body of prior work exists for IL, recent work has drawn attention to the more challenging problem of imitation learning from observation (Sermanet et al., 2017;
Revised: While a large body of prior work exists for IL, recent work has drawn attention to the more challenging problem of imitation learning from observation (Sermanet et al., 2017;

Original: Liu et al., 2018; Goo & Niekum, 2018; Kimura et al., 2018; Torabi et al., 2018a;b; Edwards et al., 2019; Sun et al., 2019).
Revised: Liu et al., 2018; Goo & Niekum, 2018; Kimura et al., 2018; Torabi et al., 2018a;b; Edwards et al., 2019; Sun et al., 2019).

Original: To more closely resemble observational learning in humans and leverage the wealth of publiclyavailable, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided.
Revised: To more closely resemble observational learning in humans and leverage the wealth of publiclyavailable, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided.

Original: Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (Gupta et al., 2017; Sermanet et al., 2017; Dwibedi et al., 2018; Liu et al., 2018).
Revised: Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (Gupta et al., 2017; Sermanet et al., 2017; Dwibedi et al., 2018; Liu et al., 2018).

Original: In contrast, Torabi et al.
Revised: In contrast, Torabi et al.

Original: (2018a) introduce Behavioral Cloning from Observation (BCO) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two represent a state and next-state pair).
Revised: (2018a) introduce Behavioral Cloning from Observation (BCO) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two represent a state and next-state pair).

Original: With this inverse model in hand, the observation-only demonstration data can be converted into the more traditional dataset of state-action pairs over which standard BC can be applied.
Revised: With this inverse model in hand, the observation-only demonstration data can be converted into the more traditional dataset of state-action pairs over which standard BC can be applied.

Original: Recognizing the previously discussed limitations of BC approaches, Torabi et al.
Revised: Recognizing the previously discussed limitations of BC approaches, Torabi et al.

Original: (2018b) introduce the natural GAIL counterpart for ILfO, Generative Adversarial Imitation from
Revised: (2018b) introduce the natural GAIL counterpart for ILfO, Generative Adversarial Imitation from

Original: Observation (GAIFO)
Revised: Observation (GAIFO)

Original: ; GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), as opposed to stateaction pairs requiring expert action labels.
Revised: ; GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), as opposed to stateaction pairs requiring expert action labels.

Original: While Torabi et al.
Revised: While Torabi et al.

Original: (2018b) offer empirical results for continuous-control tasks with low-dimensional features as well as raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks.
Revised: (2018b) offer empirical results for continuous-control tasks with low-dimensional features as well as raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks.

Original: A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms.
Revised: A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms.

Original: For a more in-depth survey of ILfO approaches, we refer readers to
Revised: For a more in-depth survey of ILfO approaches, we refer readers to

Original: Torabi et al.
Revised: Torabi et al.

Original: We refer readers to the Appendix for a broader overview of prior work.
Revised: We refer readers to the Appendix for a broader overview of prior work.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We begin by formulating the problems of imitation learning and imitation learning from observation respectively before taking a closer look at f -divergences and connecting them to imitation learning.
Revised: We begin by formulating the problems of imitation learning and imitation learning from observation respectively before taking a closer look at f -divergences and connecting them to imitation learning.

Original: 3. I MITATION L EARNING & I MITATION FROM O BSERVATION
Revised: 3. I MITATION L EARNING & I MITATION FROM O BSERVATION

Original: We operate within the Markov Decision Process (MDP) formalism (Bellman, 1957; Puterman, 2014) deﬁned as a ﬁve-tuple M = (cid:104)S , A , R , T , γ (cid:105) where S denotes a (potentially inﬁnite) set of states,
Revised: We operate within the Markov Decision Process (MDP) formalism (Bellman, 1957; Puterman, 2014) deﬁned as a ﬁve-tuple M = (cid:104)S , A , R , T , γ (cid:105) where S denotes a (potentially inﬁnite) set of states,

Original: A denotes a (potentially inﬁnite) set of actions, R : S ×
Revised: A denotes a (potentially inﬁnite) set of actions, R : S ×

Original: A × S → R is a reward function, T :
Revised: A × S → R is a reward function, T :

Original: S × A → ∆( S ) is a transition function, and γ ∈ [0 , 1) is a discount factor.
Revised: S × A → ∆( S ) is a transition function, and γ ∈ [0 , 1) is a discount factor.

Original: At each timestep, the agent observes the current state of the world, s t ∈ S , and randomly samples an action according to its stochastic policy π : S → ∆( A ) .
Revised: At each timestep, the agent observes the current state of the world, s t ∈ S , and randomly samples an action according to its stochastic policy π : S → ∆( A ) .

Original: The environment then transitions to a new state according to the transition function T and produces a reward signal according to the reward function R that is communicative of the agent’s progress through the overall task.
Revised: The environment then transitions to a new state according to the transition function T and produces a reward signal according to the reward function R that is communicative of the agent’s progress through the overall task.

Original: Unlike, the traditional reinforcement learning paradigm, the decision-making problem presented in IL lacks a concrete reward function; in lieu of R , a learner is provided with a dataset of expert demonstrations D = { τ 1 , τ 2 , . . .
Revised: Unlike, the traditional reinforcement learning paradigm, the decision-making problem presented in IL lacks a concrete reward function; in lieu of R , a learner is provided with a dataset of expert demonstrations D = { τ 1 , τ 2 , . . .

Original: τ N } where each τ i =
Revised: τ N } where each τ i =

Original: ( s i 1 , a i 1 , s i 2 , a i 2 , . . . )
Revised: ( s i 1 , a i 1 , s i 2 , a i 2 , . . . )

Original: represents the sequence of states and corresponding actions taken by an expert policy, π ∗ .
Revised: represents the sequence of states and corresponding actions taken by an expert policy, π ∗ .

Original: Naturally, the goal of an IL algorithm is to synthesize a policy π using D , along with access to the MDP M , whose behavior matches that of π ∗ .
Revised: Naturally, the goal of an IL algorithm is to synthesize a policy π using D , along with access to the MDP M , whose behavior matches that of π ∗ .

Original: While the previous section outlines several possible avenues for using D to arrive at a satisfactory imitation policy, our work focuses on adversarial methods that build around GAIL (Ho & Ermon, 2016).
Revised: While the previous section outlines several possible avenues for using D to arrive at a satisfactory imitation policy, our work focuses on adversarial methods that build around GAIL (Ho & Ermon, 2016).

Original: Following from the widespread success of GANs (Goodfellow et al., 2014), GAIL offers a highly-performant approach to IL wherein, at each iteration of the algorithm, transitions sampled from the current imitation policy are ﬁrst used to update a discriminator, D ω ( s, a ) , that acts a binary classiﬁer to distinguish between state-action pairs sampled according to the distributions induced by the expert and student.
Revised: Following from the widespread success of GANs (Goodfellow et al., 2014), GAIL offers a highly-performant approach to IL wherein, at each iteration of the algorithm, transitions sampled from the current imitation policy are ﬁrst used to update a discriminator, D ω ( s, a ) , that acts a binary classiﬁer to distinguish between state-action pairs sampled according to the distributions induced by the expert and student.

Original: Subsequently, treating the imitation policy as a generator, policy-gradient reinforcement learning is used to shift the current policy towards expert behavior, issuing higher rewards for those generated state-action pairs that are regarded as belonging to the expert according to D ω ( s, a ) .
Revised: Subsequently, treating the imitation policy as a generator, policy-gradient reinforcement learning is used to shift the current policy towards expert behavior, issuing higher rewards for those generated state-action pairs that are regarded as belonging to the expert according to D ω ( s, a ) .

Original: More formally, this minimax optimization follows as
Revised: More formally, this minimax optimization follows as

Original: [Equation]
Revised: [Equation]

Original: where ρ π ∗ ( s, a ) and ρ π ( s, a ) denote the undiscounted stationary distributions over state-action pairs for the expert and imitation policies respectively.
Revised: where ρ π ∗ ( s, a ) and ρ π ( s, a ) denote the undiscounted stationary distributions over state-action pairs for the expert and imitation policies respectively.

Original: Here D ω ( s, a ) =
Revised: Here D ω ( s, a ) =

Original: σ ( V ω ( s, a )) where V ω ( s, a ) represents the unconstrained output of a discriminator neural network with parameters ω and σ ( v ) =
Revised: σ ( V ω ( s, a )) where V ω ( s, a ) represents the unconstrained output of a discriminator neural network with parameters ω and σ ( v ) =

Original: (1 + e − x ) − 1 denotes the sigmoid activation function.
Revised: (1 + e − x ) − 1 denotes the sigmoid activation function.

Original: Since the imitation policy only exerts control over the latter term in the above objective, the per-timestep reward function maximized by reinforcement learning is given as r ( s, a, s (cid:48) )
Revised: Since the imitation policy only exerts control over the latter term in the above objective, the per-timestep reward function maximized by reinforcement learning is given as r ( s, a, s (cid:48) )

Original: =
Revised: =

Original: − log(1 − D ω ( s, a )) .
Revised: − log(1 − D ω ( s, a )) .

Original: In practice, an entropy regularization term is often added to the objective when optimizing the imitation policy so as to avoid premature convergence to a suboptimal solution (Mnih et al., 2016;
Revised: In practice, an entropy regularization term is often added to the objective when optimizing the imitation policy so as to avoid premature convergence to a suboptimal solution (Mnih et al., 2016;

Original: Ho & Ermon, 2016; Neu et al., 2017).
Revised: Ho & Ermon, 2016; Neu et al., 2017).

Original: In order to accommodate various observation-only data sources (Zhou et al., 2018) and remove the burden of requiring expert action labels, the imitation from observation setting adjusts the expert demonstration dataset D such that each trajectory τ i =
Revised: In order to accommodate various observation-only data sources (Zhou et al., 2018) and remove the burden of requiring expert action labels, the imitation from observation setting adjusts the expert demonstration dataset D such that each trajectory τ i =

Original: ( s i 1 , s i 2 , . . . )
Revised: ( s i 1 , s i 2 , . . . )

Original: consists only of expert observation sequences.
Revised: consists only of expert observation sequences.

Original: Retaining the goal of recovering an imitation policy that closely resembles expert behavior, Torabi et al.
Revised: Retaining the goal of recovering an imitation policy that closely resembles expert behavior, Torabi et al.

Original: (2018b) introduce GAIFO as the natural extension of GAIL for matching the state transition distribution of the expert policy.
Revised: (2018b) introduce GAIFO as the natural extension of GAIL for matching the state transition distribution of the expert policy.

Original: Note that an objective for matching the stationary distribution over expert state transitions enables the provision of per-timestep feedback while simultaneously avoid the issues of temporal alignment that arise when trying to match trajectories directly.
Revised: Note that an objective for matching the stationary distribution over expert state transitions enables the provision of per-timestep feedback while simultaneously avoid the issues of temporal alignment that arise when trying to match trajectories directly.

Original: The resulting algorithm iteratively ﬁnds a solution to the following minimax optimization:
Revised: The resulting algorithm iteratively ﬁnds a solution to the following minimax optimization:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where ρ π ∗ ( s, s (cid:48) ) and ρ π ( s, s (cid:48) ) now denote the analogous stationary distributions over successive state pairs while D ω ( s, s (cid:48) )
Revised: ) where ρ π ∗ ( s, s (cid:48) ) and ρ π ( s, s (cid:48) ) now denote the analogous stationary distributions over successive state pairs while D ω ( s, s (cid:48) )

Original: =
Revised: =

Original: σ ( V ω ( s, s (cid:48) )) represents binary classiﬁer over state pairs.
Revised: σ ( V ω ( s, s (cid:48) )) represents binary classiﬁer over state pairs.

Original: Similar to
Revised: Similar to

Original: GAIL, the imitation policy is optimized via policy-gradient reinforcement learning with per-timestep rewards computed according to r ( s, a, s (cid:48) )
Revised: GAIL, the imitation policy is optimized via policy-gradient reinforcement learning with per-timestep rewards computed according to r ( s, a, s (cid:48) )

Original: =
Revised: =

Original: − log(1 − D ω ( s, s (cid:48) )) and using entropy regularization as needed.
Revised: − log(1 − D ω ( s, s (cid:48) )) and using entropy regularization as needed.

Original:  A PPROACH
Revised:  A PPROACH

Original: In this section, we begin with an overview of f -divergences, their connection to GANs, and their impact on IL through the f -VIM framework (Ke et al., 2019)
Revised: In this section, we begin with an overview of f -divergences, their connection to GANs, and their impact on IL through the f -VIM framework (Ke et al., 2019)

Original: (Section 4.1).
Revised: (Section 4.1).

Original: We then present an alternative view of the framework that transparently exposes the fundamental choice practictioners must make in order to circumvent practical issues that arise when applying f -VIM to high-dimensional tasks (Section 4.2).
Revised: We then present an alternative view of the framework that transparently exposes the fundamental choice practictioners must make in order to circumvent practical issues that arise when applying f -VIM to high-dimensional tasks (Section 4.2).

Original: We conclude by presenting our approach for ILfO as f -divergence minimization
Revised: We conclude by presenting our approach for ILfO as f -divergence minimization

Original: [Table]
Revised: [Table]

Original: (Section 4.3) followed by a brief discussion of a regularization technique used to stabilize discriminator training in our experiments (Section 4.4).
Revised: (Section 4.3) followed by a brief discussion of a regularization technique used to stabilize discriminator training in our experiments (Section 4.4).

Original: 4. f - DIVERGENCES AND I MITATION L EARNING
Revised: 4. f - DIVERGENCES AND I MITATION L EARNING

Original: The GAIL and GAIFO approaches engage in an adversarial game where the discriminator estimates the divergence between state-action or state transition distributions according to the JensenShannon divergence (Goodfellow et al., 2014).
Revised: The GAIL and GAIFO approaches engage in an adversarial game where the discriminator estimates the divergence between state-action or state transition distributions according to the JensenShannon divergence (Goodfellow et al., 2014).

Original: In this work, our focus is on a more general class of divergences, that includes the Jensen-Shannon divergence, known as Ali-Silvey distances or fdivergences (Ali & Silvey, 1966; Csisz ´ ar et al., 2004).
Revised: In this work, our focus is on a more general class of divergences, that includes the Jensen-Shannon divergence, known as Ali-Silvey distances or fdivergences (Ali & Silvey, 1966; Csisz ´ ar et al., 2004).

Original: For two distributions P and Q with support over a domain X and corresponding continuous densities p and q , we have the f -divergence between them according to:
Revised: For two distributions P and Q with support over a domain X and corresponding continuous densities p and q , we have the f -divergence between them according to:

Original: [Equation]
Revised: [Equation]

Original: where f : R + → R is a convex, lower-semicontinuous function such that f (1) = 0 .
Revised: where f : R + → R is a convex, lower-semicontinuous function such that f (1) = 0 .

Original: As illustrated in Table 1, different choices of function f yield well-known divergences between probability distributions.
Revised: As illustrated in Table 1, different choices of function f yield well-known divergences between probability distributions.

Original: In order to accommodate the tractable estimation of f -divergences when only provided samples from P and Q , Nguyen et al.
Revised: In order to accommodate the tractable estimation of f -divergences when only provided samples from P and Q , Nguyen et al.

Original: (2010) offer an approach for variational estimation of f -divergences.
Revised: (2010) offer an approach for variational estimation of f -divergences.

Original: Central to their procedure is the use of the convex conjugate function or Fenchel conjugate (Hiriart-Urruty & Lemar ´ echal, 2004), f ∗ , which exists for all convex, lower-semicontinuous functions f and is deﬁned as the following supremum:
Revised: Central to their procedure is the use of the convex conjugate function or Fenchel conjugate (Hiriart-Urruty & Lemar ´ echal, 2004), f ∗ , which exists for all convex, lower-semicontinuous functions f and is deﬁned as the following supremum:

Original: [Equation]
Revised: [Equation]

Original: Using the duality of the convex conjugate ( f ∗∗ = f ), Nguyen et al. (2010) represent f ( u ) =
Revised: Using the duality of the convex conjugate ( f ∗∗ = f ), Nguyen et al. (2010) represent f ( u ) =

Original: sup t ∈ dom f ∗ { tu − f ∗ ( t ) } enabling a variational bound:
Revised: sup t ∈ dom f ∗ { tu − f ∗ ( t ) } enabling a variational bound:

Original: [Equation]
Revised: [Equation]

Original: (5) where T is an arbitrary class of functions T : X → dom f ∗ .
Revised: (5) where T is an arbitrary class of functions T : X → dom f ∗ .

Original: Nowozin et al.
Revised: Nowozin et al.

Original: (2016) extend the use of this variational lower bound for GANs that utilize arbitrary f -divergences, or f -GANs.
Revised: (2016) extend the use of this variational lower bound for GANs that utilize arbitrary f -divergences, or f -GANs.

Original: Speciﬁcally, the two distributions of interest are the real data distribution P and a synthetic distribution represented by a generative model Q θ with parameters θ .
Revised: Speciﬁcally, the two distributions of interest are the real data distribution P and a synthetic distribution represented by a generative model Q θ with parameters θ .

Original: The variational function is also parameterized as T ω acting as the discriminator.
Revised: The variational function is also parameterized as T ω acting as the discriminator.

Original: This gives rise to the VDM principle which deﬁnes the f -GAN objective
Revised: This gives rise to the VDM principle which deﬁnes the f -GAN objective

Original: [Equation]
Revised: [Equation]

Original: Nowozin et al.
Revised: Nowozin et al.

Original: (2016) represent the variational function as T ω ( x ) =
Revised: (2016) represent the variational function as T ω ( x ) =

Original: g f ( V ω ( x )) such that V ω ( x ) :
Revised: g f ( V ω ( x )) such that V ω ( x ) :

Original: X → R represents the unconstrained discriminator network while g f : R → dom f ∗ is an activation function chosen in accordance with the f -divergence being optimized.
Revised: X → R represents the unconstrained discriminator network while g f : R → dom f ∗ is an activation function chosen in accordance with the f -divergence being optimized.

Original: Table 1 includes the “somewhat arbitrary” but effective choices for g f suggested by Nowozin et al.
Revised: Table 1 includes the “somewhat arbitrary” but effective choices for g f suggested by Nowozin et al.

Original: and we refer readers to their excellent work for more details and properties of f -divergences and f -GANs.
Revised: and we refer readers to their excellent work for more details and properties of f -divergences and f -GANs.

Original: Recently, Ke et al.
Revised: Recently, Ke et al.

Original: (2019) have formalized the generalization from GAN to f -GAN for the traditional
Revised: (2019) have formalized the generalization from GAN to f -GAN for the traditional

Original: IL problem.
Revised: IL problem.

Original: They offer the f -Variational Imitation ( f -VIM) framework for the speciﬁc case of estimating and then minimizing the divergence between state-action distributions induced by expert and imitation policies:
Revised: They offer the f -Variational Imitation ( f -VIM) framework for the speciﬁc case of estimating and then minimizing the divergence between state-action distributions induced by expert and imitation policies:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where V ω : S ×
Revised: where V ω : S ×

Original: A → R denotes the discriminator network that will supply per-timestep rewards during the outer policy optimization which itself is carried out over policy parameters θ via policygradient reinforcement learning (Sutton et al., 2000).
Revised: A → R denotes the discriminator network that will supply per-timestep rewards during the outer policy optimization which itself is carried out over policy parameters θ via policygradient reinforcement learning (Sutton et al., 2000).

Original: In particular, the per-timestep rewards provided to the agent are given according to r ( s, a, s (cid:48) ) =
Revised: In particular, the per-timestep rewards provided to the agent are given according to r ( s, a, s (cid:48) ) =

Original: f ∗ ( g f ( V ω ( s, a ))) .
Revised: f ∗ ( g f ( V ω ( s, a ))) .

Original: While Ke et al.
Revised: While Ke et al.

Original: (2019) do an excellent job of motivating the use of f -divergences for IL (by formalizing the relationship between divergences over trajectory distributions vs. state-action distributions) and connecting f -VIM to existing imitation-learning algorithms, their experiments focus on smaller problems to study the mode-seeking/mode-covering aspects of different f -divergences and the implications of such behavior depending on the multimodality of the expert trajectory distribution.
Revised: (2019) do an excellent job of motivating the use of f -divergences for IL (by formalizing the relationship between divergences over trajectory distributions vs. state-action distributions) and connecting f -VIM to existing imitation-learning algorithms, their experiments focus on smaller problems to study the mode-seeking/mode-covering aspects of different f -divergences and the implications of such behavior depending on the multimodality of the expert trajectory distribution.

Original: Meanwhile, in the course of attempting to apply f -VIM to large-scale imitation problems, we empirically observe numerical instabilities stemming from function approximation, demanding a reformulation of the framework.
Revised: Meanwhile, in the course of attempting to apply f -VIM to large-scale imitation problems, we empirically observe numerical instabilities stemming from function approximation, demanding a reformulation of the framework.

Original: 4. R EPARAMETERIZING f -VIM
Revised: 4. R EPARAMETERIZING f -VIM

Original: In their presentation of the f -VIM framework, Ke et al.
Revised: In their presentation of the f -VIM framework, Ke et al.

Original: (2019) retain the choices for activation function g f introduced by Nowozin et al.
Revised: (2019) retain the choices for activation function g f introduced by Nowozin et al.

Original: (2016) for f -GANs.
Revised: (2016) for f -GANs.

Original: Recall that these choices of g f play a critical role in deﬁning the reward function optimized by the imitation policy on each iteration of f -VIM, r ( s, a, s (cid:48) )
Revised: Recall that these choices of g f play a critical role in deﬁning the reward function optimized by the imitation policy on each iteration of f -VIM, r ( s, a, s (cid:48) )

Original: =
Revised: =

Original: f ∗ ( g f ( V ω ( s, a ))) .
Revised: f ∗ ( g f ( V ω ( s, a ))) .

Original: It is well known in the reinforcement-learning literature that the nature of the rewards provided to an agent have strong implications on learning success and efﬁciency (Ng et al., 1999; Singh et al., 2010).
Revised: It is well known in the reinforcement-learning literature that the nature of the rewards provided to an agent have strong implications on learning success and efﬁciency (Ng et al., 1999; Singh et al., 2010).

Original: While the activation choices made for f -GANs are suitable given that both optimization problems are carried out by backpropagation, we assert that special care must be taken when specifying these activations (and implicitly, the reward function) for imitation-learning algorithms.
Revised: While the activation choices made for f -GANs are suitable given that both optimization problems are carried out by backpropagation, we assert that special care must be taken when specifying these activations (and implicitly, the reward function) for imitation-learning algorithms.

Original: A combination of convex conjugate and activation function could induce a reward function that engenders numerical instability or a simply challenging reward landscape, depending on the underlying policy-gradient
Revised: A combination of convex conjugate and activation function could induce a reward function that engenders numerical instability or a simply challenging reward landscape, depending on the underlying policy-gradient

Original: algorithm utilized (Henderson et al., 2018).
Revised: algorithm utilized (Henderson et al., 2018).

Original: Empirically, we found that the particular activation choices for the KL and reverse KL divergences shown in Table 1 (linear and exponential, respectively) produced imitation-learning algorithms that, in all of our evaluation environments, failed to complete execution due to numerical instabilities caused by exploding policy gradients.
Revised: Empirically, we found that the particular activation choices for the KL and reverse KL divergences shown in Table 1 (linear and exponential, respectively) produced imitation-learning algorithms that, in all of our evaluation environments, failed to complete execution due to numerical instabilities caused by exploding policy gradients.

Original: In the case of the Total Variation distance, the corresponding f -GAN activation for the variational function is a tanh , requiring a learning agent to traverse a reward interval of [ − 1 , 1] by crossing an intermediate region with reward signals centered around 0 .
Revised: In the case of the Total Variation distance, the corresponding f -GAN activation for the variational function is a tanh , requiring a learning agent to traverse a reward interval of [ − 1 , 1] by crossing an intermediate region with reward signals centered around 0 .

Original: To refactor the f -VIM framework so that it more clearly exposes the choice of reward function to practictioners and shifts the issues of reward scale away from the imitation policy, we propose uniformly applying an activation function g f ( v ) =
Revised: To refactor the f -VIM framework so that it more clearly exposes the choice of reward function to practictioners and shifts the issues of reward scale away from the imitation policy, we propose uniformly applying an activation function g f ( v ) =

Original: f ∗− 1 ( r ( v )) where f ∗− 1 ( t ) denotes the inverse of the convex conjugate (see Table 1).
Revised: f ∗− 1 ( r ( v )) where f ∗− 1 ( t ) denotes the inverse of the convex conjugate (see Table 1).

Original: Here r is effectively a free parameter that can be set according to one of the many heuristics used throughout the ﬁeld of deep reinforcement learning for maintaining a reasonable reward scale (Mnih et al., 2015; 2016;
Revised: Here r is effectively a free parameter that can be set according to one of the many heuristics used throughout the ﬁeld of deep reinforcement learning for maintaining a reasonable reward scale (Mnih et al., 2015; 2016;

Original: Henderson et al., 2018) so long as it obeys the domain of the inverse conjugate dom f ∗− 1 .
Revised: Henderson et al., 2018) so long as it obeys the domain of the inverse conjugate dom f ∗− 1 .

Original: In selecting g f accordingly, the reparameterized saddlepoint optimization for f -VIM becomes
Revised: In selecting g f accordingly, the reparameterized saddlepoint optimization for f -VIM becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the per-timestep rewards used during policy optimization are given by r ( s, a, s (cid:48) )
Revised: where the per-timestep rewards used during policy optimization are given by r ( s, a, s (cid:48) )

Original: =
Revised: =

Original: r ( V ω ( s, a )) .
Revised: r ( V ω ( s, a )) .

Original: In applying this choice, we shift the undesirable scale of the latter term in VDM towards the discriminator, expecting it to be indifferent since training is done by backpropagation.
Revised: In applying this choice, we shift the undesirable scale of the latter term in VDM towards the discriminator, expecting it to be indifferent since training is done by backpropagation.

Original: As one potential instantiation, we consider r ( u ) =
Revised: As one potential instantiation, we consider r ( u ) =

Original: σ ( u ) where σ ( · ) denotes the sigmoid function leading to bounded rewards in the interval [0 , 1] that conveniently adhere to dom f ∗− 1 for almost all of the f -divergences examined in this work 1 .
Revised: σ ( u ) where σ ( · ) denotes the sigmoid function leading to bounded rewards in the interval [0 , 1] that conveniently adhere to dom f ∗− 1 for almost all of the f -divergences examined in this work 1 .

Original: In Section 5, we evaluate imitation-learning algorithms with this choice against those using f -VIM with the original f -GAN activations; we ﬁnd that, without regard for the scale of rewards and the underlying reinforcement-learning problem being solved, the f -GAN activation choices either produce degenerate solutions or completely fail to produce an imitation policy altogether.
Revised: In Section 5, we evaluate imitation-learning algorithms with this choice against those using f -VIM with the original f -GAN activations; we ﬁnd that, without regard for the scale of rewards and the underlying reinforcement-learning problem being solved, the f -GAN activation choices either produce degenerate solutions or completely fail to produce an imitation policy altogether.

Original: 4. f - DIVERGENCES AND I MITATION FROM O BSERVATION
Revised: 4. f - DIVERGENCES AND I MITATION FROM O BSERVATION

Original: Applying the variational lower bound of Nguyen et al.
Revised: Applying the variational lower bound of Nguyen et al.

Original: (2010) and
Revised: (2010) and

Original: the corresponding f -GAN extension, we can now present our Variational Imitation from Observation ( f -VIMO) extension for a general family of ILfO algorithms that leverage the VDM principle in the underlying saddle-point optimization.
Revised: the corresponding f -GAN extension, we can now present our Variational Imitation from Observation ( f -VIMO) extension for a general family of ILfO algorithms that leverage the VDM principle in the underlying saddle-point optimization.

Original: Since optimization of the generator will continue to be carried out by policy-gradient reinforcement learning, we adhere to our reparameterization of the f -VIM framework and present the f -VIMO objective as:
Revised: Since optimization of the generator will continue to be carried out by policy-gradient reinforcement learning, we adhere to our reparameterization of the f -VIM framework and present the f -VIMO objective as:

Original: [Equation]
Revised: [Equation]

Original: with the per-timestep rewards given according to r ( s, a, s (cid:48) )
Revised: with the per-timestep rewards given according to r ( s, a, s (cid:48) )

Original: =
Revised: =

Original: r ( V ω ( s, s
Revised: r ( V ω ( s, s

Original: [Equation]
Revised: [Equation]

Original: We present the full approach as Algorithm 1.
Revised: We present the full approach as Algorithm 1.

Original: Just as in Section 4.2, we again call attention to Line 5 where the discriminator outputs (acting as individual reward signals) scale the policy gradient, unlike the more conventional discriminator optimization of Line 4 by backpropagation; this key difference is the primary motivator for our speciﬁc reparameterization of the f -VIM framework.
Revised: Just as in Section 4.2, we again call attention to Line 5 where the discriminator outputs (acting as individual reward signals) scale the policy gradient, unlike the more conventional discriminator optimization of Line 4 by backpropagation; this key difference is the primary motivator for our speciﬁc reparameterization of the f -VIM framework.

Original: Just as in the previous section, we take r ( u ) =
Revised: Just as in the previous section, we take r ( u ) =

Original: σ ( u ) as a particularly convenient choice of activation given its agreement to the inverse conjugate domains dom f ∗− 1 for many choices of f -divergence and we employ this instantiation throughout all of our experiments.
Revised: σ ( u ) as a particularly convenient choice of activation given its agreement to the inverse conjugate domains dom f ∗− 1 for many choices of f -divergence and we employ this instantiation throughout all of our experiments.

Original: We leave the examination of alternative choices for r to future work.
Revised: We leave the examination of alternative choices for r to future work.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 4. D ISCRIMINATOR R EGULARIZATION
Revised: 4. D ISCRIMINATOR R EGULARIZATION

Original: The refactored version of f -VIM presented in Section 4.2 is fundamentally addressing instability issues that may occur on the generator side of adversarial training; in our experiments, we also examine the utility of regularizing the discriminator side of the optimization for improved stability.
Revised: The refactored version of f -VIM presented in Section 4.2 is fundamentally addressing instability issues that may occur on the generator side of adversarial training; in our experiments, we also examine the utility of regularizing the discriminator side of the optimization for improved stability.

Original: Following from a line of work examining the underlying mathematical properties of GAN optimization (Roth et al., 2017; 2018; Mescheder et al., 2018), we opt for the simple gradient-based regularization of Mescheder et al.
Revised: Following from a line of work examining the underlying mathematical properties of GAN optimization (Roth et al., 2017; 2018; Mescheder et al., 2018), we opt for the simple gradient-based regularization of Mescheder et al.

Original: (2018) which (for f -VIMO) augments the discriminator loss with the following regularization term:
Revised: (2018) which (for f -VIMO) augments the discriminator loss with the following regularization term:

Original: [Equation]
Revised: [Equation]

Original: (10) where ψ is a hyperparameter controlling the strength of the regularization.
Revised: (10) where ψ is a hyperparameter controlling the strength of the regularization.

Original: The form of this speciﬁc penalty follows from the analysis of Roth et al.
Revised: The form of this speciﬁc penalty follows from the analysis of Roth et al.

Original: (2017); intuitively, its purpose is to disincentivize the discriminator from producing a non-zero gradient that shifts away from the Nash equilibrium of the minimax optimization when presented with a generator that perfectly matches the true data distribution.
Revised: (2017); intuitively, its purpose is to disincentivize the discriminator from producing a non-zero gradient that shifts away from the Nash equilibrium of the minimax optimization when presented with a generator that perfectly matches the true data distribution.

Original: While originally developed for traditional GANs and shown to empirically exhibit stronger convergence properties over Wasserstein GANs (Gulrajani et al., 2017), this effect is still desirable for the adversarial IL setting where the reward function (discriminator) used for optimizing the imitation policy should stop changing once the expert state-transition distribution has been matched.
Revised: While originally developed for traditional GANs and shown to empirically exhibit stronger convergence properties over Wasserstein GANs (Gulrajani et al., 2017), this effect is still desirable for the adversarial IL setting where the reward function (discriminator) used for optimizing the imitation policy should stop changing once the expert state-transition distribution has been matched.

Original: In practice, we compare f -VIM and f -VIMO both with and without the use of this regularization term and ﬁnd that R ( ω ) can improve the stability and convergence of f -VIMO across almost all domains.
Revised: In practice, we compare f -VIM and f -VIMO both with and without the use of this regularization term and ﬁnd that R ( ω ) can improve the stability and convergence of f -VIMO across almost all domains.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We examine four instantiations of the f -VIM and f -VIMO frameworks (as presented in Sections 4. and 4.3) corresponding to imitation algorithms with the following choices of f -divergence: GAN,
Revised: We examine four instantiations of the f -VIM and f -VIMO frameworks (as presented in Sections 4. and 4.3) corresponding to imitation algorithms with the following choices of f -divergence: GAN,

Original: Kullback-Leibler, reverse KL, and Total Variation.
Revised: Kullback-Leibler, reverse KL, and Total Variation.

Original: We conduct our evaluation across four MuJoCo environments (Todorov et al., 2012) of varying difﬁculty: Ant, Hopper, HalfCheetah, and Walker (see the Appendix for more details on individual environments).
Revised: We conduct our evaluation across four MuJoCo environments (Todorov et al., 2012) of varying difﬁculty: Ant, Hopper, HalfCheetah, and Walker (see the Appendix for more details on individual environments).

Original: The core questions we seek to answer through our empirical results are as follows:
Revised: The core questions we seek to answer through our empirical results are as follows:

Original: What are the implications of the choice of activation for the variational function in f -VIM on imitation policy performance?
Revised: What are the implications of the choice of activation for the variational function in f -VIM on imitation policy performance?

Original: Do f -divergences act as a meaningful axis of variation for IL and ILfO algorithms?
Revised: Do f -divergences act as a meaningful axis of variation for IL and ILfO algorithms?

Original: What is the impact of discriminator regularization on the stability and convergence properties of f -VIM/ f
Revised: What is the impact of discriminator regularization on the stability and convergence properties of f -VIM/ f

Original: -VIMO? 4.
Revised: -VIMO? 4.

Original: How does the impact of different f -divergences vary with the amount of expert demonstration data
Revised: How does the impact of different f -divergences vary with the amount of expert demonstration data

Original: provided?
Revised: provided?

Original: To answer the ﬁrst three questions above, we report the average total reward achieved by the imitation policy throughout the course of learning with rewards as deﬁned by the corresponding OpenAI
Revised: To answer the ﬁrst three questions above, we report the average total reward achieved by the imitation policy throughout the course of learning with rewards as deﬁned by the corresponding OpenAI

Original: Gym environment (Brockman et al., 2016).
Revised: Gym environment (Brockman et al., 2016).

Original: Shading in all plots denote 95% conﬁdence intervals computed over 10 random trials with 10 random seeds.
Revised: Shading in all plots denote 95% conﬁdence intervals computed over 10 random trials with 10 random seeds.

Original: Expert demonstration datasets of 50 trajectories were collected from agents trained via Proximal Policy Optimization (PPO) (Schulman et al., 2017); 20 expert demonstrations were randomly subsampled at the start of learning and held ﬁxed for the duration of the algorithm.
Revised: Expert demonstration datasets of 50 trajectories were collected from agents trained via Proximal Policy Optimization (PPO) (Schulman et al., 2017); 20 expert demonstrations were randomly subsampled at the start of learning and held ﬁxed for the duration of the algorithm.

Original: We also utilize PPO as the underlying reinforcement-learning algorithm for training the imitation policy with a clipping parameter of 0 . 2 , advantage normalization, entropy regularization coefﬁcient 1 e − 3 , and the Adam optimizer (Kingma & Ba, 2014).
Revised: We also utilize PPO as the underlying reinforcement-learning algorithm for training the imitation policy with a clipping parameter of 0 . 2 , advantage normalization, entropy regularization coefﬁcient 1 e − 3 , and the Adam optimizer (Kingma & Ba, 2014).

Original: Just as in
Revised: Just as in

Original: Ho & Ermon (2016)
Revised: Ho & Ermon (2016)

Original: we use a discount factor of γ = 0 .
Revised: we use a discount factor of γ = 0 .

Original: 995 and apply Generalized Advantage Estimation (Schulman et al., 2015) with parameter λ = 0 . 97 .
Revised: 995 and apply Generalized Advantage Estimation (Schulman et al., 2015) with parameter λ = 0 . 97 .

Original: We run both f -VIM and f -VIMO for a total of 500 iterations, collecting 50000 environment samples per iteration.
Revised: We run both f -VIM and f -VIMO for a total of 500 iterations, collecting 50000 environment samples per iteration.

Original: The policy and discriminator architectures are identically two separate multi-layer perceptrons each with two hidden layers of 100 units separated by tanh nonlinearities.
Revised: The policy and discriminator architectures are identically two separate multi-layer perceptrons each with two hidden layers of 100 units separated by tanh nonlinearities.

Original: A grid search was used for determining the initial learning rate, number of PPO epochs, and number of epochs used for discriminator training (please see the Appendix for more details) and we report results for the best hyperparameter settings.
Revised: A grid search was used for determining the initial learning rate, number of PPO epochs, and number of epochs used for discriminator training (please see the Appendix for more details) and we report results for the best hyperparameter settings.

Original: To address our ﬁnal question we take the best hyperparameter settings recovered when givenexpert demonstrations and re-run all algorithms with { 1 , 5 , 10 , 15 } expert demonstrations that are randomly sampled at the start of each random trial and held ﬁxed for the duration of the algorithm.
Revised: To address our ﬁnal question we take the best hyperparameter settings recovered when givenexpert demonstrations and re-run all algorithms with { 1 , 5 , 10 , 15 } expert demonstrations that are randomly sampled at the start of each random trial and held ﬁxed for the duration of the algorithm.

Original: We then record the average return of the ﬁnal imitation policy for each level of expert demonstration.
Revised: We then record the average return of the ﬁnal imitation policy for each level of expert demonstration.

Original:  R ESULTS & D ISCUSSION
Revised:  R ESULTS & D ISCUSSION

Original: To highlight the importance of carefully selecting the variational function activation g f and validate our modiﬁcations to the f -VIM framework, we present results in Figure 2 comparing to the original f -VIM framework of Ke et al.
Revised: To highlight the importance of carefully selecting the variational function activation g f and validate our modiﬁcations to the f -VIM framework, we present results in Figure 2 comparing to the original f -VIM framework of Ke et al.

Original: and its natural ILfO counterpart.
Revised: and its natural ILfO counterpart.

Original: Activation functions for the original methods are chosen according to the choices outlined in Ke et al.
Revised: Activation functions for the original methods are chosen according to the choices outlined in Ke et al.

Original: ; Nowozin et al.
Revised: ; Nowozin et al.

Original: In our experiments using the KL and reverse KL divergences, we found that none of the trials reached completion due to numerical instabilities caused by exploding policy gradients.
Revised: In our experiments using the KL and reverse KL divergences, we found that none of the trials reached completion due to numerical instabilities caused by exploding policy gradients.

Original: Consequently, we only present results for the Total Variation distance.
Revised: Consequently, we only present results for the Total Variation distance.

Original: We observe that under the original f -GAN activation selection, we fail to produce meaningful imitation policies with learning stagnating after 100 iterations or less.
Revised: We observe that under the original f -GAN activation selection, we fail to produce meaningful imitation policies with learning stagnating after 100 iterations or less.

Original: As previously mentioned, we suspect that this stems from the use of tanh with TV leading to a dissipating reward signal.
Revised: As previously mentioned, we suspect that this stems from the use of tanh with TV leading to a dissipating reward signal.

Original: We present results in Figure 3 to assess the utility of varying the choice of divergence in f -VIM and f -VIMO across each domain.
Revised: We present results in Figure 3 to assess the utility of varying the choice of divergence in f -VIM and f -VIMO across each domain.

Original: In considering the impact of f -divergence choice, we ﬁnd that most of the domains must be examined in isolation to observe a particular subset of f -divergences that stand out.
Revised: In considering the impact of f -divergence choice, we ﬁnd that most of the domains must be examined in isolation to observe a particular subset of f -divergences that stand out.

Original: In the IL setting, we ﬁnd that varying the choice of f -divergence can yield different learning curves but, ultimately, produce near-optimal (if not optimal) imitation policies across all domains.
Revised: In the IL setting, we ﬁnd that varying the choice of f -divergence can yield different learning curves but, ultimately, produce near-optimal (if not optimal) imitation policies across all domains.

Original: In contrast, we ﬁnd meaningful choices of f -divergence in the ILfO setting including { KL, TV } for
Revised: In contrast, we ﬁnd meaningful choices of f -divergence in the ILfO setting including { KL, TV } for

Original: Hopper, RKL for HalfCheetah, and { GAN, TV } for Walker.
Revised: Hopper, RKL for HalfCheetah, and { GAN, TV } for Walker.

Original: We note that the use of discriminator regularization per Mescheder et al.
Revised: We note that the use of discriminator regularization per Mescheder et al.

Original: (2018) is crucial to achieving these performance gains, whereas the regularization generally fails to help performance in the IL setting.
Revised: (2018) is crucial to achieving these performance gains, whereas the regularization generally fails to help performance in the IL setting.

Original: This ﬁnding is supportive of the logical intuition that ILfO poses a fundamentally more-challenging problem than standard IL.
Revised: This ﬁnding is supportive of the logical intuition that ILfO poses a fundamentally more-challenging problem than standard IL.

Original: As a negative result, we ﬁnd that the Ant domain (the most difﬁcult environment with S ⊂ R 111 and
Revised: As a negative result, we ﬁnd that the Ant domain (the most difﬁcult environment with S ⊂ R 111 and

Original: A ⊂ R 8 ) still poses a challenge for ILfO algorithms across the board.
Revised: A ⊂ R 8 ) still poses a challenge for ILfO algorithms across the board.

Original: More speciﬁcally, we observe that discriminator regularization hurts learning in both the IL and ILfO settings.
Revised: More speciﬁcally, we observe that discriminator regularization hurts learning in both the IL and ILfO settings.

Original: While the choice of
Revised: While the choice of

Original: RKL does manage to produce a marginal improvement over GAIFO, the gap between existing stateof-the-art and expert performance remains unchanged.
Revised: RKL does manage to produce a marginal improvement over GAIFO, the gap between existing stateof-the-art and expert performance remains unchanged.

Original: It is an open challenge for future work to either identify the techniques needed to achieve optimal imitation policies from observations only or characterize a fundamental performance gap when faced with sufﬁciently large observation spaces.
Revised: It is an open challenge for future work to either identify the techniques needed to achieve optimal imitation policies from observations only or characterize a fundamental performance gap when faced with sufﬁciently large observation spaces.

Original: In Figure 4, we vary the total number of expert demonstrations available during learning and observe that certain choices of f -divergences can be more robust in the face of less expert data, both in the IL and ILfO settings.
Revised: In Figure 4, we vary the total number of expert demonstrations available during learning and observe that certain choices of f -divergences can be more robust in the face of less expert data, both in the IL and ILfO settings.

Original: We ﬁnd that KL-VIM and TV-VIM are slightly more performant than
Revised: We ﬁnd that KL-VIM and TV-VIM are slightly more performant than

Original: GAIL when only provided with a single expert demonstration.
Revised: GAIL when only provided with a single expert demonstration.

Original: Notably, in each domain we see that certain choices of divergence for f -VIMO do a better job of residing close to their f -VIM counterparts suggesting that future improvements may come from examining f -divergences in the small-data regime.
Revised: Notably, in each domain we see that certain choices of divergence for f -VIMO do a better job of residing close to their f -VIM counterparts suggesting that future improvements may come from examining f -divergences in the small-data regime.

Original: This idea is further exempliﬁed when accounting for results collected while using discriminator regularization (Mescheder et al., 2018).
Revised: This idea is further exempliﬁed when accounting for results collected while using discriminator regularization (Mescheder et al., 2018).

Original: We refer readers to the Appendix for the associated learning curves.
Revised: We refer readers to the Appendix for the associated learning curves.

Original: Our work leaves many open directions for future work to close the performance gap between student and expert policies in the ILfO setting.
Revised: Our work leaves many open directions for future work to close the performance gap between student and expert policies in the ILfO setting.

Original: While we found the sigmoid function to be a suitable instantiation of our framework, exploring alternative choices of variational function activations could prove useful in synthesizing performant ILfO algorithms.
Revised: While we found the sigmoid function to be a suitable instantiation of our framework, exploring alternative choices of variational function activations could prove useful in synthesizing performant ILfO algorithms.

Original: Alternative choices of f -divergences could lead to more substantial improvements than the choices we examine in this paper.
Revised: Alternative choices of f -divergences could lead to more substantial improvements than the choices we examine in this paper.

Original: Moreover, while this work has a direct focus on f -divergences, Integral Probability Metrics (IPMs) (M¨uller, 1997; Gretton et al., 2012) represent a distinct but well-established family of divergences between probability distributions.
Revised: Moreover, while this work has a direct focus on f -divergences, Integral Probability Metrics (IPMs) (M¨uller, 1997; Gretton et al., 2012) represent a distinct but well-established family of divergences between probability distributions.

Original: The success of Total Variation distance in our experiments, which doubles as both a fdivergence and IPM (Sriperumbudur et al., 2009), is suggestive of future work building IPM-based
Revised: The success of Total Variation distance in our experiments, which doubles as both a fdivergence and IPM (Sriperumbudur et al., 2009), is suggestive of future work building IPM-based

Original: ILfO algorithms (Sun et al., 2019).
Revised: ILfO algorithms (Sun et al., 2019).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present a general framework for imitation learning and imitation learning from observations under arbitrary choices of f -divergence.
Revised: In this work, we present a general framework for imitation learning and imitation learning from observations under arbitrary choices of f -divergence.

Original: We empirically validate a single instantiation of our framework across multiple f -divergences, demonstrating that we overcome the shortcomings of prior work and offer a wide class of IL and ILfO algorithms capable of scaling to larger problems.
Revised: We empirically validate a single instantiation of our framework across multiple f -divergences, demonstrating that we overcome the shortcomings of prior work and offer a wide class of IL and ILfO algorithms capable of scaling to larger problems.

Original: A R ELATED W ORK A. L EARNING FROM D EMONSTRATION
Revised: A R ELATED W ORK A. L EARNING FROM D EMONSTRATION

Original: Our work broadly falls within the category of Learning from Demonstration (LfD)
Revised: Our work broadly falls within the category of Learning from Demonstration (LfD)

Original: (Schaal, 1997;
Revised: (Schaal, 1997;

Original: Atkeson & Schaal, 1997; Argall et al., 2009), where an agent must leverage demonstration data (typically provided as trajectories, each consisting of expert state-action pairs) to produce an imitation policy that correctly captures the demonstrated behavior.
Revised: Atkeson & Schaal, 1997; Argall et al., 2009), where an agent must leverage demonstration data (typically provided as trajectories, each consisting of expert state-action pairs) to produce an imitation policy that correctly captures the demonstrated behavior.

Original: Within the context of LfD, a ﬁner distinction can be made between behavioral cloning (BC)
Revised: Within the context of LfD, a ﬁner distinction can be made between behavioral cloning (BC)

Original: (Bain & Sommut, 1999; Pomerleau, 1989) and inverse reinforcement learning (IRL) (Ng et al.
Revised: (Bain & Sommut, 1999; Pomerleau, 1989) and inverse reinforcement learning (IRL) (Ng et al.

Original: ; Abbeel & Ng, 2004; Syed & Schapire, 2007;
Revised: ; Abbeel & Ng, 2004; Syed & Schapire, 2007;

Original: Ziebart et al., 2008;
Revised: Ziebart et al., 2008;

Original: Finn et al., 2016; Ho & Ermon, 2016) approaches; BC approaches view the demonstration data as a standard dataset of input-output pairs and apply traditional supervisedlearning techniques to recover an imitation policy.
Revised: Finn et al., 2016; Ho & Ermon, 2016) approaches; BC approaches view the demonstration data as a standard dataset of input-output pairs and apply traditional supervisedlearning techniques to recover an imitation policy.

Original: Alternatively, IRL-based methods synthesize an estimate of the reward function used to train the expert policy before subsequently applying a reinforcement-learning algorithm (Sutton & Barto, 1998; Abbeel & Ng, 2004) to recover the corresponding imitation policy.
Revised: Alternatively, IRL-based methods synthesize an estimate of the reward function used to train the expert policy before subsequently applying a reinforcement-learning algorithm (Sutton & Barto, 1998; Abbeel & Ng, 2004) to recover the corresponding imitation policy.

Original: Although not a focus of this work, we also acknowledge the myriad of approaches that operate at the intersection of IL and reinforcement learning or augment reinforcement learning with IL (Rajeswaran et al., 2017; Hester et al., 2018; Salimans & Chen, 2018; Sun et al., 2018;
Revised: Although not a focus of this work, we also acknowledge the myriad of approaches that operate at the intersection of IL and reinforcement learning or augment reinforcement learning with IL (Rajeswaran et al., 2017; Hester et al., 2018; Salimans & Chen, 2018; Sun et al., 2018;

Original: Borsa et al., 2019; Tirumala et al., 2019).
Revised: Borsa et al., 2019; Tirumala et al., 2019).

Original: While BC approaches have been successful in some settings (Niekum et al., 2015; Giusti et al., 2016; Bojarski et al., 2016), they are also susceptible to failures stemming from covariate shift where minute errors in the actions of the imitation policy compound and force the agent into regions of the state space not captured in the original demonstration data.
Revised: While BC approaches have been successful in some settings (Niekum et al., 2015; Giusti et al., 2016; Bojarski et al., 2016), they are also susceptible to failures stemming from covariate shift where minute errors in the actions of the imitation policy compound and force the agent into regions of the state space not captured in the original demonstration data.

Original: While some preventative measures for covariate shift do exist (Laskey et al., 2017b), a more principled solution can be found in methods like DAgger (Ross et al., 2011) and its descendants (Ross & Bagnell, 2014; Sun et al., 2017; Le et al., 2018) that remedy covariate shift by querying an expert to provide on-policy action labels.
Revised: While some preventative measures for covariate shift do exist (Laskey et al., 2017b), a more principled solution can be found in methods like DAgger (Ross et al., 2011) and its descendants (Ross & Bagnell, 2014; Sun et al., 2017; Le et al., 2018) that remedy covariate shift by querying an expert to provide on-policy action labels.

Original: It is worth noting, however, that these approaches are only feasible in settings that admit such online interaction with an expert (Laskey et al., 2016) and, even then, failure modes leading to poor imitation policies do exist (Laskey et al., 2017a).
Revised: It is worth noting, however, that these approaches are only feasible in settings that admit such online interaction with an expert (Laskey et al., 2016) and, even then, failure modes leading to poor imitation policies do exist (Laskey et al., 2017a).

Original: The algorithms presented in this work fall in with IRL-based approaches to IL.
Revised: The algorithms presented in this work fall in with IRL-based approaches to IL.

Original: Early successes in this regime tend to rely on hand-engineered feature representations for success (Abbeel & Ng, 2004;
Revised: Early successes in this regime tend to rely on hand-engineered feature representations for success (Abbeel & Ng, 2004;

Original: Ziebart et al., 2008; Levine et al., 2011).
Revised: Ziebart et al., 2008; Levine et al., 2011).

Original: Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, highdimensional observations found in real-world control problems (Finn et al., 2016; Ho & Ermon, 2016;
Revised: Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, highdimensional observations found in real-world control problems (Finn et al., 2016; Ho & Ermon, 2016;

Original: Duan et al., 2017; Li et al., 2017; Fu et al., 2017; Kim & Park, 2018).
Revised: Duan et al., 2017; Li et al., 2017; Fu et al., 2017; Kim & Park, 2018).

Original: Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (Ho & Ermon, 2016;
Revised: Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (Ho & Ermon, 2016;

Original: Li et al., 2017; Fu et al., 2017; Kostrikov et al., 2018); at the heart of these methods is the Generative Adversarial Imitation
Revised: Li et al., 2017; Fu et al., 2017; Kostrikov et al., 2018); at the heart of these methods is the Generative Adversarial Imitation

Original: Learning (GAIL) (Ho & Ermon, 2016) approach which produces high-ﬁdelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for modeling complex distributions over a high-dimensional support.
Revised: Learning (GAIL) (Ho & Ermon, 2016) approach which produces high-ﬁdelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for modeling complex distributions over a high-dimensional support.

Original: From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning (Sutton et al., 2000), allows the agent to shift its own behavior closer to that of the expert.
Revised: From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning (Sutton et al., 2000), allows the agent to shift its own behavior closer to that of the expert.

Original: From the perspective of
Revised: From the perspective of

Original: GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert.
Revised: GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert.

Original: While a large body of prior work exists for IL, numerous recent works have drawn attention to the more challenging problem of imitation learning from observation (Sermanet et al., 2017;
Revised: While a large body of prior work exists for IL, numerous recent works have drawn attention to the more challenging problem of imitation learning from observation (Sermanet et al., 2017;

Original: Liu et al., 2018; Goo & Niekum, 2018; Kimura et al., 2018; Torabi et al., 2018a;b; Edwards et al., 2019; Sun et al., 2019).
Revised: Liu et al., 2018; Goo & Niekum, 2018; Kimura et al., 2018; Torabi et al., 2018a;b; Edwards et al., 2019; Sun et al., 2019).

Original: In an effort to more closely resemble observational learning in humans and leverage the wealth of publicly-available, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided.
Revised: In an effort to more closely resemble observational learning in humans and leverage the wealth of publicly-available, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided.

Original: Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (Gupta et al., 2017; Sermanet et al., 2017; Dwibedi et al., 2018; Liu et al., 2018).
Revised: Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (Gupta et al., 2017; Sermanet et al., 2017; Dwibedi et al., 2018; Liu et al., 2018).

Original: In contrast, Torabi et al.
Revised: In contrast, Torabi et al.

Original: (2018a) introduce Behavioral Cloning from Observation (BCO) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two represent a state and next-state pair).
Revised: (2018a) introduce Behavioral Cloning from Observation (BCO) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two represent a state and next-state pair).

Original: With this inverse model in hand, the observation-only demonstration data can be converted into the more traditional dataset of stateaction pairs over which standard BC can be applied.
Revised: With this inverse model in hand, the observation-only demonstration data can be converted into the more traditional dataset of stateaction pairs over which standard BC can be applied.

Original: Recognizing the previously discussed limitations of BC approaches, Torabi et al.
Revised: Recognizing the previously discussed limitations of BC approaches, Torabi et al.

Original: (2018b) introduce the natural GAIL counterpart for ILfO,
Revised: (2018b) introduce the natural GAIL counterpart for ILfO,

Original: Generative Adversarial Imitation from Observation (GAIFO); GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), as opposed to state-action pairs requiring expert action labels.
Revised: Generative Adversarial Imitation from Observation (GAIFO); GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), as opposed to state-action pairs requiring expert action labels.

Original: While Torabi et al.
Revised: While Torabi et al.

Original: (2018b) offer empirical results for continuous-control tasks with low-dimensional features as well as raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks.
Revised: (2018b) offer empirical results for continuous-control tasks with low-dimensional features as well as raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks.

Original: A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms.
Revised: A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms.

Original: For a more in-depth survey of
Revised: For a more in-depth survey of

Original: ILfO approaches, we refer readers to Torabi et al.
Revised: ILfO approaches, we refer readers to Torabi et al.

Original: A. G ENERATIVE A DVERSARIAL N ETWORKS
Revised: A. G ENERATIVE A DVERSARIAL N ETWORKS

Original: With a focus on generative-adversarial methods for IL, this work leverages several related ideas in the GAN literature for offering alternative formulations as well as improving understanding of their underlying mathematical foundations (Li et al., 2015;
Revised: With a focus on generative-adversarial methods for IL, this work leverages several related ideas in the GAN literature for offering alternative formulations as well as improving understanding of their underlying mathematical foundations (Li et al., 2015;

Original: Dziugaite et al., 2015; Zhao et al., 2016;
Revised: Dziugaite et al., 2015; Zhao et al., 2016;

Original: Nowozin et al., 2016; Roth et al., 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Roth et al., 2018; Mescheder et al., 2018).
Revised: Nowozin et al., 2016; Roth et al., 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Roth et al., 2018; Mescheder et al., 2018).

Original: Critical to the ideas presented in many of these previous works is an understanding that discriminator networks are estimating a divergence between two probability distributions of interest, usually taken to be the real data distribution and the fake or synthetic distribution represented by the generator.
Revised: Critical to the ideas presented in many of these previous works is an understanding that discriminator networks are estimating a divergence between two probability distributions of interest, usually taken to be the real data distribution and the fake or synthetic distribution represented by the generator.

Original: Formal characterizations of this divergence, either by Integral
Revised: Formal characterizations of this divergence, either by Integral

Original: Probability Metrics (IPMs) (M¨uller, 1997; Gretton et al., 2012) or f -divergences (Ali & Silvey, 1966; Csisz´ar et al., 2004; Liese & Vajda, 2006), yield different variations on the classic GAN formulation which is itself a slight variation on the Jensen-Shannon (JS) divergence (Li et al., 2015;
Revised: Probability Metrics (IPMs) (M¨uller, 1997; Gretton et al., 2012) or f -divergences (Ali & Silvey, 1966; Csisz´ar et al., 2004; Liese & Vajda, 2006), yield different variations on the classic GAN formulation which is itself a slight variation on the Jensen-Shannon (JS) divergence (Li et al., 2015;

Original: Dziugaite et al., 2015; Zhao et al., 2016;
Revised: Dziugaite et al., 2015; Zhao et al., 2016;

Original: Nowozin et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017).
Revised: Nowozin et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017).

Original: Following from work by Nowozin et al.
Revised: Following from work by Nowozin et al.

Original: (2016) to generalize the GAN objective to arbitrary f -divergences, Ke et al.
Revised: (2016) to generalize the GAN objective to arbitrary f -divergences, Ke et al.

Original: (2019) offer a generalization of GAIL to an arbitrary choice of f -divergence for quantifying the gap between the state-action visitation distributions of the imitation and expert policies; moreover, Ke et al.
Revised: (2019) offer a generalization of GAIL to an arbitrary choice of f -divergence for quantifying the gap between the state-action visitation distributions of the imitation and expert policies; moreover, Ke et al.

Original: (2019) propose a unifying framework for IL, f -Variational IMitation ( f -VIM), in which they highlight a correspondence between particular choices of f -divergences and existing IL algorithms (speciﬁcally BC ⇐⇒ Kullback-Leibler (KL) divergence, DAgger ⇐⇒ TotalVariation distance, and GAIL ⇐⇒ JS-divergence 2 ).
Revised: (2019) propose a unifying framework for IL, f -Variational IMitation ( f -VIM), in which they highlight a correspondence between particular choices of f -divergences and existing IL algorithms (speciﬁcally BC ⇐⇒ Kullback-Leibler (KL) divergence, DAgger ⇐⇒ TotalVariation distance, and GAIL ⇐⇒ JS-divergence 2 ).

Original: While Ke et al.
Revised: While Ke et al.

Original: (2019) focus on providing empirical results in smaller toy problems to better understand the interplay between f -divergence choice and the multimodality of the expert trajectory distribution, we provide an empirical evaluation of their f -VIM framework across a range of continous control tasks in the Mujoco domain (Todorov et al., 2012).
Revised: (2019) focus on providing empirical results in smaller toy problems to better understand the interplay between f -divergence choice and the multimodality of the expert trajectory distribution, we provide an empirical evaluation of their f -VIM framework across a range of continous control tasks in the Mujoco domain (Todorov et al., 2012).

Original: Empirically, we ﬁnd that some of the design choices f -VIM inherits from the original f -GAN work (Nowozin et al., 2016) are problematic when coupled with adversarial IL and training of the generator by policy-gradient reinforcement learning, instead of via direct backpropagation as in traditional GANs.
Revised: Empirically, we ﬁnd that some of the design choices f -VIM inherits from the original f -GAN work (Nowozin et al., 2016) are problematic when coupled with adversarial IL and training of the generator by policy-gradient reinforcement learning, instead of via direct backpropagation as in traditional GANs.

Original: Consequently, we refactor their framework to expose this point and provide one practical instantiation that works well empirically.
Revised: Consequently, we refactor their framework to expose this point and provide one practical instantiation that works well empirically.

Original: We then go on to extend the f -VIM framework to the IFO problem ( f -VIMO) and evaluate the resulting algorithms empirically against the state-of-the-art, GAIFO.
Revised: We then go on to extend the f -VIM framework to the IFO problem ( f -VIMO) and evaluate the resulting algorithms empirically against the state-of-the-art, GAIFO.

Original: B E XPERIMENT D ETAILS
Revised: B E XPERIMENT D ETAILS

Original: Here we provide details of the MuJoCo environments (Todorov et al., 2012) used in our experiments as well as the details of the hyperparameter search conducted for all algorithms (IL and ILfO) presented.
Revised: Here we provide details of the MuJoCo environments (Todorov et al., 2012) used in our experiments as well as the details of the hyperparameter search conducted for all algorithms (IL and ILfO) presented.

Original: B. M U J O C O E NVIRONMENTS
Revised: B. M U J O C O E NVIRONMENTS

Original: All environments have continuous observation and action spaces of varying dimensionality (as shown below).
Revised: All environments have continuous observation and action spaces of varying dimensionality (as shown below).

Original: All algorithms evaluated in each environment were trained for a total of 500 iterations, collecting 50 , 000 environment transitions per iteration.
Revised: All algorithms evaluated in each environment were trained for a total of 500 iterations, collecting 50 , 000 environment transitions per iteration.

Original: Task
Revised: Task

Original: Observation Space
Revised: Observation Space

Original: Action Space
Revised: Action Space

Original: Ant-v
Revised: Ant-v

Original: R R
Revised: R R

Original: Hopper-v
Revised: Hopper-v

Original: R R
Revised: R R

Original: HalfCheetah-v
Revised: HalfCheetah-v

Original: R R
Revised: R R

Original: Walker2d-v
Revised: Walker2d-v

Original: R R
Revised: R R

Original: B. H YPERPARAMETERS
Revised: B. H YPERPARAMETERS

Original: Below we outline the full set of hyperparameters examined for all experiments presented in this work.
Revised: Below we outline the full set of hyperparameters examined for all experiments presented in this work.

Original: We conducted a full grid search over 10 random trials with 10 random seeds and report results for the best hyperparameter setting.
Revised: We conducted a full grid search over 10 random trials with 10 random seeds and report results for the best hyperparameter setting.

Original: Hyperparameter
Revised: Hyperparameter

Original: Values
Revised: Values

Original: Discriminator learning rate {
Revised: Discriminator learning rate {

Original: [Equation]
Revised: [Equation]

Original: PPO epochs { 5 , 10 }
Revised: PPO epochs { 5 , 10 }

Original: Discriminator epochs { 1 , 5 , 10 }
Revised: Discriminator epochs { 1 , 5 , 10 }

Original: Preliminary experiments were
Revised: Preliminary experiments were

Original: C A DDITIONAL R ESULTS C. U NREGULARIZED f -VIM/VIMO
Revised: C A DDITIONAL R ESULTS C. U NREGULARIZED f -VIM/VIMO

Original: [Figure]
Revised: [Figure]

Revised: C. f -D IVERGENCE V ARIATIONAL B OUND S WAP

Revised: Throughout this paper, we advocate for the use of the following variational lower bound to the f -divergence for both f -VIM and f -VIMO:

Revised: [Equation]

Revised: In particular, we value the above form as it clearly exposes the choice of reward function for the imitation policy as a free parameter that, in practice, has strong implications for the stability and convergence of adversarial IL/ILfO algorithms.

Revised: Alternatively, one may consider appealing to the original lower bound of Nguyen et al.

Revised: (2010), used in f -GANs (Nowozin et al., 2016) unmodiﬁed, but swapping the positions of the two distributions:

Revised: [Equation]

Revised: Consequently, the term in this lower bound pertaining to the imitation policy is now similar to that of the bound in Equation 11; namely, an almost arbitrary activation function, g f , applied to the output of the variational function (discriminator) V ω .

Revised: The difference being that the codomain of g f must obey the domain of the convex conjugate, f ∗ , while the codomain of r must respect the domain of the inverse convex conjugate, f ∗− 1 .

Revised: We evaluate these two choices empirically below for the speciﬁc choice of the KL-divergence in the Ant and Hopper domains (the two most difﬁcult domains of our evaluation).

Revised: We ﬁnd that the original unswapped bound in Equation 11 used throughout this paper outperforms the variants with the distributions swapper, for both the IL and ILfO settings.

Revised: Crucially, we ﬁnd that the KL-VIM in the Ant domain no longer achieves expert performance while optimizing the swapped bound.