I MITATION L EARNING VIA O FF -P OLICY D ISTRIBUTION M ATCHING
A BSTRACT
When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm.
Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly datainefﬁcient or alter the original objective in a manner that can drastically change its optimum.
In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective.
In addition to the data-efﬁciency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary.
Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards.
We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, ﬁnding that it can achieve state-of-the-art sample efﬁciency and performance.
I NTRODUCTION
Reinforcement learning (RL) is typically framed as learning a behavior policy based on reward feedback from trial-and-error experience.
Accordingly, many successful demonstrations of RL often rely on carefully handcrafted rewards with various bonuses and penalties designed to encourage intended behavior (Nachum et al., 2019a; Andrychowicz et al., 2018).
In contrast, many real-world behaviors are easier to demonstrate rather than devise explicit rewards.
This realization is at the heart of imitation learning (Ho & Ermon, 2016; Ng et al.
; Pomerleau, 1989), in which one aims to learn a behavior policy from a set of expert demonstrations – logged experience data of a near-optimal policy interacting with the environment – without explicit knowledge of rewards.
Distribution matching via adversarial learning, or Adversarial Imitation Learning (AIL), has recently become a popular approach for imitation learning (Ho & Ermon, 2016; Fu et al., 2017; Ke et al., 2019; Kostrikov et al., 2019).
These methods interpret the states and actions provided in the expert demonstrations as a ﬁnite sample from a target distribution.
Imitation learning can then be framed as learning a behavior policy which minimizes a divergence between this target distribution and the state-action distribution induced by the behavior policy interacting with the environment.
As derived by Ho & Ermon (2016), this divergence minimization may be achieved by iteratively performing two alternating steps, reminiscent of GAN algorithms (Goodfellow et al., 2014).
First, one estimates the density ratio of states and actions between the target distribution and the behavior policy.
Then, these density ratios are used as rewards for a standard RL algorithm, and the behavior policy is updated to maximize these cumulative rewards (data distribution ratios).
The main limitation of current distribution matching approaches is that estimating distribution density ratios (the ﬁrst step of every iteration) typically requires samples from the behavior policy distribution.
This means that every iteration – every update to the behavior policy – requires new interactions with the environment, precluding the use of these algorithms in settings where interactions with the environment are expensive and limited.
Several papers attempt to relax this on-policy requirement and resolve the sample inefﬁciency problem by designing off-policy imitation learning algorithms, which may take advantage of past logged data, usually in the form of a replay buffer (Kostrikov et al., 2019; Sasaki et al., 2019).
However, these methods do so by altering the original divergence minimization objective to measure a divergence between the target expert distribution and the replay buffer distribution.
Accordingly, there is no guarantee that the learned policy will recover the desired target distribution.
In this work, we introduce an algorithm for imitation learning that, on the one hand, performs divergence minimization as in the original AIL methods, while on the other hand, is completely off-policy.
We begin by providing a new formulation of the minimum divergence objective that avoids the use of any explicit on-policy expectations.
While this objective may be used in the traditional way to estimate data distribution ratios that are then input to an RL algorithm, we go further to show how the speciﬁc form of the derived objective renders the use of a separate RL optimization unnecessary.
Rather, gradients of the minimum divergence objective with respect to behavior policy may be computed directly.
This way, an imitating behavior policy may be learned to minimize the divergence without the use of explicit rewards.
We call this stream-lined imitation learning algorithm
ValueDICE.
In addition to being simpler than standard imitation learning methods, we show that our proposed algorithm is able to achieve state-of-the-art performance on a suite of imitation learning benchmarks.
B ACKGROUND
We consider environments represented as a Markov Decision Process (MDP) (Puterman, 2014), deﬁned by the tuple, ( S , A , p 0 ( s ) , p ( s (cid:48) | s, a ) , r ( s, a ) , γ ) , where S and A are the state and action space, respectively, p 0 ( s ) is an initial state distribution, p ( s (cid:48) | s, a ) deﬁnes environment dynamics represented as a conditional state distribution, r ( s, a ) is a reward function, and γ is a return discount factor.
A behavior policy π ( ·|· ) interacts with the environment to yield experience ( s t , a t , r t , s t +1 ) , for t = 0 , 1 , . .
, where s 0 ∼ p 0 ( · ) , a t ∼ π ( ·| s t ) , s t +1 ∼ p ( ·| s t , a t ) , r t = r ( s t , a t ) .
Without loss of generality, we consider inﬁnite-horizon, non-terminating environments.
In standard RL, one aims to learn a behavior policy π ( ·| s ) to maximize cumulative rewards, based on experience gained from interacting with the environment.
In imitation learning (Pomerleau, 1989; Abbeel & Ng, 2004; Ho & Ermon, 2016), the environment reward is not observed.
Rather, one has access to a set of expert demonstrations D := { ( s k , a k , s (cid:48) k } Nk =1 given by state-action-next-state transitions in the environment induced by an unknown expert policy π exp and the goal is to learn a behavior policy π which recovers π exp .
During the learning process, in addition to the ﬁnite set of expert demonstrations D , one may also optionally interact with the environment (in these interactions, no rewards are observed).
This setting describes a number of real-world applications where rewards are unknown, such as Pomerleau (1989); Muller et al.
; Bojarski et al.
2. B EHAVIORAL C LONING (BC)
Supervised behavioral cloning (BC) is a popular approach for imitation learning.
Given a set of expert demonstrations, a mapping of state observations to actions is ﬁt using regression or density estimation.
In the simplest case, one simply trains the behavior policy π to minimize the negative log-likelihood of the observed expert actions:
[Equation]
Unlike Inverse Reinforcement Learning (IRL) algorithms (e.g. GAIL (Ho & Ermon, 2016)), BC does not perform any additional policy interactions with the learning environment and hence does not suffer from the same issue of policy sample complexity.
However, behavioral cloning suffers from distributional drift (Ross et al., 2011); i.e., there is no way for π to learn how to recover if it deviates from the expert behavior to a state ˜ s not seen in the expert demonstrations.
2. D ISTRIBUTION M ATCHING
The distribution matching approach provides a family of methods that are robust to distributional shift.
Rather than considering the policy directly as a conditional distribution π ( ·| s ) over actions, this approach considers the state-action distribution induced by a policy.
In particular, under certain conditions (Puterman, 2014), there is a one-to-one correspondence between a policy and its stateaction distribution d π deﬁned as,
[Equation]
By the same token, the unknown expert policy π exp also possesses a state-action distribution d exp , and one may usually assume that the expert demonstrations D :
= { ( s k , a k , s (cid:48) k } Nk =1 are sampled as ( s k , a k ) ∼ d exp , s (cid:48)
k ∼ p ( ·| s k , a k ) .
Accordingly, the distribution matching approach proposes to learn π to minimize the divergence between d π and d exp .
The KL-divergence is typically used to measure the discrepancy between d π and d exp (Ho & Ermon, 2016; Ke et al., 2019):
[Equation]
The use of the KL-divergence is convenient, as it may be expressed as an RL problem where rewards are given by log distribution ratios:
[Equation]
In other words, if one has access to estimates of the distribution ratios of the two policies, then the minimum divergence problem reduces to a max-return RL problem with rewards ˜
r ( s, a ) = log d exp ( s,a ) d π ( s,a ) .
Any on-policy or off-policy RL algorithm can be used to maximize the corresponding expected returns in Equation 4.
Capitalizing on this observation, Ho & Ermon (2016) and Ke et al.
(2019) propose algorithms (e.g.,
GAIL) in which the distribution ratio is estimated using a GAN-like objective:
[Equation]
In this objective, the function h acts as a discriminator , discriminating between samples ( s, a ) from d exp and d π .
The optimal discriminator satisﬁes,
[Equation]
(6) and so the distribution matching rewards may be computed as ˜ r
( s [Equation]
a ) = log h ∗ ( s
[Equation]
a ) − log(1 − h ∗ ( s, a )) .
In practice, the discriminator is not fully optimized, and instead gradient updates to the discriminator and policy are alternated.
These prior distribution matching approaches possess two limitations which we will resolve with our proposed ValueDICE algorithm:
• On-policy.
Arguably the main limitation of these prior approaches is that they require access to on-policy samples from d π .
While off-policy RL can be used for learning π , optimizing the discriminator h necessitates having on-policy samples (the second expectation in Equation 5).
Thus, in practice, GAIL requires a prohibitively large number of environment interactions, making it unfeasible for use in many real-world applications.
Attempts to remedy this, such as DiscriminatorActor-Critic (DAC)
(Kostrikov et al., 2019), often do so via ad-hoc methods; for example, changing the on-policy expectation E ( s,a ) ∼ d π [log(1 − h ( s, a ))] in Equation 5 to an expectation over the replay buffer E ( s,a ) ∼ d RB [log(1 − h ( s, a ))] .
While DAC achieves good empirical results, it does not guarantee distribution matching of π to π exp , especially when d RB is far from d π . • Separate RL optimization.
Prior approaches require iteratively taking alternating steps: ﬁrst estimate the data distribution ratios using the GAN-like objective, then input these into an RL optimization and repeat.
The use of a separate RL algorithm introduces complexity to any implementation of these approaches, with many additional design choices that need to be made and more function approximators to learn (e.g., value functions).
Our introduced ValueDICE will be shown to not need a separate RL optimization.
O FF - POLICY F ORMULATION OF THE KL-D IVERGENCE
As is standard in distribution matching, we begin with the KL-divergence between the policy stateaction occupancies and the expert.
However, in contrast to the form used in Equation 4 or 5, we use the Donsker-Varadhan representation (Donsker & Varadhan, 1983) given by,
[Equation]
Similar to Equation 5, this dual representation of the KL has a property that is important for imitation learning.
The optimal x ∗ is equal to the log distribution ratio (plus a constant):
[Equation]
In our considered inﬁnite-horizon setting, the constant does not affect optimality and so we will ignore it (take C = 0 ).
If one were to take a GAIL-like approach, they could use this form of the
KL to estimate distribution matching rewards given by ˜ r
( s, a ) =
− x ∗ ( s, a ) , and these could then be maximized by any standard RL algorithm.
However, there is no clear advantage of this objective over GAIL since it still relies on an expectation with respect to on-policy samples from d π .
To make this objective practical for off-policy learning, we take inspiration from derivations used in
DualDICE (Nachum et al., 2019b), and perform the following change of variables:
[Equation]
[Equation]
where B π is the expected Bellman operator with respect to policy π and zero reward:
[Equation]
This change of variables is explicitly chosen to take advantage of the linearity of the second expectation in Equation 7.
Speciﬁcally, the representation becomes,
[Equation]
where the second expectation conveniently telescopes and reduces to an expectation over initial states (see Nachum et al.
(2019b) for details):
[Equation]
Thus we achieve our ValueDICE 4 objective.
It allows us to express the KL-divergence between d π and d exp in terms of an objective over a ‘value-function’ ν expressed in an off-policy manner, with expectations over expert demonstrations d exp and initial state distribution p 0 ( · ) .
It is clear that the derived objective in Equation 12 possesses two key characteristics missing from prior distribution matching algorithms: First, the objective does not rely on access to samples from the on-policy distribution d π , and so may be used in more realistic, off-policy settings.
Second, the objective describes a proper divergence between d π and d exp , as opposed to estimating a divergence between d RB and d exp , and thus avoids poor behavior when d RB is far from d π .
In the following section, we will go further to show how the objective in Equation 12 also renders the use of a separate
RL optimization unnecessary.
V ALUE DICE: I MITATION L EARNING WITH I MPLICIT R EWARDS
Although it is standard in distribution matching to have separate optimizations for estimating the distribution ratios and learning a policy, in our case this can be mitigated.
Indeed, looking at our formulation of the KL in Equation 12, we see that gradients of this objective with respect to π may be easily computed.
Speciﬁcally, we may express the distribution matching objective for π as a max - min optimization:
[Equation]
If the inner objective over ν is sufﬁciently optimized [Equation] the gradients of π may be computed directly (Bertsekas, 1999)
[Equation]
noting that, [Equation] In continuous control environments when π is parameterized by a Gaussian and ν is a neural network, one may use the re-parameterization trick (Haarnoja et al., 2018) to compute gradients of the ν -values with respect to policy mean and variance directly as opposed to computing ∇ log π ( a | s ) .
Please see the appendix for a full pseudocode implementation of ValueDICE.
We note that in practice, as in GAIL, we do not train ν until optimality but rather alternate ν and π updates.
The mechanics of learning π according to the ValueDICE objective are straightforward, but what is the underlying reason for this more streamlined policy learning?
How does it relate the standard protocol of alternating data distribution estimation with RL optimization?
To better understand this, we consider the form of ν when it is completely optimized.
If we consider the original change of variables (Equation 9) and optimality characterization (Equation 8) we have,
[Equation]
From this characterization of ν ∗ , we realize that ν ∗ is a sort of Q-value function: ν ∗ ( s, a ) is the future discounted sum of rewards ˜
r ( s, a ) :
= log d π ( s,a ) d exp ( s,a ) when acting according to π .
The gradients for π then encourage the policy to choose actions which minimize ν ∗ ( s, a ) , i.e., maximize future discounted log ratios log d exp ( s,a ) d π ( s,a ) .
Thus we realize that the objective for π in ValueDICE performs exactly the RL optimization suggested by Equation 4.
The streamlined nature of ValueDICE comes from the fact that the value function ν (which would traditionally need to be learned as a critic in a separate actor-critic RL algorithm) is learned directly from the same objective as that used for distribution matching.
Thus, in addition to estimating a proper divergence between d π and d exp in an off-policy manner,
ValueDICE also greatly simpliﬁes the implementation of distribution matching algorithms.
There is no longer a need to use a separate RL algorithm for learning π , and moreover, the use of ν as a value function removes any use of explicit rewards.
Instead, the objective and implementation are only in terms of policy π and function ν .
S OME P RACTICAL C ONSIDERATIONS
In order to make use of the ValueDICE objective (Equation 13) in practical scenarios, where one does not have access to d exp or p 0 ( · ) but rather only limited ﬁnite samples, we perform several modiﬁcations.
5. E MPIRICAL E XPECTATIONS
The objective in Equation 13 contains three expectations:
An expectation over d exp (the ﬁrst term of the objective).
Note that this expectation has a logarithm outside of it, which would make any mini-batch approximations of the gradient of this expectation biased.
An expectation over p 0 ( · ) (the second term of the objective).
This term is linear, and so is very amenable to mini-batch optimization.
An expectation over the environment transition p ( ·| s, a ) used to compute B π ν ( s, a ) .
This expectation has a log-expected-exponent applied to it, so its mini-batch approximated gradient would be biased in general.
For the ﬁrst expectation, previous works have suggested a number of remedies to reduce the bias of mini-batch gradients, such as maintaining moving averages of various quantities (Belghazi et al., 2018).
In the setting we considered, we found this to have a negligible effect on performance.
In fact, simply using the biased mini-batched gradients was sufﬁcient for good performance, and so we used this for our experiments.
For the second expectation, we use standard mini-batch gradients, which are unbiased.
Although initial state distributions are usually not used in imitation learning, it is easy to record initial states as they are observed, and thus have access to an empirical sample from p 0 .
Furthermore, as detailed in
Section 5.3, it is possible to modify the initial state distribution used in the objective without adverse effects.
Finally, for the third expectation, previous works have suggested the use of Fenchel conjugates to remove the bias (Nachum et al., 2019b).
In our case, we found this unnecessary and instead use a biased estimate based on the single sample s (cid:48) ∼ p ( ·| s, a ) .
This naive approach was enough to achieve good performance on the benchmark domains we considered.
In summary, the empirical form of the objective is given by,
[Equation]
, a 0 )] (cid:21) , (17) where batch( D ) is a mini-batch from D and batch( p 0 ) is a mini-batch from the recorded initial states ˆ p 0 .
5. R EPLAY B UFFER R EGULARIZATION
The original ValueDICE objective uses only expert samples and the initial state distribution.
In practice, the number of expert samples may be small and lack diversity, hampering learning.
In order to increase the diversity of samples used for training, we consider an alternative objective, with a controllable regularization based on experience in the replay buffer:
[Equation]
where d mix ( s, a ) =
(1 − α )
d exp ( s, a ) + αd RB ( s, a ) .
The main advantage of this formulation is that it introduces ν -values into the objective on samples that are outside the given expert demonstrations.
Thus, if π deviates from the expert trajectory, we will still be able to learn optimal actions that return the policy back to the expert behavior.
At the same time, one can verify that in this formulation the optimal π still matches π exp , unlike other proposals for incorporating a replay buffer distribution (Kostrikov et al., 2019).
Indeed, the objective in Equation 18 corresponds to the Donsker-Varadhan representation,
[Equation]
x ( s, a )] , (19) and so the optimal values of ν ∗ satisfy,
[Equation]
Therefore, the global optimality of π = π exp is unaffected by any choice of α < 1 .
We note that in practice we use a small value α = 0 . 1 for regularization.
5. I NITIAL S TATE S AMPLING
Recall that d exp , d π traditionally refer to discounted state-action distributions.
That is, sampling from them is equivalent to ﬁrst sampling a trajectory ( s 0 , a 0 , s 1 , a 1 , . . .
, s T ) and then sampling a time index t from a geometric distribution Geom(1 − γ ) (appropriately handling samples that are beyond T ).
This means that samples far into the trajectory do not contribute much to the objective.
To remedy this, we propose treating every state in a trajectory as an ‘initial state.’
That is, we consider a single environment trajectory ( s 0 , a 0 , s 1 , a 1 , . . .
, s T ) as T distinct virtual trajectories { ( s t , a t , s t +1 , a t +1 , . . .
, s T ) } T − 1 t =0 .
We apply this to both d exp and d π , so that not only does it increase the diversity of samples from d exp , but it also expands the initial state distribution p 0 ( · ) to encompass every state in a trajectory.
We note that this does not affect the optimality of the objective with respect to π , since in Markovian environments an expert policy should be expert regardless of the state at which it starts (Puterman, 2014).
R ELATED W ORK
In recent years, the development of Adversarial Imitation Learning has been mostly focused on on-policy algorithms.
After Ho & Ermon (2016) proposed GAIL to perform imitation learning via adversarial training, a number of extensions has been introduced.
Many of these applications of the
AIL framework (Li et al., 2017; Hausman et al., 2017; Fu et al., 2017) maintain the same form of distribution ratio estimation as GAIL which necessitates on-policy samples.
In contrast, our work presents an off-policy formulation of the same objective.
Although several works have attempted to apply the AIL framework to off-policy settings, these previous approaches are markedly different from our own.
For example, Kostrikov et al.
(2019) proposed to train the discriminator in the GAN-like AIL objective using samples from a replay buffer instead of samples from a policy.
This changes the distribution ratio estimation to measure a divergence between the expert and the replay.
Although we introduce a controllable parameter α for incorporating samples from the replay buffer into the data distribution objective, we note that in practice we use a very small α = 0 . 1 .
Furthermore, by using samples from the replay buffer in both terms of the objective as opposed to just one, the global optimality of the expert policy is not affected.
The off-policy formulation of the KL-divergence we derive is motivated by similar techniques in
DualDICE (Nachum et al., 2019b).
Still, our use of these techniques provides several novelties.
First, Nachum et al.
(2019b) only use the divergence formulation for data distribution estimation (which is used for off-policy evaluation), assuming a ﬁxed policy.
We use the formulation for learning a policy to minimize the divergence directly.
Moreover, previous works have only applied these derivations to the f -divergence form of the KL-divergence, while we are the ﬁrst to utilize the Donsker-Varadhan form.
Anecdotally in our initial experiments, we found that using the f -divergence form leads to poor performance.
We note that our proposed objective follows a form similar to REPS (Peters et al., 2010), which also utilizes a log-average-exp term.
However, policy and value learning in REPS are performed via a bi-level optimization (i.e., the policy is learned with respect to a different objective), which is distinct from our algorithm, which trains values and policy with respect to the same objective.
Our proposed ValueDICE is also signiﬁcant for being able to incorporate arbitrary (non-expert) data into its learning.
E XPERIMENTS
We evaluate ValueDICE in a variety of settings, starting with a simple synthetic task before continuing to an evaluation on a suite of MuJoCo benchmarks.
7. R ING MDP
We begin by analyzing the behavior of ValueDICE on a simple synthetic
MDP (Figure 1).
The states of the MDP are organized in a ring.
At each state, two actions are possible: move clockwise or counter-clockwise.
We ﬁrst look at the performance of ValueDICE in a situation where the expert data is sparse and does not cover all states and actions.
Speciﬁcally, we provide expert demonstra- Sparse Expert ValueDICE Policy Stoch
Expert
ValueDICE on Stoch Data [Figure]
[Figure] [Figure] tions which cover only states 0,
[Figure]
and 2 (see Figure 1 left).
While the problem of recovering the true (unknown) expert is ill-deﬁned, it is still possible to ﬁnd a policy which recovers close to the same occupancies.
Indeed, this is the policy found by ValueDICE, which chooses the appropriate actions at each state to optimally reach states 1 and 2 (and alternating between states 1 and 2 when at these states).
In many practical scenarios, this is the desired outcome – if the imitating policy somehow encounters a situation which deviates from the expert demonstrations, we would like it to return to the expert behavior as fast as possible.
Notably, a technique like behavioral cloning would fail to learn this optimal policy, since its learning is only based on observed expert data.
We also analyzed the behavior of ValueDICE with a stochastic expert (Figure 1 right).
By using a synthetic MDP, we are able to measure the divergence D KL ( d π || d exp ) during training.
As expected, we ﬁnd that this divergence decreases during ValueDICE training.
7. M U J O C O B ENCHMARKS
We compare ValueDICE against Discriminator-Actor-Critic (DAC)
(Kostrikov et al., 2019), which is the state-of-the-art in sample-efﬁcient adversarial imitation learning, as well as GAIL (Ho &
Ermon, 2016).
We evaluate the algorithms on the standard MuJoCo environments using expert demonstrations from Ho & Ermon (2016).
We plot the average returns for the learned policies [Figure] (using a mean action for sampling)
every 1000 environment steps using 10 episodes.
We perform this procedure for 10 different seeds and compute means and standard deviations (see Fig.
and 3, we visualize a half of standard deviation on these plots).
We present the extremely low-data regime ﬁrst.
In Figure 2 we present the results of the imitation learning algorithms given only a single expert trajectory.
We ﬁnd that ValueDICE performs similar or better than DAC an all tasks, with the exception of Walker2d where it converges to a slightly worse policy.
Notably, in this low-data regime, behavioral cloning (BC) usually cannot recover the expert policy.
We also present the results of these algorithms on a larger number of expert demonstrations (Figure 3).
We continue to observe strong performance of ValueDICE as well as faster convergence on all tasks.
It is worth mentioning that in this large-data regime, Behavior Cloning can recover the expert performance as well.
In all of these scenarios, GAIL is too sample-inefﬁcient to make any progress.
C ONCLUSION
We introduced ValueDICE, an algorithm for imitation learning that outperforms the state-of-the-art on standard MuJoCo tasks.
In contrast to other algorithms for off-policy imitation learning, the algorithm introduced in this paper performs robust divergence minimization in a principled off-policy manner and a strong theoretical framework.
To the best of our knowledge, this is also the ﬁrst algorithm for adversarial imitation learning that omits learning or deﬁning rewards explicitly and directly learns a Q-function in the distribution ratio objective directly.
We demonstrate the robustness of ValueDICE in a challenging synthetic tabular MDP environment, as well as on standard MuJoCo continuous control benchmark environments, and we show increased performance over baselines in both the low and high data regimes.
A I MPLEMENTATION D ETAILS
All algorithms use networks with an MLP architecture with 2 hidden layers and 256 hidden units.
For discriminators, critic, ν we use Adam optimizer with learning rate 10 − 3 while for the actors we use the learning rate of 10 − 5 .
For the discriminator and ν networks we use gradient penalties from Gulrajani et al.
We also regularize the actor network with the orthogonal regularization (Brock et al., 2018) with a coefﬁcient 10 − 4 .
Also we perform 4 updates per 1 environment step.
We handle absorbing states of the environments similarly to Kostrikov et al.
B A LGORITHMS
In this section, we present pseudocode for the imitation learning algorithms based on DualDICE.
Algorithm 1 ValueDICE
Input : expert replay buffer R E
Initialize replay buffer R ← ∅ for n = 1 , . .
, do
Sample ( s, a, s (cid:48) ) with π θ
Add ( s, a, s (cid:48) ) to the replay buffer R { ( s ( i ) , a ( i ) , s (cid:48) ( i ) ) }
Bi =1 ∼ R (cid:46)
Geometric sampling { ( s ( i ) 0 , s ( i ) E , a ( i ) E , s (cid:48)
( i ) E ) }
Bi =1 ∼ R E (cid:46)
Geometric sampling, s ( i ) 0 is a starting episode state for s ( i ) E a ( i ) 0 ∼ π θ ( ·| s ( i ) 0 ) , for i = 1 , . . .
, B a (cid:48)
( i ) ∼ π θ ( ·| s (cid:48)
( i ) ) , for i
= 1 , . .
, B a (cid:48)
( i ) E ∼ π θ ( ·| s (cid:48)
( i ) E ) , for i
= 1 , . .
B
Compute loss on expert data: ˆ J log = log( 1 B (cid:80) Bi
=1 ((1 − α ) e ν ψ ( s ( i ) E ,a ( i ) E ) − γν ψ ( s (cid:48)
( i ) E ,a (cid:48)
( i ) E )
+ αe ν ψ ( s ( i ) ,a ( i ) )
− γν ψ ( s (cid:48)
( i ) ,a (cid:48)
( i ) ) ))
Compute loss on the replay buffer: ˆ [Equation] linear = [Equation]
[Equation] γ ) [Equation] ψ ( s ( i ) 0 , a ( i ) 0 ) + α ( ν ψ ( s ( i ) , a ( i ) ) − γν ψ ( s (cid:48)
( i ) , a (cid:48)
( i ) )))
Update ψ ← ψ − η ν ∇ ψ ( ˆ J log − ˆ J linear )
Update θ ← ψ + η π ∇ θ ( ˆ J log − ˆ J linear ) end for
C A DDITIONAL EXPERIMENTS
We also compared ValueDICE with behavioral cloning in the ofﬂine regime, when we sample no additional transitions from the learning environment (see Figure 4).
Even given only ofﬂine data,
ValueDICE outperforms behavioral cloning.
For behavioral cloning we used the same regularization as for actor training in ValueDICE.
[Figure]