O FFLINE
I MITATION L EARNING BY C ONTROLLING
THE E FFECTIVE P
LANNING
H ORIZON
Anonymous authors
Paper under double-blind review
A BSTRACT
In offline imitation learning (IL), we generally assume only a handful of expert trajectories and a supplementary offline dataset from suboptimal behaviors to learn the expert policy.
While it is now common to minimize the divergence between state-action visitation distributions so that the agent also considers the future consequences of an action, a sampling error in an offline dataset may lead to erroneous estimates of state-action visitations in the offline case.
In this paper, we investigate the effect of controlling the effective planning horizon (i.e., reducing the discount factor) as opposed to imposing an explicit regularizer, as previously studied.
Unfortunately, it turns out that the existing algorithms suffer from magnified approximation errors when the effective planning horizon is shortened, which results in a significant degradation in performance.
We analyze the main cause of the problem and provide the right remedies to correct the algorithm.
We show that the corrected algorithm improves on popular imitation learning benchmarks by controlling the effective planning horizon rather than an explicit regularization.
I NTRODUCTION
Imitation learning (IL) is one of the sequential decision making problem settings that aims to solve a task from expert demonstrations instead of explicit notions of utility (Pomerleau, 1991; Ng et al., 2000).
In a standard setting of IL, the agent is only given with a few number of expert trajectories, and it performs the imitation of the expert by interacting with the environment.
We denote this setting as online imitation learning since it is possible to query the consequence of action during the learning with the online interactions.
On the other hand, it has been recently questioned about the practicality of online interactions, and a setting called offline imitation learning that aims to learn from supplementary suboptimal dataset instead of interactions was proposed (Kim et al., 2021; 2022;
Ma et al., 2022).
As a basic approach for IL, one can come up with behavior cloning (BC) (Pomerleau, 1991), which treats the problem as a supervised learning to map state to action based on expert demonstrations.
However, BC does not make a use of given information other than expert demonstrations, and suffers from compounding error by covariate shift as it gets off track (Ross et al., 2011).
To address this issue, previous studies (Ho & Ermon, 2016; Kostrikov et al., 2018; 2019) have proposed to imitate an expert policy by matching state-action visitation distributions , which can be estimated by interacting with the environment.
State-action visitation matching allows the agent to learn how to return to the preferred states, and this approach has been successful in a wide range of domains of online IL.
Interestingly, in recent offline IL studies, it is often found that BC is quite competitive (Li et al., 2022; Kim et al., 2021), and sometimes it even performs better than visitation distribution matching algorithms depending on the experiment settings.
This is because there is an inherent error in estimating visitation distributions from a finite dataset in offline cases, and BC, which completely ignores the dynamics information, does not suffer from this problem.
Therefore, unlike the online cases where we can sample from true environment dynamics, BC or visitation distribution matching cannot always be the superior choice over the other in offline cases.
Motivated by this, we consider controlling the effective planning horizon, i.e. the discount factor γ , for the offline imitation learning problems.
The discount factor implies how important the future is compared to the present.
For visitation distribution matching algorithms, it can be interpreted as the amount of state-action distribution difference between the learned and expert policy we allow in the future compared to now.
While it has been overlooked in the online IL studies since we recover more accurate policy as we less discount the future, it will be advantageous to use an optimal discount factor in offline IL that makes the best trade-offs: using small γ to shorten the effective planning horizon helps to become robust to the errors in the inferred dynamics (Janner et al., 2019) (which also allows avoiding explicit regularization for robustness as in previous studies), whereas using large γ makes an agent less prone to the compounding errors by considering longer consequences of training.
In this paper, we start by formally analyzing the performance trade-off of a discount factor in offline
IL.
It turns out that, however, recently proposed offline IL algorithms show pathological behavior when the discount factor is lowered and naively controlling the discount factor does not lead to a performance gain.
It is because the error from the approximations they share depends on the discount factor, and it becomes revealed and maximized as we lower γ .
To this end, we propose a simple technique called Inverse Geometric Initial state sampling (IGI) to address the problem.
We show that IGI enables the algorithm to properly learn in the low discount factor settings, and by tuning the discount factor, the offline IL agent with IGI outperforms previous state-of-the-art algorithms with explicit regularizations.
B ACKGROUND 2. M ARKOV D ECISION P ROCESS (MDP)
We consider environments modeled as a Markov Decision Process (MDP), which is defined by a tuple M =
( S , A , P, p 0 , R, γ ).
Here, S is the state space, and A is the action space.
P and prepresent the dynamics and initial state distribution, respectively.
R is the reward function which is assumed to be bounded in [0 , R max ] , and γ ∈ [0 , 1) is the discount factor.
A policy π ( ·| s ) determines the probability of agent’s action in a state s .
The goal of an agent is to maximize the sum of reward discounted by γ .
The state-action visitation distribution d π ( s, a ) , which is induced by a policy π , is defined as
[Equation]
( s, a ) =
(1 − γ ) ∞ (cid:88) t
= γ t Pr ( s t =
s, a t = a | s 0 ∼ p 0 ( · ) , s t ∼ P ( ·| s t − 1 , a t − 1 ) , a t ∼ π ( ·| s t )) .
We consider an offline imitation learning problem where a small number of expert demonstrations ( D E ) and a suboptimal dataset ( D O ) consisting of transitions ( s, a, s ′ ) are given.
We denote total dataset D D
= D E ∪ D O as a union of two datasets.
Especially, we assume that D E is sampled by following the expert policy π E in underlying MDP M .
We denote empirical distribution of D E and
D D as E ( s, a ) and D ( s, a ) respectively.
On the other hand, we denote the state-action distribution of π E as d E ( s, a ) .
2. I MITATION L EARNING VIA STATE - ACTION VISITATION MATCHING
Imitation learning (IL) aims to train an agent that mimics the expert based on the expert demonstrations.
In addition to demonstrations, supplementary information on environment dynamics is given, by directly interacting with environment (online IL) or by a dataset of suboptimal behaviors (offline
IL).
One simplest and most popular approach to imitate an expert is behavior cloning (BC), which treats IL as a supervised learning problem, ignoring any of supplementary information (Pomerleau, 1991).
The objective of BC can be represented as follows:
[Equation]
However, it is well known that the error of obtained policy induced by covariate shift compounds over time, and leads to an eventual failure unless D E is large enough (Ross et al., 2011).
To address the weakness of BC, a state-action visitation distribution matching objective is now widely adopted, which minimizes the divergence between d π and d E (Kostrikov et al., 2019; Ke et al., 2020).
The distribution matching objective can be presented as follows:
[Equation]
Note that the objective can be alternatively interpreted as a reinforcement learning (RL) problem:
[Equation]
We can observe that the objective (3) can be seen as a RL problem with a reward r = log d E d π .
When online interactions are allowed, we can sample from d π by executing policy π , and Ho & Ermon (2016) proposed to estimate the newly defined reward by learning the discriminator using samples from d E and d π :
[Equation]
where the optimal discriminator c ∗ ( s, a ) can be used to recover the reward as log c ∗ ( s, a ) − log (1 − c ∗ ( s, a ))
=
log d E ( s,a ) d π ( s,a ) .
On the contrary, in a fully offline setting we consider, the interaction with the environment to receive samples from d π is not allowed.
A common choice in this case is to train a discriminator that can distinguish between the expert demonstrations and a supplementary suboptimal dataset (Kim et al., 2021; 2022; Ma et al., 2022), and is explained in detail in
Section 3.1.
O FFLINE IL VIA STATE - ACTION VISITATION MATCHING
In this section, we first derive a practical objective that can be used to optimize the distribution matching objective (2) in the offline IL case.
After the derivation of an objective that is used throughout the paper, we provide an analysis of the effect of the discount factor on the error bound of offline
IL.
In particular, we show that there is a trade-off when controlling the discount factor, unlike the online case.
3. D ERIVATION OF AN OFFLINE IL OBJECTIVE
We start from the widely used visitation distribution matching objective (2) as suggested in (Ho & Ermon, 2016; Torabi et al., 2018).
Rather than optimizing for the policy π , we follow the recent approaches that optimize directly for the visitation distribution d π (Lee et al., 2021;
Kim et al., 2021).
To optimize for d π , we need to ensure that the solution we get is a valid visitation distribution.
By making Bellman flow constraints and normalization constraints explicit, we have:
[Equation]
where B ∗ d π ( s, a ) = (cid:80) a d π ( s, a ) is a marginalization and P ∗ d π ( s, a )
= (cid:80) ¯ s, ¯ a P ( s | ¯ s, ¯ a )
d π (¯ s, ¯ a ) is an expectation over the previous state-actions.
It can be easily seen that the following equalities hold by interchanging the order of summations:
[Equation]
The Lagrangian of the constrained problem (5-7)
[Equation]
where λ and ν ( s ) are Lagrange multipliers.
Based on the relationships shown above, we can rewrite
Equation (8), where the summation is computed w.r.t.
d π ( s, a ) :
[Equation]
In the second term of Equation (9), to derive a practical algorithm, we want to make an expectation to be performed on the total dataset distribution D ( s, a ) .
For this, we use the importance sampling with importance weight ζ ( s, a ) =
d π ( s,a ) D ( s,a ) .
Then, the objective becomes
[Equation]
where A ν ( s, a ) = γ P ν ( s ) − B ν ( s ) + log d E ( s,a ) D ( s,a ) , which can be interpreted as an advantage when the action a is executed in the state s if we define the reward as a log ratio of distributions r ( s, a ) = log d E ( s,a ) D ( s,a ) .
Such a log ratio can be obtained by training a discriminator c ( s, a ) through the following objective:
[Equation]
such that the optimal discriminator c ∗ ( s, a ) satisfies
[Equation]
c ∗ ( s,a ) 1 − c ∗ ( s,a ) (cid:17) .
The advantage
A ν ( s, a ) can then be estimated based on the discriminator c being trained.
We can further simplify the optimization problem (10) by noting that the strong duality holds, which enables us to change the order of optimization without affecting the optimal value.
Closedform solution of maximization of Equation (10) can be obtained by solving first-order optimality condition ∂
L ( ζ ∗ ,ν,λ ) ∂ζ = 0 , which gives ζ ∗ =
exp ( A ν ( s, a ) + λ − 1) .
Similarly, λ ∗ = − log[ E ( s,a ) ∼ D exp
( A ν ( s, a ) − 1)] can be obtained.
By substituting in ζ ∗ and λ ∗ to (10), we derive the final objective:
[Equation]
and optimize for ν .
Based on the optimized ν ∗ , we can obtain ζ ∗ = softmax ( A ν ( s, a ) − 1) .
Using this, policy can be extracted through the following objective:
[Equation]
Note that the derived objective (12) can be seen as a special case of DemoDICE (Kim et al., 2021) and SMODICE (Ma et al., 2022), and it can be recovered by setting α = 0 in DemoDICE (no explicit regularization toward suboptimal dataset) or by setting f -divergence to be KL divergence in
SMODICE.
While the algorithm in this paper is based on KL-divergence minimization of visitation distributions, note that it can be trivially extended to any f -divergence minimization case.
3. T RADE - OFF BETWEEN TWO DISTINCT EFFECTS IN OFFLINE IL BY DISCOUNT FACTOR
Now we show that there is indeed a trade-off when controlling the discount factor γ in the offline setting.
Intuitively, an offline agent can only receive supplementary information about environment dynamics through limited finite demonstrations, and the error of estimated dynamics is inevitable.
Therefore, we can expect that learning with a long planning horizon increases the risk of compounding estimation error.
On the other hand, the trained agent is typically evaluated in a non-discounted environment by measuring the average reward it gets, and excessively lowering the discount factor for training will make a large train-test discrepancy.
The theorem below is a formal analysis backing up the argument, showing an error bound of the imitated policy with respect to a discount factor.
Theorem 1.
Let P an underlying transition dynamics and (cid:98) P an estimated transition dynamics.
γ is a discount factor used for evaluating the policy and ˆ γ is a discount factor used for training the policy where ˆ γ ≤ γ .
Let d π P,γ a state-action visitation distribution induced by π under the dynamics
[Figure]
P using γ , and d EP,γ a state-action visitation distribution induced by π E under the dynamics P using γ . Assume that the reward is bounded in [0 , R max ] .
Then, the error bound for imitated policy π is
[Equation] [Equation]
where ϵ P = E d π (cid:98) P, ˆ γ
[Equation]
(cid:105) + E d E (cid:98) P, ˆ γ (cid:104) D TV ( (cid:98) P ∥ P ) (cid:105) , and ϵ π =
E d π (cid:98) P, ˆ γ (cid:2)
D TV ( π ∥ π E ) (cid:3) .
The proof is deferred to Appendix A.
The first term reflects the train-test discrepancy caused by optimizing policy using ˆ γ instead of the discount factor γ that is used for evaluation, i.e. in the LHS of inequality (14), and is minimized as we choose ˆ γ close to γ .
The second term represents the effect of the error on estimating dynamics ϵ P .
It can be observed that this term becomes smaller if the effective planning horizon is shortened by using a lower ˆ γ .
From the definition of ϵ P , it is the accumulation of step-wise dynamics estimation error over trajectories of both π and π E , and there will always be an irreducible amount of ϵ P in the offline case.
The last term stands for the policy difference between π and π E over the estimated dynamics and will be minimized if imitation learning is done properly to maximize the objective (2).
In short, the error of the imitated policy will be mainly dependent on the first two terms of (14), which have opposite dependencies to the discount factor for training ˆ γ .
This result supports our argument about the trade-off between two distinct effects by discount factor during the offline training and shows the possibility that an offline IL algorithm may benefit from choosing ˆ γ that is smaller than γ that we use to evaluate.
C ONTROLLING THE DISCOUNT FACTOR IN OFFLINE IMITATION LEARNING
As suggested by Theorem 1, we expect that there would be a benefit from controlling the discount factor γ that is used for training.
The algorithm derivation is based on a visitation distribution matching (5-12), as well as those of previous studies, which appears independent of and applicable to any choice of the discount factor in principle.
However, it turns out that naively lowering γ in previously suggested visitation distribution matching offline IL algorithms (Kim et al., 2021; Ma et al., 2022) only results in a monotonically decreasing performance, as shown in Figure 1.
The performance reduction of using lower discount factors is significant, and it seems pointless to choose any γ below 0 .
99 in contrast to our analysis.
In this section, we analyze the cause of this pathological behavior and propose a simple scheme that can alleviate it.
[Figure]
g
[Figure]
a
[Figure]
a
[Figure]
g sa 1 = θ a
[Figure]
a 2 = 1 − θ
[Figure]
4. D ISTRIBUTION MISMATCH IN TRAINING A DISCRIMINATOR
The main problem lies in the fact that while the visitation distribution matching objective (5) requires the estimation of d E ( s, a ) , which is a discounted state-action distribution induced by the expert policy π E , a discriminator c ( s, a ) is learned to discriminate the empirical distributions of the datasets, i.e. between D ( s, a ) and E ( s, a ) .
Since the empirical distributions are undiscounted, for a state-action pair sample ( s t , a t ) at timestep t , d E ( s, a ) would weight γ t on this sample compared to
E ( s, a ) that does not weight on this sample, making the two distributions significantly different as we have smaller γ .
To observe the consequence of having a discriminator trained on empirical distributions, we assume that the reward is represented with an optimal discriminator such that r ( s, a ) =
log E ( s,a ) D ( s,a ) .
As a result of applying this to objective (10), it becomes:
[Equation]
By taking the derivation steps backward, we can confirm that using the discriminator trained on empirical distributions is equivalent to solving the visitation distribution matching objective (5-7) except for the objective:
[Equation]
That is, by training a discriminator trained on empirical distributions, we actually have matched d π ( s, a ) and E ( s, a ) to get a policy π .
Note that there is a one-to-one correspondence between the state-action visitation distributions and policies (Ho & Ermon, 2016).
It implies that unless
E ( s, a ) =
d E ( s, a ) , even if E ( s, a ) is a valid visitation distribution that satisfies Bellman flow constraints, the policy inferred by matching E ( s, a ) will be different from d E , and thus, π ∗ ̸ = π E .
In most cases, E ( s, a ) would not be a valid visitation distribution, and a policy π that gives
D KL ( d π ( s, a ) ∥
E ( s, a ))
= 0 would not exist in general.
While this discrepancy between E and d E is negligible when γ is large and the length of trajectories in D E is long enough, large error on the imitated policy can be incurred in other cases.
We give a simple toy example below.
Illustrative example Assume an infinite horizon MDP with 2 states and an absorbing state, and with 2 actions as shown in Figure 2.
The expert policy π E has 0 .
6 probability of doing a 1 in s 0 , 0 .
4 of doing a 2 in s 0 , and 1 of doing a 1 in the other states.
Assuming that trajectories are sampled up to timestep 2, the expert demonstration D E will be consisting of two kinds of trajectories τ 1 = ( s 0 , a 2 , g, a 1 , g, a 1 ) and τ 2 = ( s 0 , a 1 , s 1 , a 1 , g, a 1 ) with the ratio of 6 : 4 .
E ( s, a ) can be calculated by
[Table]
accordingly.
The learned policy is parametrized as π ( a 1 | s 0 )
= θ , π ( a 2 | s 0 )
= 1 − θ , and π ( a 1 |· )
= 1 in other states.
By computing d π based on the above, we can express D KL ( d π ∥ E ) in terms of θ and γ .
Then, we can minimize D KL to get θ ∗ , and see if we can recover θ ∗ = 0 .
In this toy example, it turns out that θ ̸ = 0 .
unless γ = 0 .
5 (detailed
[Figure]
o
[Figure]
li z
[Figure]
v i n g A v e r a g e
[Figure]
IGI ( D )
[Figure]
=
[Figure]
=
[Figure] [Figure]
=0. =0.
=0. =0.
derivations can be found in Appendix B).
It can be seen in this example that we cannot match two distributions d π ( s, a ) =
d π ( s ) π ( a | s ) , E ( s, a ) =
E ( s ) π E ( a | s ) in general, and by minimizing the distribution between state visitation distributions, optimized π will be different to π E .
4. I NVERSE GEOMETRIC INITIAL STATE SAMPLING
As shown in the previous subsection, we cannot recover π = π E unless d E = E , where E is the distribution that the discriminator is trained on.
One straightforward solution to this problem is to train the discriminator to distinguish d E and E , eliminating the root cause of the problem.
We can simulate the sampling from d E by weighting each sample in the dataset with respect to their timesteps when they were sampled.
For example, we can first sample the timestep t from a geometric distribution Geom (1 − γ ) and then sampling ( s t , a t ) pair that had been sampled at t .
This procedure will approximate the sampling from d E sufficiently well, given that the trajectories stored in D E is long enough.
However, training the discriminator to distinguish d E gives rise to another problem; if we use lower γ , the samples from d E will mostly consist of early timestep samples of D E , since the probability assigned decreases exponentially over timesteps.
This results in a significant under-usage of dataset
D E by throwing away samples of later timesteps, and significantly deteriorates the performance.
We need a way to ensure that we get π = π E while not hurting the effective number of data.
To this end, we propose to devise a different initial state distribution other than the actual p 0 to satisfy both conditions.
Note that even if we change the initial distribution, the optimality of policies are not affected as mentioned in Kostrikov et al.
d π and d E that π
and π E induces will be different, but one-to-one correspondence does still hold that matching visitation distributions ensure π = π E with arbitrary initial state distribution.
In particular, if we use an initial distribution that makes d E = E , i.e. flattens the visitation distribution to look like an undiscounted distribution, recovering π = π E will be guaranteed even if we use the discriminator that learns from the empirical distributions.
Note that sampling from d E corresponds to sample timestep t from Geom (1 − γ ) and choose ( s t , a t ) that had been sampled at t .
Hence, if we use a modified initial distribution ˜ p 0 that samples from all the timesteps in D E with weights inversely proportional to Geom (1 − γ ) , the resultant ˜
d E will have uniform weights regarding timesteps and therefore ˜ d E = E .
We call this modified initial state sampling method as Inverse Geometric Initial state sampling (IGI).
In practice, there are a few more things to consider determining the weightings for each possible initial state samples; e.g. due to datasets being truncated at certain timesteps and trajectories that are terminated early.
Nevertheless, it is possible to set up a system of linear equations to obtain the required weights for each sample to ensure ˜ d E = E .
Details on how to set up a linear system is shown in Appendix C.
Unfortunately, it turns out that the exact IGI that ensures ˜ d E = E can be too restrictive.
Following the recent practices of using a single expert policy for D E in evaluating imitation performances, only the state-action pairs in the single trajectory can be sampled as an initial state, which cripples the diversity.
Furthermore, we found that sampling the initial states only from D E also limited the opportunity of our agent to learn how to act outside D E , incurring the covariate shift and compounding error problems that BC has suffered.
On the other hand, using IGI to sample from the total dataset
D D did not exhibit such a problem, while alleviating the discrepancy between ˜ d E and E .
In Figure 3, we compare the performance of two options: IGI( D E ) and IGI( D D ).
It is clearly visible that the imitation performance of IGI( D E ) is limited, whereas IGI( D D ) performs robustly over a wide set of γ s. As a consequence, we used IGI with samples from D D in the following experiments.
R ELATED WORK
Controlling the effective planning horizon In reinforcement learning (RL), the discount factor effectively controls the amount of forward planning the agent considers making a decision.
Petrik & Scherrer (2008) has shown that the approximation error bound incurred in an inaccurate model can be tightened by using a low discount factor.
Jiang et al.
(2015) has shown the similar result, but they have analyzed in terms of the complexity of the class of policies.
They show that there is a trade-off between the complexity of the policy space and the error due to the approximated model according to the effective planning horizon, which can controlled by a discount factor.
Recently,
Hu et al.
(2022) analyzed the role of discount factor from the perspective of offline RL.
In addition to the trade-off relationship between optimality and sample efficiency, they show theoretically and empirically that the low discount factor can also be seen as a model-based pessimism.
On the other hand, there has not been a study on the effect of discount factor on offline imitation learning, up to our knowledge.
Offline imitation learning by leveraging the duality of RL Imitation learning (IL) aims to mimic the expert policy by using the expert demonstrations and the online interactions, but offline supplementary dataset is given instead in offline case.
Recently, there have been a large amount of literature that leverages the duality in RL to develop a novel algorithm estimating quantities related to visitation distributions.
In a field of offline IL, DemoDICE (Kim et al., 2021) proposed to solve it by deriving the dual of visitation distribution matching problem with an explicit regularizer minimizing the f -divergence between the visitation distribution induced by policy and the supplement dataset distribution.
SMODICE (Ma et al., 2022) considers the IfO problem, which solves IL with state-only demonstrations by expert, and proposes a versatile offline IL algorithm by leveraging f -divergence and Fenchel duality instead of KL used in (Kim et al., 2021).
E XPERIMENTS
We evaluate the performance of our algorithm in both discrete and continuous MDP.
We first show the trade-off effect by the discount factor in finite-discrete MDP, then evaluate the imitation performance of our algorithm using IGI in continuous MDP.
6. F INITE AND D ISCRETE MDP
In finite-discrete MDP, we empirically show the trade-off effect by the discount factor in the Random
MDP.
This environment generates the finite and discrete MDP randomly.
We follow the environment configuration of Lee et al.
In this experiment setting, we can see that the performance can be optimized by choosing a lower discount factor.
Details of the experiment in finite-discrete MDP and its results with various settings are shown in Appendix E.1 due to the lack of space.
6. C ONTINUOUS MDP
For continuous MDP, we evaluate the imitation performance of our algorithm using IGI in the MuJoCo continuous control environment (Todorov et al., 2012): HalfCheetah-v2 , Hopper-v2 ,
Walker2d-v2 and Ant-v2 .
The dataset used for learning is obtained from the D4RL datasets (Fu et al., 2020).
We sampled expert trajectories and random trajectories from the expert dataset and random dataset of D4RL benchmark respectively.
We use a single expert trajectory for D E , and our suboptimal dataset D O is constructed as a union of expert trajectories and random-policy trajectories with specific ratios.
[Figure]
We conduct experiments by changing the ratio between the expert data and the random-policy data of the suboptimal dataset.
We call suboptimal dataset consisting of 400 expert trajectories andrandom-policy trajectories as D1, 100:800 as D2, 50:800 as D3.
We compare our algorithm using
IGI with other offline IL algorithms, DemoDICE (Kim et al., 2021), SMODICE (Ma et al., 2022), and BC.
Performance of other algorithms is measured based on the official hyperparameters made public by authors without any modification.
For BC, we show the result of learning with D D , which has the best performance among the various combination of the data to be cloned.
All experiment results in this paper are averaged over 3 seeds and normalized so that 0 corresponds to the average score of the random-policy dataset, and 100 corresponds to the average score of the expert policy dataset.
In order to demonstrate the robustness of our algorithm over different γ s, we show the results by applying various discount factor.
We use discount factor for γ ∈ { 0 . 99 , 0 . 95 , 0 . 9 , 0 . 8 , 0 . 7 , 0 . 6 } .
We also tested every algorithm with γ below 0.6, but due to exploding gradient, it was not possible to run DemoDICE and SMODICE for lower γ s below 0 .
Therefore, we report only the result of γ s mentioned above.
The results of baselines on different γ s are shown in Figure 4.
It can be seen that the introduced method IGI makes the algorithm perform well regardless of γ .
Additional results on other environments and D1, D3-ratio datasets are shown in the Appendix E.2.
In addition, the experiment in Figure 4 has too large enough size of the dataset to make ϵ P in the second term of Theorem 1 negligible, and the effect of controlling the discount factor is not clearly visible.
Therefore, we make additional experiments by lowering the amount of the dataset about 10 times less (40 expert trajectories and 80 random-policy trajectories) than D1 setting to show the effect of controlling the discount factor clearly.
Relevant results are shown in Appendix F.1.
Also, the pseudocode and the hyperparameters used to demonstrate IGI can be found in Appendix D.1.
C ONCLUSION
In this paper, we analyze the effect of controlling the discount factor on offline IL and motivate that the discount factor can take a role of a regularizer to prevent the sampling error of the supplementary dataset from hurting the performance.
We show that the previously suggested imitation learning algorithms that utilize discriminators and a visitation distribution matching objective suffer from the discrepancy between the visitation distribution and the empirical distribution when low discount factor γ is applied.
To this end, we proposed Inverse Geometric Initial state sampling (IGI), which uses the whole dataset with the weighting inversely proportional to the geometric distribution, to alleviate the problem that we cannot recover the expert policy π E .
We show that our approach shows stable and competitive performance regardless of the discount factor compared to other visitation distribution matching algorithms with explicit regularization.
A P ROOF OF T HEOREM
The proof of Theorem 1 is based on the analysis in previous works (Petrik & Scherrer, 2008; Jiang et al., 2015; Lee et al., 2020;
Xu et al., 2020).
We provide three Lemmas first and use them to prove the Theorem 1.
To prove three Lemmas, we define several notations.
P and ˆ P denote a matrix of the underlying transition dynamics and estimated transition dynamics, respectively.
E d πP,γ [ r ( s, a )] denotes the expected total reward discounted by γ for the policy π under transition dynamics P .
dand r denote a vector of the initial state probability and the reward, respectively.
For r and P , if the π is at superscript, it means “following policy π ”.
Lemma 1.
For any MDP with bounded rewards | r ( s, a ) | ≤ R max , for all π : S → A and ˆ γ ≤ γ ,
[Equation]
Proof.
[Equation]
We define one more notation for Lemma 2 and 3.
d πP,γ denotes a vector of the marginal state probability induced by π under transition dynamics P using γ .
Lemma 2.
We can bound the difference of the evaluations of policy π on two different MDPs with bounded rewards | r ( s, a ) | ≤ R max as
[Equation]
Proof.
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
Lemma 3.
We can bound the difference of the evaluations of two policies π and µ with bounded rewards | r ( s, a ) | ≤ R max as
[Equation]
Proof.
We can expand the difference of the evaluations of two policies like
[Equation]
The first term is bounded as
[Equation] [Equation]
Using the process (18-22)
[Equation]
the second term is bounded as
[Equation]
Combining (25) and (29)
[Equation]
we have the proof of Lemma 3.
Theorem 1.
Let P an underlying transition dynamics and (cid:98) P an estimated transition dynamics.
γ is a discount factor used for evaluating the policy and ˆ γ is a discount factor used for training the policy where ˆ γ ≤ γ .
Let d πP,γ a state-action visitation distribution induced by π under the dynamics
P using γ
[Equation]
and d EP,γ a state-action visitation distribution induced by π E under the dynamics P using γ .
Assume that the reward is bounded in [0 , R max ] .
Then, the error bound for imitated policy π is
[Equation]
where ϵ P
[Equation]
Proof.
The error bound for imitated policy π can be expanded as
[Equation]
The first and last terms are bounded by Lemma 1, and the second and fourth terms are bounded by
Lemma 2.
Lastly, the third term is bounded by Lemma 3.
This completes the proof.
B P ROOF FOR SIMPLE MDP
In Section 4.
[Equation]
we define expert policy as
[Equation]
we said if N trajectories are sampled, the number of τ 1 and τ 2 is 0.6N and
0.4N. Since each trajectories have three ( s, a ) pairs, total number of ( s, a ) pairs is 3N. Calculating the previously defined
E ( s, a ) is as follows
[Equation]
Next, imitation policy is defined as
[Equation]
Then
[Equation]
d π ( s
[Equation]
a ) has the following values
[Equation]
Using E(s,a) and d π ( s, a )
[Equation]
D KL ( d π ∥ E ) is expressed as
[Equation]
To find a minimizer θ ∗ of above optimization, we use first-order optimality condition.
That is, ∂D KL ( d π ( s, a ) ∥ E ( s, a ))
∂θ = 0 .
We derive first derivative of D KL ( d π ( s, a )
∥ E ( s, a )) .
Result is ∂D KL ( d π ( s, a ) ∥
E ( s, a ))
[Equation]
It is hard to get the exact θ ∗ value because equation is very complicated.
Therefore, we substitute the probability value 35 , which is probability of expert policy, into θ to check whether the derivative becomes 0.
[Equation]
Since γ ∈ [0 , 1) , we exclude γ = 1 case.
Solving last equation yields γ = 0 .
In summary, if γ = 0 .
5 , the expert policy can be recovered through optimization min π D KL ( d π ( s, a )
∥ E ( s, a )) , but if other γ values are applied, it can be interpreted that the optimal policy is not the same as the expert policy.
C F INDING IGI D ISTRIBUTION
Our goal is to find the initial distribution ˜ p 0 that makes the distribution made by t
+ t geom uniform, where t 0 ∼ ˜ p 0 ( t ) and t geom ∼ Geom 1 − γ ( t | t 0 ) .
Note that Geom 1 − γ
( t | t 0 ) is conditional distribution.
Because, in general, there is a maximum timestep, | T | , for episode in the learning environment, so in order to prevent the sum of timestep exceed over | T | , Geom 1 − γ ( t | t 0 ) must be conditional distribution which considers the sampled initial timestep.
Let T =
t 0 + t geom , Then P T ( T ) is defined as
[Equation]
For the convenience of notation, we rewrite (cid:80) | T |− t 0 i = (1 − γ ) γ i as Sum ( | T | − t 0 ) .
We can express
Equation (30) in linear problem form.
To this end, it is represented as follows.
[Equation]

[Equation] [Equation]

[Equation]
(cid:125)
[Equation]
 
[Equation] [Equation]
 
[Equation]
When we actually obtain P T , we obtain it through P T ( T ) =
[Table]
Finally, solving ˜ p
= P − 1 t geom · P T gives us the ˜ p 0 ( t ) we want.
D E XPERIMENTAL D ETAILS D. A LGORITHM DETAILS
In this subsection, we provide a simple pseudocode for our algorithm using IGI.
Algorithm
Our algorithm using IGI
Require: dataset D D = D E ∪ D O , IGI distribution ˜ p 0 from Appendix C, policy network π θ with parameter θ , neural networks ν ϕ and c ψ with parameter ϕ and ψ , discount factor γ , batch size B , learning rate α
Ensure: P T D is a distribution P T from Appendix C using dataset D , and D t is an dataset D consisted of data with timestep t
Make IGI distribution ˜ p 0 using γ and P T D D while total iterations do
Sample initial timestep: { t ( i ) 0 } Bi =1 ∼ ˜ p▷ using IGI distribution
Sample uniform timestep: { t ( i ) } Bi =1 ∼ P T D D
Sample expert timestep: { t ( i ) e } Bi =1 ∼ P T D E
Sample initial state: { s ( i ) 0 } ∼ D D t ( i ) 0 , for i = 1 , ..., B
Sample total data: { ( s ( i ) , a ( i ) , s ′ ( i ) ) } ∼ D
Dt ( i ) , for i = 1 , ..., B
Sample expert data: { ( s ( i ) e , a ( i ) e , s ′
e ( i ) ) } ∼ D E t ( i ) e , for i
= 1 , ...
B
Compute discriminator loss J c (11) using { ( s ( i ) , a ( i ) ) }
Bi =1 and { ( s ( i ) e , a ( i ) e ) } Bi =
Compute critic loss J ν (12) using { s ( i ) 0 } Bi =1 and { ( s ( i ) , a ( i ) , s ′ ( i ) ) } Bi =
Compute policy loss J π (13) using { ( s ( i ) , a ( i ) , s ′ ( i ) ) }
Bi =
Update ψ ← ψ − α ∇ ψ J c
Update ϕ ← ϕ − α ∇ ϕ J ν
Update θ ← θ − α ∇ θ J π end while
To compute three loss functions in Algorithm 1, we need sampled data from d E , D, and p 0 .
This begins by sampling the timesteps for each distribution.
First of all, compute the IGI distribution using the geometric distribution Geom (1 − γ ) given a discount factor γ as we described in Appendix
C. At this time, uniform distribution P T for the total dataset D D is also used for practical performance as shown in Figure 3 of Section 4.2.
P T is defined as
[Table]
as in
Appendix C. Since IGI enables sampling for the discounted distribution uniformly regardless of the timestep, sampling from d E is replaced with sampling from P T DE .
After three types of timesteps are sampled, we need actual data corresponding to each type of timestep to use in loss functions.
We uniformly sample actual data of corresponding timesteps in each three dataset.
To compute the loss function (11) of the discriminator, uniformly sampled expert data is used to compute the first term and the total data is used to compute the second term.
In the case of the critic network, the initial data from IGI is used to compute the first term in (12) and the total data is used to compute the second term.
When we compute A ν ( s, a ) in the second term, use γν ( s ′ ( i ) ) instead of γ P ν ( s ( i ) ) and use the discriminator to compute log d E ( s,a ) D ( s,a ) as notified in the sentence below (11).
For the policy network, total data is used to compute (13).
Lastly, update the parameters of each network with a learning rate α .
Repeating this process for the total number of iterations.
D. I MPLEMENTATION DETAIL
In this subsection
[Table]
we provide the hyperparameter settings of our algorithm.
We use an absorbing state for practical implementation (Kostrikov et al., 2018).
For discriminator learning, we use
WGAN-GP (Gulrajani et al., 2017) to achieve more robust learning
[Table] [Table]
E E XPERIMENT R ESULTS E. F INITE AND D ISCRETE MDP
In this subsection
[Figure]
we provide experiment details in finite-discrete MDP and its results
[Figure]
d A
[Figure]
In finite and discrete MDP, we empirically show the trade-off effect by the discount factor in the
Random MDP.
This environment generates the finite and discrete MDP randomly.
We follow the environment configuration of Lee et al.
We construct the expert dataset D E and suboptimal dataset D O by rolling expert policy and suboptimal policy, respectively.
We characterized the expert policy as the stochastic policy based on the optimal Q (state-action) value of the randomly generated MDP.
The suboptimal policy has performance between optimal and uniformly random policy by controlling the hyperparameter ω ∈ [0 , 1] as ωV ∗ ( s 0 ) + (1 − ω )
V π unif ( s 0 ) where V ∗ and V π unif is the value function of optimal policy and uniformly random policy, respectively.
Since the MDP is finite and discrete, we can compute the discounted visitation distribution of the policy directly, which means learning the discriminator is unnecessary.
Thus, there is no need to use IGI in this environment.
We use Maximum likelihood estimation (MLE) transition dynamics ˆ P and discount factor ˆ γ for training the policy as we discussed in Theorem 1.
We use discount factor for ˆ γ ∈ { 0 . 99 , 0 . 9 , 0 . 8 , 0 . 7 , 0 . 6 , 0 .
5 , 0 . 4 , 0 . 3 , 0 . 2 , 0 . 1 } with varying the number of suboptimal trajectories for { 5 , 20 , 60 , 100 } .
The number of expert trajectory is 1 for all settings.
Figure 5 shows the policy evaluation result on the true MDP with true discount factor γ .
It shows evaluation trends so that the trade-off effect by discount factor suggested in Section 3.2 can be confirmed.
We can see that the performance can be optimized by choosing lower γ .
Furthermore, the figure shows that the performance of using large discount factors becomes better as the number of suboptimal trajectories gets larger.
E. C ONTINUOUS MDP
We provide the supplementary experiment results of Figure 4 here.
We plot the learning curve according to γ and dataset ratio.
Figure 6 represents the result in HalfCheetah-v2 environment, and Figure 7, 8, 9 is the result on the Hopper-v2 , Walker2d-v2 and Ant-v2 , respectively
[Figure]
g
[Figure]
n
[Figure]
o r m a li
z
e
d
A
v e r a g e
[Figure]
u
[Figure]
n
[Figure]
li
[Figure]
g
[Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure]
N
[Figure]
r m a li z e d A v e r a g e
R e t u r n
[Figure] [Figure]
v
[Figure]
r a g
[Figure]
u
[Figure]
li z
[Figure]
A v
[Figure]
r a g
[Figure]
t
[Figure] [Figure]
[Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure]
[Figure] [Figure]
[Figure] [Figure]
N
[Figure]
r m a li z e d A v e r a g e
R e t u r n
[Figure]
r m a li z e d A v e r a g e
R e t u r n
[Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure]
o r m a li
z
e
d
A
v e r a g e t n
[Figure]
N o r m a li z e d
A v e r a g e
R e t u r n
[Figure]
F
A DDTIONAL E XPERIMENT R ESULT F. E XPERIMENT ON S MALL D ATASET IN C ONTINUOUS MDP
[Figure]
o r m a li
z
e
d
A
v e r a g e
[Figure]
t u r n
[Figure]
We make additional experiments by lowering the amount of dataset about 10 times less (40 expert trajectories and 80 random-policy trajectories) than D1 / D2 / D3 settings.
Figure 10 shows the last evaluation of IGI with applying aforementioned setting in 4 MuJoCo environments over 3 random seeds, and green line shown in the Figure 10 is the maximum last evaluation over 3 seed of DemoDICE among different discount factors for each environment (Note that best discount factor of
DemoDICE is 0.99 for all environments).
As shown in the attached figure, we confirmed that the optimal discount factor appeared at lower than 0.99, except for the HalfCheetah-v2 environment.
To explain the reason why a different trend came out only in HalfCheetah-v2 , we inform that unlike the other environments,
HalfCheetah-v2 environment has the peculiarity that learning is stable in the offline RL problem, even when the policy is unregularized.
For example, refer to the experiment result by varying the policy regularization factor in Figure 9 of Wu et al.
(2019), HalfCheetah-v2 environment shows good performance compared to other environments even when a low regularization factor α is applied.
Combining these experimental results with the regularization effect on the discount factor in the offline RL referred to Hu et al.
(Section 3), we can expect that the performance of the learned policy can be improved when a high discount factor is used in the
HalfCheetah-v2 environment.
Eventually, the reason why Figure 10 shows a different trend only in the HalfCheetah-v2 environment is interpreted as reflecting this environmental disposition, and the error bound that we proposed in Theorem 1 is valid under general situations.
F. T EST ON VALIDITY OF T HEOREM
[Figure]
o r m
[Figure]
li z e d p o li c
y e v a l u a t
i o n
[Figure]
To check the validity of Theorem
[Figure]
we measured the magnitude of the second and third terms in finite-discrete MDP described in E.1 where ϵ P and ϵ π can be exactly computed.
Figure 11 shows the trend of two terms in (14) with different discount factors at each setting.
Results show that the third term has a much lower value than the second term.
As a result, Theorem 1 is mainly dependent on the first and second terms.
The peculiar part is that the third term increases as the suboptimal data increases.
In this situation, the proportion of expert data in the total dataset is reduced, and in that case, the state-marginal distribution, d π (cid:98) P, ˆ γ ( s ) generated by MLE MDP will give unnecessary amount of probability for the suboptimal data.
Therefore, it is interpreted that the third term is increased because samples with high D TV ( π ∥ π E ) are more frequently reflected in ϵ π as suboptimal data increases.
F. P ERFORMANCE ON D IFFERENT S UBOPTIMAL D ATASET
[Figure]
li
[Figure]
d A v
[Figure]
a g
[Figure]
t
[Figure]
We additionally conducted an experiment using medium trajectory as suboptimal data.
Figureshows the evaluations of learning 1 million steps according to the discount factor when the suboptimal dataset is consisted of 850 medium trajectories, and the gray line represents DemoDICE’s best performance.
As a result, in an environment other than Hopper-v2 it was possible to obtain a higher return than DemoDICE.
We also included the learning curve of this experiment in next page,
Figure 13.
Here, we can see that the IGI still shows robust learning on the discount factor.
[Figure]
r m a li z e d
A v e r a g
[Figure]
R
[Figure]
n
[Figure]
o r
[Figure]
a li z e d
A v e r a g e R
[Figure]
n
[Figure]