Self-Imitation Learning from Demonstrations
Abstract
Despite the numerous breakthroughs achieved with Reinforcement Learning (RL), solving environments with sparse rewards remains a challenging task that requires sophisticated exploration.
Learning from Demonstrations (LfD) remedies this issue by guiding agent’s exploration towards states experienced by an expert.
Naturally, the beneﬁts of this approach hinge on the quality of demonstrations, which are rarely optimal in realistic scenarios.
Modern LfD algorithms lack robustness to suboptimal demonstrations and introduce additional hyperparameters to control the inﬂuence of demonstrations.
To address these issues, we extend Self-Imitation
Learning (SIL), a recent RL algorithm that exploits agent’s past good experience, to the LfD setup by initializing its replay buffer with demonstrations.
We denote our algorithm as SIL from Demonstrations (SILfD).
Our theoretical analysis highlights that SILfD is safe to apply to demonstrations of any degree of suboptimality and automatically adjusts the inﬂuence of demonstrations throughout the training.
Our empirical investigation shows the superiority of SIL over existing LfD algorithms in settings of suboptimal demonstrations and sparse rewards.
Introduction
Deep Reinforcement Learning (RL) algorithms have recently achieved multiple breakthroughs in solving games [18, 20, 3, 14, 4], hard visuomotor [16] and manipulation [10] tasks, but some of these algorithms additionally rely on incorporating information from human demonstrations [34, 38].
Finding the optimal solution requires tremendous amount of environment interactions, which makes
RL algorithms costly and dependent on sophisticated exploration techniques.
This is especially true for environments with sparse rewards where encountering a positive reward requires a long and precise sequence of actions.
An alternative approach is to additionally leverage a set of expert demonstrations, which has shown to help with exploration difﬁculties and provide magnitudes of improvement in learning speed and performance.
This approach is known in the literature as Learning from Demonstrations (LfD)
LfD algorithms can be attributed to one of the three categories based on the technique to incorporate demonstrations.
The ﬁrst technique is to treat demonstrations as additional learning references by placing them in the experience replay buffer [11, 37, 9, 21, 25].
The second technique is to optimize a mixture of reinforcement and imitation objectives, either by mixing rewards [15, 40, 42, 5, 13] or by introducing an auxiliary imitation loss term [11, 28, 21].
The third technique is to initialize agent’s parameters with supervised [34, 28, 32] or imitation [7] pretraining.
Despite the impressive results on a variety of problems, modern algorithms typically assume access to high-quality demonstrations [Figure] and, as our experiments conﬁrm, degrade when demonstrations are suboptimal.
Furthermore, these algorithms have to rely on additional hyperparameters to properly balance between learning from agent and expert experience, such as priority bonus of demonstrations in the replay buffer [11, 37, 25] and coefﬁcients of auxiliary losses [11, 37, 21, 32] or rewards [28, 15, 40, 13].
Self-Imitation Learning (SIL)
[22] is a recent RL algorithm that imitates its past positive experience while ignoring negative experience.
SIL has shown to ﬁt particularly well in environments with sparse rewards where it can mimic complex behaviour required to reach the reward.
Still, encountering the reward in the ﬁrst place can be problematic, especially when using naive exploration.
In this paper we argue that SIL can greatly beneﬁt from expert demonstrations by alleviating the need to encounter positive experience and propose Self-Imitation Learning from Demonstrations (SILfD).
The idea behind SILfD is to initialize the experience replay buffer with demonstrations that are preserved in the buffer throughout the training.
While this idea is not new and is also used in algorithms like DQfD [11] and DDPGfD [37], we argue that the ability of SIL to prioritize positive experience while ignoring negative experience makes it a natural choice to learn from demonstrations.
On the one hand, prioritization mechanism ensures that demonstrations are not under-exploited when they are more useful than agent experience.
Moreover, being able to focus on the most useful experience makes it safe to learn from suboptimal demonstrations with varying quality of behaviour.
On the other hand, SILfD can forgo learning from demonstrations once they become obsolete.
As a result, SILfD dynamically balances between learning from agent and expert experience based on its current usefulness.
Our theoretical analysis supports these claims.
Additionally, we propose SILfIL, an extension of SILfD to the case where rewards are not observed in demonstrations, which can be especially relevant when demonstrations are collected by a human expert.
To test the effectiveness of SILfD, we compare it with the existing LfD algorithms in several environments: a toy hard-exploration environment Chain [35]; four DeepMind Control Suite tasks [36] with continuous actions and sparsiﬁed rewards; and Pommerman environment [29] with procedural map generation.
Experiments show that both SILfD and SILfIL outperform the existing LfD algorithms, especially when demonstrations are highly suboptimal.
Background 2.1 Reinforcement Learning
We consider the standard Markov Decision Process (cid:104) S, A, r, T, γ (cid:105) , where
• S denotes the space of states s , A denotes the space of actions a , • r : S ×
A → R denotes the reward function, • T : S ×
A → ∆( S ) denotes the transition function, where ∆ denotes probability distribution, • γ ∈ (0 , 1) denotes the discount factor.
• Further, R t = (cid:80) ∞ n = t γ n − t r n denotes return, where subscripts t and n denote time steps, • π θ : S → ∆( A ) denotes policy parameterized by θ , • V φ ( s ) =
E π θ [ R t | s t = s ] , Q φ ( s, a ) =
E π θ [ R t | s t = s, a t = a ] , and A φ ( s, a )
=
Q φ ( s, a ) − V φ ( s ) denote value, Q-value, and advantage functions parameterized by φ .
Advantage-Actor-Critic (A2C)
[19] is one of the most prevalent RL frameworks where Actor chooses actions in the environment by predicting policy in a given state while Critic evaluates the state to aid
Actor’s learning.
Proximal Policy Optimization (PPO)
[33] is based on A2C framework and focuses on staying within a trust region during updates of policy parameters.
2.2 Imitation Learning
The purpose of Imitation Learning (IL) is to train a policy that mimics expert behaviour, the samples from which are stored in a buffer of demonstrations D
= ( s, a ) .
Behavioral Cloning (BC) is a classic
IL algorithm that trains a policy to predict the demonstrated action for a given state in a supervised manner without environment interactions [27].
Generative Adversarial Imitation Learning (GAIL) is a modern algorithm that trains two adversarial models: discriminator and generator [12].
Discriminator is a binary classiﬁer that distinguishes generated and expert transitions, whereas generator interprets discriminator’s predictions as rewards and tries to confuse it.
2.3 Learning from Demonstrations
Unlike IL, Learning from Demonstrations (LfD) assumes both the reward signal r and a buffer of demonstrations D =
( s, a, r ) to be available.
Typically, LfD algorithms use demonstrations to increase sample efﬁciency and aid exploration in environments with sparse rewards.
Hester et al. [11] propose DQfD which extends DQN to the LfD setup by initializing the replay buffer with demonstrations, pretraining Q-network ofﬂine, mixing 1-step and n-step losses, and regularizing the network with an auxiliary supervised loss.
As an analogue of DQfD for continuous control,
Vecerik et al.
[37] propose DDPGfD by applying similar modiﬁcations to the critic of DDPG.
As an alternative approach, POfD [15] enforces occupancy measure matching between the agent and the expert by shaping the reward with the predictions of a GAIL-like discriminator.
Similar to POfD ideas are employed in [40, 42].
Finally, supervised pretraining from demonstrations with BC is routinely used to assist solving complex tasks, e.g. Go [34] and Minecraft [32].
2.4 Self-Imitation Learning
SIL [22] aims to reproduce agent’s past good decisions based on the experience stored in a replay buffer B =
( s, a, r ) .
The algorithm alternates between the standard on-policy update of A2C or PPO and the off-policy update that minimizes A2C loss with clipped advantages:
[Equation]
where ( · ) + = max ( · , 0) ensures that only good transitions are considered for updates, advantage is estimated as A ( s, a ) =
R − V ( s ) , H ( π θ ) =
E π θ [ − log π θ ( a | s )] denotes the entropy of the policy, and α ≥ 0 .
Oh et al.
[22] theoretically justify SIL by connecting it to lower-bound soft Q-learning that approximates the lower bound of the optimal soft Q-values Q ∗ by minimizing:
[Equation]
where R µt = r t + (cid:80) ∞ n = t +1 γ n − t ( r n + α H n ( µ )) is the entropy-regularized return.
Speciﬁcally, the authors show that minimizing L lb is equivalent to minimizing L silpolicy and L silvalue when α → 0 .
To improve training efﬁciency, SIL uses prioritized replay buffer [31].
Self-Imitation Learning from Demonstrations
The previous section described how SIL is able to exploit past good experience.
However, encountering such experience can be problematic, especially in environments with sparse rewards.
An alternative source of good experience can be a set of expert demonstrations D .
As an extension of SIL to LfD setting, we propose SIL from Demonstrations (SILfD) based on one simple modiﬁcation of the original SIL: the experience replay buffer is initialized with demonstrations D that are preserved in the buffer throughout the training (Fig. 1a).
The policy and value losses are accordingly modiﬁed:
[Equation]
3.1 Theoretical Analysis
This section provides insights into the effect of adding demonstrations to the experience replay buffer of SIL.
Speciﬁcally:
• We establish the conditions under which demonstrations improve the speed of convergence to the optimal policy (Lemma 1) •
We show that completely useless demonstrations are ignored and thus are safe to include in the buffer (Lemma 2) •
We describe how demonstrations are sampled progressively less according to an automated schedule that naturally arises as agent improves (Lemma 3)
Since SIL can be viewed as a form of lower-bound soft Q-learning, we perform our analysis on the latter.
Like the original SIL paper, we restrict the analysis to deterministic environments.
Preliminaries.
We start by rewriting the lower-bound objective (2).
We consider the LfD setup where agent and expert transitions are stored in the same replay buffer with λ being the proportion of expert transitions.
In this setup, (2) can be rewritten as:
[Equation]
where ρ ( s, a ) is occupancy measure, ∆ + ν = ∆
+ ν ( s, a, R ) =
( R ν − Q φ ( s, a )) + will be referred as clipped advantage, ν denotes an arbitrary policy, and subscript d means that the loss is augmented with demonstrations.
We now rewrite objective (4) for the prioritized sampling employed in SIL.
Each transition is sampled from the buffer with probability proportional to the clipped advantage:
[Equation]
where sg [ · ] is a stopgradient operation that indicates that the gradient is not propagated through the sampling probability, Z = (1 − λ )
Z π + λZ π E is the normalizing constant for the joint buffer,
Z ν = (cid:80) s,a ρ ν ( s, a ) E R ∼ ν sg [∆ + ν ] .
We can further rewrite this by multiplying the ﬁrst term by Z π Z π , the second term by Z πE Z πE :
[Equation]
where (1 − p )
= (1 − λ )
Z π Z and p = λZ πE Z are the respective marginal probabilities to sample an agent or an expert transition, p ν ( s, a, R )
=
ρ ν ( s, a ) sg
[∆ + ν ] /Z
ν is the probability of prioritized sampling of a triple ( s, a, R ) from the policy ν .
When do demonstrations help SIL to converge faster?
To answer this question, we ﬁrst notice that because the lower bound Q φ can only increase, the speed of its convergence to Q ∗ is directly tied into the magnitude of the gradient of the loss function ∇ φ L lb , which we estimate as its l1-norm.
For this reason, we are interested in the conditions under which adding demonstrations to the buffer increases the gradient.
We formulate such conditions for one step of update in Lemma 1.
Lemma 1.
[Equation]
We leave the proof to the Appendix.
Lemma 1 can be seen as a formalization of a simple statement that ‘demonstrations help if they are more useful than the collected experience‘, giving precise deﬁnitions to ‘help‘ and ‘more useful‘.
In particular, the condition in Lemma 1 can be read as follows: expected clipped advantage, weighted by the sampling probability proportional to the clipped advantage, multiplied by the norm of the gradient of the Q-value, is higher for the expert than the agent.
This condition might hold due to expert outperforming agent in states visited by both or due to expert visiting more rewarding states.
Completely useless demonstrations are ignored.
Lemma 1 establishes the condition under which demonstrations help.
However, we are still interested in the case of suboptimal demonstrations where this condition is not satisﬁed.
A particularly interesting special case is the case of completely useless demonstrations, which we deﬁne below:
Deﬁnition 1.
Let demonstrations be called completely useless if:
[Equation]
Since the deﬁnition depends on the current estimate of Q-value, the demonstrations can be completely useless at the very beginning of training (if a random agent outperforms expert) or become completely useless as a result of Q φ becoming tighter during training.
Completely useless demonstrations can potentially be harmful to LfD algorithms by guiding exploration towards regions of low returns.
Fortunately, SILfD is robust to such demonstrations as they do not affect the loss function and thus are safe to include in the replay buffer.
We formulate this property in the following lemma:
Lemma 2.
If demonstrations are completely useless, ∇ φ L lb d [Equation] ∇ φ L lb .
We provide the formal proof in the Appendix for completion.
Automated scheduling.
We now discuss how SILfD adheres to an automated schedule that arises from agent’s improvement and results in a gradual decrease of the inﬂuence of demonstrations.
Suppose that lower bound Q φ got tighter by Q diff as a result of an update.
Suppose that the agent’s policy has also improved from π to π new and that the buffer got ﬁlled with experience from π new instead of π .
As a result, the expected clipped advantage of the collected experience has changed by d ( π, π new ) =
E s,a,R ∼ π new (∆ π new − Q diff ) + − E s,a,R ∼ π ∆ + π .
In the next lemma we connect d ( π, π new ) to the change of the sampling probability of demonstrations.
Lemma 3.
Suppose that Q φ ( s, a ) increases by Q diff ( s, a ) and the agent’s policy changes from π to π new .
Then, p decreases if the following condition is satisﬁed:
[Equation]
The formal proof of Lemma 3 is reported in the Appendix.
Notice that the right-hand side of the condition in Lemma 3 is negative since Q diff > 0 .
Therefore, for the condition to hold it is sufﬁcient that d ( π, π new ) > 0 . Consequently, as long as the agent improves proportionally to the lower bound estimate, the condition in Lemma 3 is satisﬁed and the probability to sample from demonstrations p decreases.
In the next subsection, we compare SILfD with the most prominent existing LfD algorithms, none of which exhibits the same useful properties.
3.2 Comparison with existing algorithms
DQfD [11] and DDPGfD [37] utilize demonstrations by placing them into the replay buffer alongside with the agent experience.
As Kang et al.
[15] show, DQfD and DDPGfD employ guided exploration based on occupancy measure matching between the agent and the expert.
This approach exhibits several issues.
First, demonstrations can be under-exploited by being diluted with agent experience.
In attempts to mitigate this problem, DQfD and DDPGfD speciﬁcally give a priority bonus to demonstrations, and DQfD additionally introduces a supervised loss term as a regularizer.
SILfD does not rely on such heuristics and enjoys a provable improvement of convergence speed regardless of proportions λ (Lemma 1).
Second, suboptimal demonstrations can be over-exploited.
In case when useful and completely useless demonstrations are mixed, such demonstrations can potentially hurt convergence of DQfD and DDPGfD by guiding exploration towards regions of low returns, whereas SILfD ignores completely useless demonstrations (Lemma 2) and is able to only learn from useful experience.
Our experiments in Chain environment also support this claim.
Third, dynamic adjustment of the inﬂuence of demonstrations requires hand-tuned schedules of hyperparameters.
Unless expert behaves optimally, agent’s exploration becomes a better source of experience at some point of training.
DQfD and DDPGfD might be unable to forgo learning from obsolete demonstrations if the priority bonus is set too high, whereas scheduling it requires tuning additional hyperparameters.
As we show in Lemma 3, SILfD schedules learning from demonstrations automatically.
Like DQfD and DDPGfD, POfD [15] employs guided exploration based on occupancy measure matching and is therefore susceptible to the same issues.
POfD is trained on a mixture of environmental and imitation rewards.
To avoid under- or over-dependence on demonstrations, the mixture coefﬁcient needs to be carefully tuned and potentially scheduled.
Furthermore, the beneﬁts of exploration via POfD are tied in to a crucial assumption that π E is more advantageous than π in expectation.
This assumption is not satisﬁed when useful demonstrations are mixed with a sufﬁcient number of completely useless demonstrations, which SILfD is robust to.
3.3 Demonstrations without rewards
One drawback of SILfD is the assumption that rewards are observed in expert demonstrations.
This limits applicability of SILfD in the most interesting cases where demonstrations are collected by a human expert, possibly optimizing a different reward or acting in a different environment [41, 8, 32, 26].
To mitigate this drawback, we modify SILfD when rewards are unavailable in demonstrations.
Instead of directly ﬁlling the replay buffer with demonstrations, we train an IL algorithm like BC to mimic demonstrations and initialize the replay buffer with its experience (Fig. 1b).
As an incidental beneﬁt, IL algorithm may produce imperfect imitations that serve as more diverse learning references than the original demonstrations.
We denote this modiﬁcation as SILfIL.
Another alternative we test is supervised pretraining.
Experimental Procedure 4.1 Environments
We have designed the experiments with the following desiderata in mind.
First, SILfD should be tested in both discrete and continuous environments.
To this end, we have chosen Pommerman and Chain as discrete environments and four DeepMind Control Suite tasks as continuous environments.
Second, environments should have sparse rewards since in this setting demonstrations are the most helpful.
While rewards in Chain and Pommerman are already sparse, we have additionally sparsiﬁed the rewards in DMC.
Third, demonstrations should be imperfect to test both robustness of algorithms and their ability to surpass the expert.
To this end, we vary the proportion of suboptimal demonstrations mixed with one optimal demonstration in Chain and use a suboptimal expert in DMC.
While expert always wins in Pommerman, procedural map generation prevents agent from copying the expert.
Chain [35] is a simple but popular exploration benchmark [23, 24].
An illustration is provided in
Appendix.
The environment represents a square grid where the agent starts at the upper-left corner, its goal is to reach the bottom-right corner, and its actions are to move diagonally either to the lower-left cell or to the lower right cell.
In our modiﬁcation, the agent is penalized for moving right but receives a signiﬁcant positive reward that exceeds any achievable sum of penalties by 100 when reaching the bottom-right corner.
Additionally, if the agent is located on the left edge of the grid and steps left, it simply moves one cell down and does not receive a penalty.
Without speciﬁc exploration strategies or relying on demonstrations, standard RL algorithms cannot ﬁnd the positive reward and converge to the policy that avoids penalties by always moving left.
We record one optimal demonstration where the expert only moves right and dilute it with n ∈ [0 , 99] adversarial demonstrations where the expert only moves left.
We ﬁx the size of the map at 40x40.
Pommerman [29] is a challenging multi-agent environment with discrete control and highdimensional observations.
We adapt the single-agent regime proposed in [2] where the agent needs to defeat a single random opponent.
Similarly to [2], we use the champion solution of FFAcompetition [39] to gather 300 demonstrations.
However, we do not modify the original environment used in the competition, including its procedural generation.
This ampliﬁes the difﬁculty for the algorithms, as they have to imitate the expert in unseen states.
Furthermore, the only reward signal the agent receives is +1 for defeating the opponent or -1 for being eliminated.
DeepMind Control Suite [36] is a set of popular benchmarks with continuous control.
For our experiments, we select Cartpole Swing Up, which is a classic task where the agent needs to balance an unactuated pole by moving a cart, and three locomotive tasks: Cheetah Run, Walker Run and
Hopper Hop, where the agent is supposed to control and to move forward a speciﬁc robot.
In order to make exploration more challenging, we sparsify the rewards in all environments, the details of which are reported in the Appendix.
We collect 25 suboptimal demonstrations with similar score in each environment.
The details about the experts are also reported in the Appendix.
4.2 Algorithms
For technical details such as hyperparameters and how they were tuned, please refer to Appendix.
SILfD and SILfIL Our algorithms are implemented according to Section 3.
In SILfD, the replay buffer is initialized with demonstrations.
In SILfIL, the buffer is instead initialized with experience generated by a pretrained IL algorithm, in our case, BC.
SIL Since vanilla SIL [22] does not leverage demonstrations, comparing it with our algorithms highlights the beneﬁts of demonstrations in hard-exploration environments.
BC This is a classic IL approach based on supervised learning that predicts a demonstrated actions in a given state [27].
Since BC ignores rewards, it is unlikely to outperform the expert.
IL → SIL As an ablation of SILfIL, we test an alternative approach to utilize demonstrations without rewards. In particular, the sequential baseline IL → SIL ﬁrst pretrains the policy on demonstrations with an BC and then ﬁne-tunes it in the environment with SIL.
This way, we can directly compare utilizing demonstrations in SIL through the buffer and through pretraining (note that both could be applied simultaneously).
Similar two-step approaches are common in the LfD literature [34, 28, 32].
POfD
A modern LfD approach employed in POfD [15] and several other studies [40, 42] is to optimize a mixture of environmental and imitation rewards: r = r env + λ 1 r im , λ 1 ∈ [0 , ∞ ] .
Similarly to GAIL, the imitation reward is obtained using predictions of a discriminator trained to distinguish agent and expert experience.
As the base policy optimization algorithm, we use PPO.
DQfD and DDPGfD Similarly to SILfD, DQfD [11] and DDPGfD [37] store demonstrations in the replay buffer.
Our implementations of DQN and DDPG are based on RLlib framework [17].
These algorithms are more sample efﬁcient than other baselines and, according to our preliminary experiments, can overﬁt if trained for too long, so we train them less and show only the ﬁnal performance rather than the learning curves on charts.
Decision Transformer Decision Transformer (DT) is a recent application of transformers to ofﬂine
RL based on a framework that casts RL as conditional sequence modelling [6].
This model treats a projection of a past state-action pair and a desired return as a token.
A casually masked sequence of such tokens representing past trajectory is passed through several attention layers and a linear decoder to predict an action that achieves a desired return in a given state.
DT performs at least comparably with ofﬂine TD-based algorithms and BC and is able to extrapolate to returns beyond those provided during training.
We use the authors’ implementation of DT
[Figure]
Experimental Results
Chain The results for the Chain environment are presented in Figure 2a.
In these experiments, we put emphasis on the inﬂuence of suboptimal demonstrations and intentionally dilute one optimal expert trajectory with completely useless episodes.
We ﬁnd that only three of the tested algorithms can consistently solve all the tasks: both our algorithms and DT.
For SILfD and SILfIL, this result complements the theoretical robustness to completely useless demonstrations (Lemma 2).
Furthermore, since in SILfIL the buffer is initialized with demonstrations collected by BC, the performance of BC is crucial.
On the plot, we can see that the average return of BC is close to 0 on the hardest task, so most of the generated experience is useless, but even a rare successful episode is sufﬁcient for SIL.
Similarly, the sequential baseline can also successfully leverage pretrained BC to train SIL but is not as stable on harder tasks.
By itself, SIL is unable to encounter the positive reward.
Regarding DT, its ﬂawless performance on Chain is expected since it is designed to be able to distinguish episodes with different returns.
The performance of other algorithms gradually falls off on harder tasks.
BC is expected to approximate expert performance, which it does.
However, POfD and DQfD perform only slightly better than BC, despite optimizing (at least partly) for environmental reward.
This supports our discussion in Section 3.2 about these algorithms having difﬁculties with controlling inﬂuence and discerning usefulness of demonstrations.
Pommerman The results for the Pommerman environment are presented in Figure 2b.
Due to randomized map initialization, Pommerman presents a challenge for all algorithms, but only our algorithms reach the performance of the optimal expert consistently.
The sequential baseline, an alternative way to leverage demonstrations with SIL, performs signiﬁcantly less consistently: while it manages to learn a winning policy almost instantly on one of the seeds, it completely fails on the other two seeds.
BC performs relatively well and outperforms all other baselines, including DT.
While both algorithms experience a mismatch between training and inference due to random maps, BC makes decisions based only on a given state, whereas DT processes the whole past trajectory up to a given state.
This might make BC more robust to such mismatches.
Finally, POfD and DQfD fail on the task almost completely.
The results for the DMC environments are presented in Figure 3.
In all DMC environments, our SILfD and SILfIL perform nearly indistinguishably and almost always outperform the baselines.
The only difference between the two is that SILfD performs slightly better on average in Hopper.
The sequential baseline behaves similarly in Hopper and Half Cheetah, which is not surprising given that [Figure] it is also based on SIL.
While it fails to provide SIL with useful experience in CartPole, it excels in sample efﬁciency in Walker.
All algorithms that are not based on SIL perform worse.
POfD, while performing well in CartPole, slightly falls off in Half Cheetah, signiﬁcantly falls off in Hopper, and fails to reach the expert performance in Walker.
DDPGfD generally performs even worse.
While consistently outperforming the expert, DDPGfD never reaches the level of performance of SIL-based algorithms.
Once again, the results point towards the inability of these algorithms to efﬁciently leverage suboptimal demonstrations.
As anticipated, BC replicates the performance of the expert (with the exception of Hopper where it performs better).
Unlike the experiments in Chain, DT fails to outperform the expert and performs similarly to BC.
Apparently, DT is having difﬁculties with extrapolating to returns beyond the expert performance due to being a purely ofﬂine algorithm.
Conclusion
In this paper, we present SILfD, a novel algorithm that incorporates demonstrations into SelfImitation Learning.
We characterize the ability of SILfD to leverage useful demonstrations, ignore completely useless demonstrations, and dynamically adjust inﬂuence of demonstrations, providing both intuition and theoretical justiﬁcation to superiority of SILfD. Our experiments complement the theoretical results by showing that SILfD can learn from suboptimal demonstrations better than existing algorithms and reliably surpass the expert.
This conclusion holds for different kinds of suboptimality: when useful and useless demonstrations are mixed, when the expert completes the task successfully but not ideally, and when the expert acts in a limited subset of possible environment instances.
Additionally, we propose SILfIL, a modiﬁcation of SILfD that relaxes the requirement to observe rewards in demonstrations, and ﬁnd that it performs as well as the original algorithm. Acknowledgments This research was supported in part through computational resources of HPC facilities at HSE University.
Support from the Basic Research Program of the National Research University Higher
School of Economics is gratefully acknowledged.