SparseDice: Imitation Learning for Temporally Sparse Data via Regularization
Abstract
Imitation learning learns how to act by observing the behavior of an expert demonstrator.
We are concerned with a setting where the demonstrations comprise only a subset of state-action pairs (as opposed to the whole trajectories).
Our setup reﬂects the limitations of real-world problems when accessing the expert data.
For example, user logs may contain incomplete traces of behavior, or in robotics non-technical human demonstrators may describe trajectories using only a subset of all state-action pairs.
A recent approach to imitation learning via distribution matching, ValueDice, tends to overﬁt when demonstrations are temporally sparse.
We counter the overﬁtting by contributing regularization losses.
Our empirical evaluation with Mujoco benchmarks shows that we can successfully learn from very sparse and scarce expert data.
Moreover, (i) the quality of the learned policies is often comparable to those learned with full expert trajectories, and (ii) the number of training steps required to learn from sparse data is similar to the number of training steps when the agent has access to full expert trajectories.
1. Introduction
Sequential decision making is one of the central problems in computer science.
We are concerned with imitation learning , an approach to sequential decision making in which the agent has to learn to imitate observed behavior (Abbeel &
Ng, 2004).
Imitation learning shares commonalities with reinforcement learning (RL), another popular approach to sequential decision making (Puterman, 1994; Russell &
Norvig, 2016).
In either model, the agent does not know the dynamics of the world and has to learn a policy by interaction.
Imitation learning differs from RL in the way that the objective is speciﬁed.
In RL, the agent has to maximize accrued reward over time.
Designing a good reward function is often difﬁcult, and is prone to errors that result in learning unintended behaviors.
Imitation learning alleviates the burden of having to design reward functions.
Some approaches to imitation learning ﬁrst learn a reward function that is consistent with observed behavior, and then recast the problem as reinforcement learning (e.g., (Abbeel & Ng, 2004; Camacho et al., 2020a;b)).
Typical approaches to imitation learning make an implicit assumption that the behavior to imitate is demonstrated with dense execution traces.
We relax such assumption (Section 3).
Sometimes, the demonstrations are sparse—for example, when demonstrations are extracted from videos that were recorded with a lower bitrate than the frequency in which robot cameras and actuators operate.
Moreover, generating expert data is usually time- and cost-expensive, especially if demonstrations are generated manually by humans.
We are interested in addressing two questions: (i) Can we learn to imitate expert behavior with just a few data points?
(ii) How shall we generate expert data if we had limited resources?
We observe that existing approaches to imitation learning, originally designed to operate with dense demonstrations, may not perform well when demonstrations are sparse.
This is the case of ValueDice, a recent approach to imitation learning via distribution matching (Kostrikov et al., 2019).
In Section 3.2 we show that ValueDice has a high risk to overﬁt to sparse data and learned policies do not replicate observed behavior.
We contribute a number of regularizers that stabilize the learning of ValueDice (Section 4).
We refer to our approach as SparseDice.
We evaluate the performance of SparseDice on four different domains for continuous control, extracted from the Mujoco suite, provided with sparse demonstrations (Section 5).
SparseDice is more stable than ValueDice, and makes it still possible to learn from sparse demonstrations.
We also examined the tradeoffs between the diversity of demonstration data (when examples come from different demonstration traces), and their sparsity.
SparseDice can learn better-quality policies when there is a good balance between the diversity and sparsity of demonstration data.
Our results open the door to future research on imitation learning from sparse demonstrations.
2. Background on Imitation Learning
Imitation learning is the problem of learning a policy that replicates observed behavior (Abbeel & Ng, 2004).
The problem is framed in the context of an MDP (cid:104) S, A, p 0 , p, r, γ (cid:105) that the agent can interact with.
The agent can observe the current state s ∈ S and apply an action a ∈ A .
As a result, the state of the MDP transitions to a state s (cid:48) ∈ S with probability p ( s (cid:48) | s, a ) .
Such interaction yields a sequence τ = { ( s i , a i , s i +1 ) } h − 1that
we call execution .
In the scope of this paper we consider that executions have ﬁnite horizon h .
At the end of the execution, the state of the MDP is reset to a state s with probability p 0 ( s ) .
A policy for an MDP is a function π that maps each state s ∈ S into a probability distribution over actions, π ( ·| s ) :
A → [0 , 1] .
An execution of a policy π from state s 0 is a sequence τ = { ( s i , a i , s i +1 ) }
h − 1such that each a i has been sampled from π exp ( ·| s i ) , and s i +1 ∼ p ( ·| s i , a i ) .
The value of a policy π in a state s 0 , V π ( s 0 ) , is the expected cumulative discounted reward of a policy execution:
[Equation]
where a i ∼ π ( ·| s ) and s i +1 ∼ p ( ·| s i , a i ) , r : S ×
A → R is the reward function, and 0 < γ ≤ 1 is the discount factor of the MDP.
A policy π is optimal if it maximizes V π ( s ) in all states s ∈ S .
The agent has access to a set of demonstrations , D .
Formally, a demonstration is an execution drawn from an optimal policy π exp that optimizes for the expected cumulative discounted reward of the MDP.
Demonstrations showcase how an expert may act in the MDP.
The objective of the agent is to recover π exp .
In imitation learning the agent does not know the dynamics model of the MDP, and does not have access to the reward function either.
Note that imitation learning makes the assumption that demonstrations are drawn from an expert policy that optimizes for some reward function.
However, such a reward function does not need to be explicitly speciﬁed.
2.1. Imitation Learning Via Distribution Matching
ValueDice is a recent, promising approach for imitation learning, that approaches the problem through the lens of distribution matching (Kostrikov et al., 2019).
In this setting, we want to ﬁnd a policy π whose state-action distribution d π matches with the distribution d exp of an expert policy π exp , which are unknown to the agent.
Following (Puterman, 1994), there exists a one-to-one correspondence between a policy π and d π , deﬁned as: [Equation] a
[Equation]
where s 0 ∼ p 0 ( · ) , s t ∼ p ( ·| s t − 1 , a t − 1 ) , and a t ∼ π ( ·| s t ) .
With this notion, we can presume that expert demonstrations
[Equation]
= { [Equation] s i , a i , s
[Equation]
[Equation]
are sampled from a distribution
s i +1 ∼ d exp ( ·| s i , a i ) associated to some expert policy π exp .
In order to measure the discrepancy between d π and d exp , it is common to use (and minimize) the KL divergence D KL :
[Equation]
One of the practical problems of the above expression is that we cannot sample from the expert policy.
The authors of ValueDice perform several manipulations to the KL divergence to derive a max-min objective function J DICE that is more practical to optimize for.
[Equation]
To obtain the expression J DICE , they start from an alternative representation of the KL divergence, known as the
Donsker-Varadhan form (Donsker & Varadhan, 1983):
[Equation]
Then, the objective J DICE can be obtained by performing a change of variables x
( s, a ) =
ν ( s, a ) − B π ( s, a ) , where
[Equation]
[Equation] ) is the expected
Bellman operator with respect to π and zero reward.
In practice, the expression for J DICE ( π, ν ) can be adapted to perform mini-batch training with a mix of samples from the set of expert demonstrations and the tuples of experience ( s, a, s (cid:48) ) stored in the replay buffer, obtained by interaction with the environment.
Namely,
[Equation]
where d RB is the (uniform) distribution of state-action pairs in the replay buffer (RB), and d mix ( s, a ) =
(1 − α )
d exp ( s, a ) + αd RB ( s, a ) .
Relation with Q learning.
It has been observed that ν is some sort of Q value function of the underlying MDP (Kostrikov et al., 2019).
Such MDP has rewards r ( s, a ) =
− x ( s, a ) .
The function that optimizes the KL divergence is x ∗ = log d π ( s,a ) d exp ( s,a ) + C for some constant C (see (Kostrikov et al., 2019)).
3. Learning from Sparse Demonstrations
The literature on imitation learning commonly makes an underlying assumption that the number of expert demonstrations available to the agent is scarce.
This assumption reﬂects the limitations of many real-world learning systems for which generating expert data incurs a cost, there are limited time resources to collect such data, and the amount of storage is limited—e.g., generating expert demonstrations for a grasping robot is time- and cost-expensive (e.g., (Kalashnikov et al., 2018)).
Perhaps surprisingly, there is also the de facto conception that each individual demonstration comprises all the state-action pairs that appear along the execution trajectory.
We argue that this assumption does not accommodate many real-world systems in which collecting dense demonstrations is prohibitively costly or not even feasible.
For example, we may use demonstration videos to teach a robot how to assemble furniture, but the bitrate of such videos may be lower than the frequency rate of robot cameras, sensors, and actuators.
We study imitation learning with sparse demonstrations.
In this setting we relax the assumption that demonstrations are dense, and we consider that expert demonstrations constitute a number of examples —rather than dense trajectories.
Formally, an example is a triplet ( s, a, s (cid:48) ) , where a ∼ π ( ·| s ) is drawn from an expert policy π exp and s (cid:48) ∼ p ( ·| s, a ) .
An sparse demonstration is a subsequence of examples { ( s i , a i , s i +1 ) } J for some J ⊆ [1 , h − 1] , where τ = { ( s i , a i , s i +1 ) } h − 1is a (dense) demonstration.
Imitation learning from sparse examples has been underexplored.
One of the objectives of this paper is to evidence different trade-offs related to generating expert data and learning from it.
Certainly, we want to better understand how we shall generate and process such expert data.
Our contributions can be summarized as follows:
• We evidence that current methods for imitation learning are brittle when they are trained with sparse demonstrations, thereby motivating the need for new algorithms that can handle sparse demonstrations; • We show that current methods for imitation learning can be made more robust to sparse demonstrations, thereby unveiling opportunities for future research on this topic; • We study the performance of our methods relative to different distributions of sparse demonstration data, gaining insights on how we shall generate demonstrations when our resources to do so are limited.
3.1. Generating Datasets with Sparse Experience
One of the reasons that motivated the work presented in this paper was that we wanted to gain a better understanding on how to generate expert data for imitation learning.
Given a ﬁxed amount of resources to generate data, shall we generate a few dense demonstrations, or shall we generate a larger number of sparse demonstrations?
Existing datasets for imitation learning contain dense demonstrations.
We design different methods to generate datasets of sparse demonstrations, that we detail below.
Informally, we generate sparse data by sampling dense trajectories according to a ﬁxed distribution.
Such distributions are parameterized in a way that we can tune the degree of sparsity of data.
Uniform( p ): Each state-action pair in every demonstration is sampled iid with probability p .
On average, the number of state-action pairs is reduced by a factor p .
Periodic( t )
A state-action pair is sampled at every t timesteps.
The number of state-action pairs is reduced by a factor 1 /t .
Snippets( (cid:96) ; p ) Snippets of (cid:96) consecutive timesteps are sampled independently with probability p .
Approximately and on average, the number of state-action pairs is reduced by a factor p .
3.2. Limitations of ValueDice
The ﬁrst contribution of this paper is an empirical evaluation of the performance of ValueDice when demonstrations are sparse.
We found that the quality of the policies learned by ValueDice can degrade signiﬁcantly in such a setting.
This is not an unexpected result, because ValueDice (as well as other algorithms for imitation learning) was originally designed to learn from dense demonstrations.
The result, however, motivates the development of new algorithms for imitation learning with sparse demonstrations.
We evaluate our methods on four popular benchmarks for continous control extracted from the Mujoco suite: Ant,
Hopper, Half Cheetah, and Walker 2D. To generate sparce demonstation datasets, we downsample dense demonstrations with the Uniform, Periodic, and Snippets methods described earlier in this section.
We use the same set of dense demonstrations used by Kostrikov et al.
(2020a), generated by executing policies trained using TRPO (Schulman et al., 2015).
The TRPO agent was trained with a reward function, but our agents for imitation learning do not have access to such reward function.
To evaluate the performance of the policies learned by ValueDice, we measure the reward collected by policy rollouts, and averaged results over 10 runs.
The reward function is the same used by the TRPO agent that generates expert demonstrations.
However, the reward function is hidden to the ValueDice agent, which has to learn solely from (sparse) demonstrations.
ValueDice can overﬁt data, even when trajectories are sampled periodically every other timestep.
ValueDice is sensitive to data sparsity.
We conducted a series of experiments in which we trained ValueDice with a diverse set of sparse demonstration datasets, for which we varied the degree of sparsity.
We observed that the quality of the policies learned by ValueDice degrades when the demonstrations are sparse.
Figure 1 summarizes the results of some of our experiments.
We show the quality of the policies along a wide range of 500 , 000 training steps.
When the demonstrations are dense, the quality of the policies learned by ValueDice is high and stable.
However, when the demonstrations are sparse the quality of the policies is lower and not stable, and it degrades signiﬁcantly.
It is very noticeable that ValueDice is unstable even when demonstrations are obtained by sampling Periodically every t =timesteps.
Such subsampling is not very aggressive and, a priori, we had expected that ValueDice was able to interpolate demonstration data.
However, ValueDice overﬁtted data very easily, and their performance is very low also in that setting.
Batch Normalization in ValueDice.
We conducted a series of experiments to determine whether the overﬁtting of
ValueDice to sparse demonstrations can be solved via simple batch normalization.
Unfortunately, this was not the case.
Figure 2 summarizes the results of our evaluations.
We augmented the network architecture used in ValueDice with batch normalization in the intermediate layers, and also in the ﬁnal layer.
The performance of these two conﬁgurations was very similar to the the performance of the original implementation of ValueDice.
These ablations suggest that we need more sophisticated regularization methods.
4. SparseDice: Regularizing ValueDice
We propose SparseDice, a strongly regularized DICE algorithm, to overcome the overﬁtting of ValueDice and improve its stability (Section 3.2).
The main objective, J DICE ( π, ν ) , is a functional that takes both π , and ν functions as inputs.
We exploit this dependency and regularize the objective by ensuring that both π and ν stay close to the expert demonstrations without overﬁtting.
In the following, we introduce a series of loss functions that we add to J DICE ( π, ν ) .
Notation.
Recall from Section 2.1 that we denote by
RB the replay buffer , i.e.., the set of tuples of experience ( s, a, s (cid:48) ) obtained by interaction with the environment.
We denote by D the set of (sparse) demonstrations, that also comprise tuples of examples ( s ∗ , a ∗ , ( s ∗ ) (cid:48) ) .
In the deﬁnition of the regularization losses, we denote with ν t (resp., π t ) the target network that results from freezing the weights of ν (resp., π ) to make them untrainable.
Using target networks is a standard practice to make training more stable.
4.1. Regularizers for the Value Function
Expert contrastive loss ν can be interpreted as the value function of the underlying MDP where the reward is deﬁned as log d exp ( s,a ) d
π ( s,a ) .
Based on this observation, we introduce contrastive learning losses to enforce ν to take high values on expert demonstrations and low values otherwise,
[Equation]
The action space is continuous and computing max a ν ( s ∗ , a ) is intractable.
Instead, we replace max a ν ( s ∗ , a ) with ν ( s ∗ , a (cid:48) ) , where a (cid:48) is sampled randomly from the set of actions that appear in the expert demonstrations.
Finally, we impose a non-negativity constraint as the above objective is always non-negative and use a mini-batch approximation,
[Equation]
[Table]
RB contrastive loss This loss enforces ν to take high values in the state-action pairs that are on-policy.
For each pair ( s, · ) ∈ RB sampled from the replay buffer, we compute the value ν
( s, a (cid:48) )
with the action a (cid:48) ∼ π ( ·| s ) returned by the current policy and enforce that it is higher than values with random actions.
It is important to note that the parameters of the policy π are frozen and not made trainable.
[Equation]
Similar to the expert contrastive loss, we replace max a with random actions and use mini-batch approximation,
[Equation]
ν RB expert contrastive loss We regularize ν to stay close to values of state and action pairs on expert demonstrations by minimizing the delta between on-policy and expert ν estimates,
[Equation]
where µ exp =
E ( s ∗ ,a ∗ ) ∼ d exp ν t ( s ∗ , a ∗ ) is the mean of ν values over the demonstration data.
In practice we use,
[Equation]
where µ D =
[Equation]
ν expert greater than mean loss We also regularize ν if it diverges too much from average expert values,
[Equation]
In practice we use,
[Equation]
where µ [Equation] =
[Equation]
ν bellman update loss The value function ν can be interpreted as a Q value function that optimizes for the expected cumulative discounted reward r ( s, a ) =
− x ( s, a ) (Kostrikov et al., 2019).
This is a consequence of reinterpreting the KL divergence as an MDP with rewards − x ( s, a ) .
Furthermore, the optimal values r ∗ ( s, a ) = − x ∗ ( s, a ) = log d exp ( s,a ) d π ( s,a ) + C are constant when π converges to π exp .
If we interpret ν as a Q value function, then we can apply
Bellman updates to ν in the same manner that is done in
SARSA—i.e., for a transition ( s, a, s (cid:48) )
and reward r , we update Q ( s, a ) to r + γQ ( s (cid:48) , π ( s (cid:48) )) with some learning rate α .
There is one missing piece: in imitation learning the agent does not have a reward function.
However, we can estimate r from the values of the value function, v .
Observe that ν , interpreted as a Q value, is the sum of discounted rewards r + γr + γ 2 r + . . .
=
r 1 − γ , and we have the equivalence r
= ν (1 − γ ) .
Finally, we can estimate a (constant) reward function r by sampling a batch of expert experience, and averaging over the values of ν : r = (
[Equation]
−
[Equation]
∗
[Equation]
a ∗ ) ∈ batch ( D ) ν
( s ∗ , a ∗ ) .
We use this reward in conjunction with Q learning updates to derive the following loss,
[Equation]
4.2. Regularizers for the Policy
Policy expert imitation loss We enforce π to imitate the actions that appear in the expert demonstrations.
More precisely, we use the squared L2 distance between an expert action a ∗ for state s ∗ , and the action a (cid:48) ∼ π ( ·| s ∗ ) output by the policy.
[Equation]
Policy self imitation loss Since the policy might overﬁt to expert demonstrations, we regularize the policy based on its distance to the replay buffer.
More precisely, this loss samples state-action pairs ( s, a ) from the replay buffer, and computes the squared L2 distance between an action a for
[Figure]
state s , and the action a (cid:48) ∼ π
[Equation] [Equation]
output by the policy.
[Equation]
5. Experimental Evaluation
ValueDice can learn from a small set of dense demonstrations (Kostrikov et al., 2020a).
However, as we saw in Section 3.2, one of its limitations is that demonstrations have to contain all the time steps in a trajectory.
Otherwise, ValueDice has a high risk of overﬁtting when demonstrations are sparse.
We saw that in such a case, the policies learned by ValueDice did not replicate expert behavior.
Overﬁtting occured even when we sampled every other time step in the expert trajectories.
The purpose of our experimental evaluation is to address two practical questions:
Can we learn to imitate expert behavior with just a few data points?
If so, how does the quality of the policies learned degrade when we reduce the number of training examples?
How shall we generate expert data?
If we had limited resources, we want to understand whether it is better to generate a few dense demonstrations, or a more larger set of sparse demonstrations.
Experimental setup.
We evaluated the performance of
SparseDice on four popular continuous control tasks (Ant,
Hopper, Half Cheetah, and Walker 2D) extracted from the
Mujoco suite (Todorov et al., 2012).
Expert demonstrations were generated by running executions of a TRPO agent (that was trained with rewards).
Each (dense) expert demonstration comprises several thousands of consecutive examples ( s, a, s (cid:48) ) , and showcases several cycles of a robot moving
[Figure]
straightforward.
The actions have certain stochasticity that make the robotic behavior be not exactly cyclic.
Our results are averaged over 10 runs.
Implementation details.
We implemented SparseDice on top of ValueDice (Kostrikov et al., 2020a), by adding the regularization losses listed in the preceding section.
The objective function, J ( π, ν ) , is parameterized on two functions ν and π .
Each of these two functions is modeled with a feed-forward neural network with three layers.
In
SparseDice, we augmented the output layer of the ν network with a sigmoid activation.
Otherwise, we observed that the outputs diverge as a side effect of some of the regularizers.
We perform mini-batch training steps after each interaction with the environment, with a batch size of 256.
When we perform training steps we freeze the weights of the target networks to compute the gradients.
5.1. Limitations of Learning with Sparse Data
We wanted to know whether it is possible to learn from a few sparse data points.
To this end, we conducted a series of experiments to evaluate the performance of SparseDice when it is trained with sparse demonstrations.
We generated a variety of sparse datasets by subsampling a ﬁxed number
N = 10 of dense expert demonstrations.
We experimented with different sampling distributions, and different sampling rates.
The results of our experiments are summarized in Figure 3.
As expected, the quality of the policies learned by
SparseDice degrades when downsampling is more aggressive.
However, it is remarkable that SparseDice is more stable than ValueDice, even when SparseDice is given less amount of expert data.
Our empirical evaluation showcased that SparseDice is more robust to sparse demonstrations than ValueDice.
The policies learned with SparseDice are more stable, and in many cases they have better-quality than those learned by ValueDice.
5.2. Adding More Diversity on Training Data
Generating training data is costly, and the amount of training data that we can make available to the learning system may be limited.
Under these conditions, we wanted to learn how we shall generate our training data.
More precisely, it better to generate a few dense trajectories, or many sparse trajectories?
Furthermore, what are the best sampling strategies at the time of generating the dataset of expert examples?
We conducted a series of experiments where we ﬁxed (on average) the number of examples in the datasets of demonstrations.
We constructed different datasets of experience by sampling from 10, 30, 50, and 100 demonstrations.
Sampling was done in a way that the expected number of data samples is ﬁxed.
However, by sampling from the set of 100 demonstrations we get more diverse data than sampling from the set of 10 demonstrations.
The results are summarized in Figure 4.
We observed that
SparseDice can beneﬁt from being trained with more diverse data.
However, there exists tradeoffs.
Given a ﬁxed amount of resources to generate examples, the best number of individual demonstrations to sample from is uncertain, and we do not have a rule to determine such number beforehand.
5.3. Ablations
We conducted ablations to assess the contribution of each of the regularization losses to the quality of the policies learned by SparseDice.
We found this to be the case.
In our experiments, the “policy expert imitation loss” resulted to be the most beneﬁcial regularizer.
6. Related Work
Regularization is a core paradigm in machine learning.
While regularization has a long history in supervised learning, it has recently come into focus for RL and imitation learning due to interest in transfer learning (Cobbe et al., 2018) and ofﬂine RL (Levine et al., 2020).
In transfer RL, the focus is on learning feature representations which generalize to other tasks given online access to a set of training tasks, whereas our setting is closer to ofﬂine RL, for which the main challenge is learning from a static dataset that exhibits sparsity in the state-action space.
This may partially explain why we found poor performance of regularizers (e.g., batch normalization) which are known to perform well in transfer RL (Cobbe et al., 2018).
In the ofﬂine RL setting, much of the regularization is focused on keeping the learned policy close to the ofﬂine dataset, via various behavior regularization techniques (Fujimoto et al., 2019; Wu et al., 2019).
In the imitation learning scenario, this is not as much of an issue, as the imitation learning already compels the learned policy to stay close to the ofﬂine dataset.
Rather, many of our techniques can be interpreted as regularization on the value function , which also appears in the ofﬂine RL literature.
Namely, our various regularization objectives on ν can be related to critic regularizers in the ofﬂine RL literature (Nachum et al., 2019;
Kostrikov et al., 2021; Kumar et al., 2020).
Many of these existing works also contrastive-like regularizers which encourage larger critic values on actions in the ofﬂine dataset and smaller values on actions sampled from the learned policy.
As the ValueDICE objective (Kostrikov et al., 2020a) is derived from a GAN-like loss (Goodfellow et al., 2014), it is important to mention that regularizers are popular in the GAN literature as well, where mode-collapse is a wellknown issue (Srivastava et al., 2017).
One of the simplest and most popular regularizers is the gradient penalty (Arjovsky et al., 2017; Mroueh & Sercu, 2017).
However, we note that ValueDICE already includes a gradient penalty based on these existing works; although such a regularizer helps, we found that it is still not enough to maintain good performance in extremely sparse settings.
Other than applying regularizers, other works have suggested mitigating effects of sparse data by augmenting the data.
Namely, one applies perturbations (e.g., random noise) to the given dataset to generate more synthetic data.
This technique is popular in supervised learning (Chen et al., 2020) and has also been demonstrated in online
RL (Kostrikov et al., 2020b) and ofﬂine RL (Sinha & Garg, 2021) settings.
However, the use of such techniques relies on prior knowledge of the task to understand what perturbations are valid, and in fact, choosing the right perturbations can be a challenging problem on its own (Raileanu et al., 2020).
7. Discussion
Imitation learning is a convenient approach to sequential decision making.
We argued that many real-world applications of imitation leaning may need to handle sparse demonstration data—e.g., because of limited budget.
Existing algorithms for imitation learning that were designed to handle dense demonstrations, such as ValueDice, may not be as effective with sparse data.
We contributed with regularizers for ValueDice, that make learning more stable.
We saw that it is still possible to learn from a few sparse data points, algthough there exist tradeoffs between data sparsity and diversity that need be further studied.
Our studies open the door to future research on imitation learning with sparse demonstrations.