Learning from Imperfect Demonstrations via Adversarial Conﬁdence Transfer
Abstract —Existing learning from demonstration algorithms usually assume access to expert demonstrations.
However, this assumption is limiting in many real-world applications since the collected demonstrations may be suboptimal or even consist of failure cases.
We therefore study the problem of learning from imperfect demonstrations by learning a conﬁdence predictor.
Speciﬁcally, we rely on demonstrations along with their conﬁdence values from a different correspondent environment (source environment) to learn a conﬁdence predictor for the environment we aim to learn a policy in (target environment— where we only have unlabeled demonstrations. )
We learn a common latent space through adversarial distribution matching of multi-length partial trajectories to enable the transfer of conﬁdence across source and target environments.
The learned conﬁdence reweights the demonstrations to enable learning more from informative demonstrations and discarding the irrelevant ones.
Our experiments in three simulated environments and a real robot reaching task demonstrate that our approach learns a policy with the highest expected return.
We show the videos of the real robot arm experiments on our website.
I. I NTRODUCTION
Standard imitation learning algorithms assume that collected demonstrations are provided by experts, who can always achieve successful outcomes [ 1], [2], [3].
However, this assumption is usually violated in real-world applications.
Expert demonstrations require domain expertise and thus are expensive to obtain.
On the other hand, we often have easy access to a plethora of imperfect demonstrations ranging from expert behavior to noise.
This is evident when collecting human demonstrations on a robot arm since controlling manipulators with high degrees of freedom can be challenging for human users [4] or a bounded rational demonstrator may have difﬁculty providing optimal demonstrations [5]
Prior work on learning from imperfect demonstrations can be roughly categorized into conﬁdence-based and rankingbased methods: Conﬁdence-based methods learn a conﬁdence over demonstrations to re-weight them [6], while rankingbased methods recover the reward function [7], [8], [9].
However, all these works require some level of annotation, e.g., conﬁdence or rankings of the demonstrations.
For complex environments, large-scale annotations are required to cover the whole state-action space, which is labor-intensive and expensive to obtain.
Furthermore, the annotations can often contain a signiﬁcant amount of noise when collected through crowdsourcing platforms.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
/E/r5NieOVnQiUpcsXmi8JUEozJNADSF5ozlGNLKNPC3krYkGrK0MZUtCF4iy8vk2a14p1XqncX5dp1HkcBjuEEzsCDS6jBLdShAQwSeIZXeHNS58V5dz7mrStOPnMEf+B8/gCrHJIY</latexit> Demonstrations w/ c tar <latexit sha1_base64="kkE4WodyDL5Dwd2KFznLjokROjs=">AAAB83icbVBNS8NAEJ34WetX1aOXxSJ4KkkV9Fj04rGC/YA2ls120y7dbMLuRCyhf8OLB0W8+me8+W/ctjlo64OBx3szzMwLEikMuu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LRzdRvPXJtRKzucZxwP6IDJULBKFqpyx66yJ8wQ6onvVLZrbgzkGXi5aQMOeq90le3H7M04gqZpMZ0PDdBP6MaBZN8UuymhieUjeiAdyxVNOLGz2Y3T8ipVfokjLUthWSm/p7IaGTMOApsZ0RxaBa9qfif10kxvPIzoZIUuWLzRWEqCcZkGgDpC80ZyrEllGlhbyVsSDVlaGMq2hC8xZeXSbNa8c4r1buLcu06j6MAx3ACZ+DBJdTgFurQAAYJPMMrvDmp8+K8Ox/z1hUnnzmCP3A+fwCpiJIX</latexit>
[Figure]
In this paper, we focus on conﬁdence-based methods. As shown in Fig. 1, our insight is that instead of using manually annotated conﬁdence over demonstrations, we leverage conﬁdence annotations in a different but related environment—the source environment. The source environment may have a different state-action space from the target environment—the environment we are operating in.
The source environment and the target environment should have a correspondence mapping between state-action pairs [ 10].
We assume that collecting ofﬂine demonstrations along with their conﬁdence annotations is easy in the source environment.
As an example, the source and target environments can be a simulated robot and a real robot respectively or two different robot arms with different degrees of freedom.
Our key idea is to leverage the correspondence between state-action pairs in the source and target environments to learn a conﬁdence measure without the restrictive assumption of access to conﬁdence annotations of target demonstrations.
This problem setting introduces a new challenge: The source and target environments have different state-action spaces.
That means the dimensions of the states and actions and even the semantic meaning of each dimension can be different between the two environments, so we cannot directly apply the source conﬁdence predictor to the target state-action pairs.
We instead need to transfer the source conﬁdence predictor to the target environment.
We propose an approach that leverages the assumption of the correspondence between state-action pairs in the source and target environments leading to the existence of a common latent space, where one can learn a shared conﬁdence predictor for both environments.
Our approach aligns both the featurelevel distributions and the conﬁdence-level distributions of partial trajectories with different lengths across domains by a domain adversarial loss.
We design a multi-length partial trajectory matching, which preserves temporal relationships across consecutive states and actions, and enables an accurate matching with the corresponding trajectory in the target environment.
Our contribution is a new conﬁdence-based imitation learning algorithm that learns from imperfect demonstrations.
Instead of relying on manual annotations in our environment, we leverage conﬁdence-annotated demonstrations in a related environment—where collecting this data is easier—to learn a conﬁdence predictor for our environment of interest.
We experiment with several simulation environments and a real robot reaching task.
Our results suggest that our approach achieves both more accurate conﬁdence prediction and higher performance of the imitation policy than other methods.
II. R ELATED W ORK
Standard Imitation Learning.
Imitation learning aims to learn a policy by imitating the behavior of expert demonstrations, where mainstream methods can be categorized into behavior cloning (BC), inverse reinforcement learning (IRL), and generative adversarial imitation learning (GAIL).
BC directly learns the policy mapping state to action by supervised learning over state-action pair data in the demonstrations [11], [12], [13].
IRL ﬁrst recovers the reward function from demonstrations and learns a policy using reinforcement learning based on the learned reward function [14], [1], [2].
GAIL aligns the occupancy measure between the policy and the demonstrations with adversarial learning [3], [15], [16].
These standard imitation learning methods usually assume that all of the demonstrations are from experts who successfully ﬁnish the task.
Our problem goes beyond this assumption by allowing for learning from potentially suboptimal or even malicious demonstrations.
Imitation Learning from Imperfect Demonstrations.
Imitation learning from imperfect demonstrations aims to recover a well-performing policy from demonstrations that can range from random noise to optimal.
Ranking-based and conﬁdence-based methods are the two core approaches to address this problem.
Ranking-based methods recover the reward function from provided ground-truth [7], [17], selfgenerated [8], [9], or interactively collected rankings [18].
In our setting, we can only generate rankings and learn the reward in the source environment, which cannot be applied to the target environment with a different state space.
Target rankings can be self-generated by injecting noise in the target demonstrations as in [8], but that would require the selfgenerated demonstrations to be ranked worse than the original ones.
This may not hold in our setting with the potential of nearly random demonstrations in our dataset.
Conﬁdence-based methods typically learn the conﬁdence predictor from conﬁdence annotations [6] but the annotations are assumed to be on target demonstrations.
We go beyond this assumption in our setting, since the conﬁdence predictors are from a different environment.
Cao et al.
learn the target conﬁdence predictor from the reward of demonstrations in the source environment but still assume the two environments share the same state space [19].
We target a more general setting, where the source and target environments have different state-action spaces and dynamics.
Correspondence of Dynamics.
To learn the correspondence [20] between different dynamics, there are mainly two categories of methods: learning a translation mapping [21], or a common latent space [22] for states and actions.
To learn a translation mapping, Tzeng et al.
[23] weakly align pairs of images with paired image observations.
Ammar el al.
[24] utilize unsupervised manifold alignment to ﬁnd correspondence between states of demonstrations.
Kim et al.
[25] and Zhang et al.
[10] recently translated states and actions across environments by learning a unidirectional mapping of states and a cycle of forward mapping and inverse mapping between actions.
Learning a translation mapping is often much more challenging than simply ﬁnding a common latent space.
A common latent space is sufﬁcient for the transfer of a conﬁdence predictor across different dynamics.
To learn a common latent space, domain randomization methods align the different dynamics by randomizing the dynamics with hand-crafted augmentations [26], [27], [28], but they require the differences between the two dynamics to be covered by the augmentations.
Gupta et al.
[29] require pairs of states from different dynamics to learn the latent space.
The assumptions of paired data or covering augmentations are often restrictive in practice.
We remove these assumptions and only assume the existence of correspondence and learn the invariant representation space using multi-length partial trajectory matching.
III. P ROBLEM S TATEMENT
Let the source and target environments be represented by two deterministic Markov decision processes (MDP),
M src and M tar , where M (cid:15) = hS (cid:15) ; A (cid:15) ; p (cid:15) ; R (cid:15) ; (cid:26) (cid:15) 0 ; (cid:13) (cid:15) i ; (cid:15)f src ; tar g .
(cid:15) =
src indicates the source domain, while (cid:15) = tar indicates the target domain.
S (cid:15) is the state space, A (cid:15) is the action space, p (cid:15) : S (cid:15) (cid:2)
A (cid:15) (cid:2) ! S (cid:15) is the transition function, (cid:26) (cid:15) 0 is the initial state distribution, R (cid:15) : S (cid:15) (cid:2) A (cid:15) !
R is the reward function, and (cid:13) (cid:15) is the discount factor.
A policy for M tar is (cid:25) tar : S tar (cid:2) A tar ! [0 ; 1] deﬁning a probability distribution over the target action space in a given state.
To evaluate a policy (cid:25) tar , we use the expected return, which is deﬁned as (cid:17) (cid:25) tar =
{+E s 0 (cid:24) (cid:26) tar 0 ;(cid:25) tar [ P 1 t =
+}
( (cid:13) tar ) t R tar ( s t ; a t ; s t +1 )] , where t indicates the time step.
A trajectory (cid:24) (cid:15) is drawn from either the source or target environments, and is a sequence of state-action pairs
[Equation]
Problem Deﬁnition.
In our problem setting, we are given a set of imperfect target demonstrations (cid:4) tar = f (cid:24) tar 1 ; (cid:24) tar 2 ; : : : g in the target environment M tar , and a set of imperfect source demonstrations (cid:4) src = f (cid:24) src 1 ; (cid:24) src 2 ; : : : g along with a measure of conﬁdence c src in the source environment M src , where the conﬁdence function c src : S src (cid:2)
A src ! R evaluates how useful the input state-action pairs are and if they are available or easy to access.
Our goal is to ﬁnd a well-performing policy (cid:25) tar for the target environment.
Note that we go beyond standard imitation learning by allowing these demonstrations in both environments to range from useless or even harmful demonstrations to nearly optimal or optimal ones.
At the high level, our approach is to learn a conﬁdence function c tar : S tar (cid:2)A tar !
R , which assigns a value to how conﬁdent we are in the optimality of the action a tar t in a particular state s tar t .
We can then reweight the target state-action pairs and conduct standard imitation learning on the reweighted stateaction pairs to learn a policy.
To learn c tar , we would like to learn a conﬁdence predictor f tar for the target environment by leveraging knowledge of the conﬁdence function c src from the source environment.
Assumptions.
To transfer the conﬁdence predictor from the source to the target environment, we assume that the source and target environments are correspondent , which means that there exists a relational mapping between the stateaction pairs of the source agent and the target agent [10].
Speciﬁcally, there exists a relation on states: (cid:8) : S src (cid:2) S tar .
There also exists a mapping from source state-action pairs to the target action space, H 1 : S src (cid:2)
A src ! A tar , and another mapping from target state-action pairs to the source action space, H 2 : S tar (cid:2) A tar ! A src .
The mappings (cid:30) ,
H 1 , and H 2 satisfy the following: if ( s src ; s tar ) 2 (cid:8) , then 8 a src 2 A src ; ( s srcnext ; s tar H 1 ) 2 (cid:8) and 8 a tar 2 A tar ; ( s tarnext ; s src H 2 )(cid:8) .
Here, s srcnext is the successor state from s src when taking the action a src , and s tar H 1 is the successor state from state s tar when taking the action H 1 ( s src ; a src ) , and s tarnext and s src Hare deﬁned similarly.
The assumption builds a connection between the source and target agents.
Without this assumption, the two environments may have no relations to each other, and there will not be any reason for knowledge transfer across them.
This assumption is often satisﬁed in many real-world applications.
For instance, we are interested in two particular settings: (1) The source environment being a simulation of a real robot that would deﬁne the target environment.
For example, the source environment can be a simulated robotic arm picking an apple from a bowl in simulation, while the target environment can be a real robotic arm picking an apple from a bowl.
The source and target environments are different robots performing the same task in a similar context.
For example, the source environment can be a simpler 3-DoF robot arm reaching the center of a bowl without colliding with obstacles, while the target robot can be a 7-DoF robot performing the same task.
We note that the correspondence assumption is less restrictive compared to assumptions in related prior work. For instance, correspondence is implied, when assuming the same dynamics as in [6], [8].
IV. A DVERSARIAL C ONFIDENCE T RANSFER
We aim to leverage the conﬁdence annotation in the source environment and transfer the conﬁdence knowledge to the target environment.
Our key insight is to map the source and target state-action pairs to a common latent space Z . We enforce the latent features for correspondent state-action pairs in the source and target environments, z src and z tar , to be the same.
Here, ( s src ; a src ) and ( s tar ; a tar ) satisfy ( s src ; s tar ) 2 (cid:8) and ( s srcnext ; s tarnext ) 2 (cid:8) .
We then build a shared decoder to predict the conﬁdence from the latent features.
Since the source and target environments have different state-action spaces, we learn two different encoders E src : S src (cid:2)
A src ! Z and E tar : S tar (cid:2) A tar ! Z to map the source and target state-action pairs into a common latent space Z .
We then learn a shared decoder F : Z ! R to predict the conﬁdence from latent features with conﬁdencelabeled demonstrations in the source environment.
The target conﬁdence predictor can be computed by F (cid:14) E tar .
Now, the challenge is how to learn the encoders to ensure that the common latent space satisﬁes the above requirements.
A. Multi-length Partial Trajectory Matching
To map correspondent state-action pairs to a similar latent feature, we develop a distribution matching objective to align the latent distribution of source and target state-action pairs.
We can directly align the distribution of state-action latent features, i.e., matching E src ( s src t ; a src t ) and E tar ( s tar t ; a tar t ) .
However, this does not capture the temporal dependency between state-action pairs over a trajectory.
To address this issue, we incorporate a prior that captures this temporal dependency when learning the encoders.
Imagine a setting where a simulated robot and a real robot are both making a burger.
The state-action pairs corresponding to placing the ingredients (e.g., the patty, lettuce, and tomatoes) are always after placing the bottom bun.
If we only match the stateaction pairs one at a time between the simulated and the real robot, placing the bottom bun may match placing any of the ingredients or even the top bun, but if we also match the consecutive steps of the state-action pairs, placing the bottom bun in both environments will more likely capture the information that the bottom bun comes before the rest of the ingredients.
So, as shown in Fig.
2 (right), we propose a multi-length partial trajectory alignment that emphasizes learning the temporal relationship between state-action pairs.
Speciﬁcally, we match the latent feature distribution of length k , ( k = 1 ; 2 ; : : : ; K ) partial trajectories, which preserves the temporal relationship between consecutive states and actions, and make the latent features of correspondent state-action pairs more likely to be aligned.
B. Feature-Level and Conﬁdence-Level Matching
We develop our learning algorithm that aligns the latent distribution of state-action pairs.
We ﬁrst train the source encoder E src and the shared decoder F with source conﬁdence-labeled state-action pairs using a regression loss:
[Equation]
After learning E src and F , we ﬁx the parameters of E src , and
F and use the latent feature space of E src as the common feature space.
We then only need to make the target encoder
E tar encode the target state-action pairs into this shared feature space and match the source latent feature distribution.
We learn the latent feature distribution alignment using a generative adversarial network.
Since we only want the partial trajectories of the same length to be matched, for each length of partial trajectories k , we adopt a discriminator D k and a loss L k fea to match the latent distribution of length- k
Multi-length Partial Trajectory Matching Train ! !"
# & ( in Source Env
Adversarial Transfer ($ $%& ,& '%& ) ($ $%& ,& '%& ) ($ ()% ,& ()% ) ($ ()% ,& ()% )
E v a l u a t i o n ( s r c ) ( !"
Conf
E v a l u a t
i
o n ( t a r ) ( $%" *
Conf
D’ ( $%" ( !"
# fixed
Source Env
Target Env /
Conf Score Inference ($ ('%& ,& ('%& ) ($ (()% ,& (()% ) D( $ % " ( ! "
# ($ (-.'%& ,& (-.'%& ) ( ! "
# ($ (-.()% ,& (-.()% ) ( $ % " Dfixed
D’
D’* * * * ( $%" * D * Speciﬁcally, the discriminator D k aims to discriminate the latent features of source and target length- k partial trajectories, while the target encoder E tar is trained to confuse the discriminator.
The loss for the discriminator D k can be derived as a binary classiﬁcation loss:
[Equation]
We concatenate the encoded features of multiple consecutive state-action pairs as the latent feature for the length- k partial trajectories.
The discriminator D k is trained to minimize the discriminator loss L k fea and the target encoder E tar is trained to maximize the discriminator loss.
However, the above losses only match the latent feature distributions of state-action pairs, which is conﬁdenceagnostic.
So there may still exist mismatches of state-action pairs with different conﬁdence values.
As shown in Fig.(right), to address the issue, we introduce a conﬁdence-level distribution matching to match the predicted conﬁdence of source and target state-action pairs, which further guides the latent features to be matched in a conﬁdence-aware way.
Speciﬁcally, we use another discriminator D 0 k for match the length- k conﬁdence sequence of length- k partial trajectories using a similar binary classiﬁcation loss:
[Equation]
Similarly, D 0 k is trained to minimize the discriminator loss
L k con and E tar is trained to maximize the discriminator loss.
[Equation]
[Equation]
To optimize the whole network, we ﬁrst train the source encoder and the shared decoder with the source conﬁdencelabeled data as shown in
Eqn.
(4) (the ﬁrst stage in Fig.
2 (left)).
We then iteratively train the feature-level and conﬁdence-level discriminators and the target encoder as shown in Eqn.
(5) and Eqn.
(6), where (cid:21) is the trade-off between L fea and L con (the second stage in Fig.
2 (left)).
After convergence, we chain the target encoder E tar and the conﬁdence decoder F to form the target conﬁdence predictor as shown under Conf.
Score Inference in Fig.
We can then predict the conﬁdence of each state-action pair ( s; a ) in the target environment as F ( E tar ( s; a )) .
With the conﬁdence of each state-action pair, when conducting imitation learning on the target demonstrations, we use the conﬁdence to re-weight each state-action pair in the imitation loss as [6].
Then the imitation learning policy can learn more from useful demonstrations with higher conﬁdence.
Relation to Existing Works.
Our method relaxes assumptions in prior works [6], [19] and targets a more realistic problem.
However, we argue that our work is compatible with prior works.
The common latent feature space in our work can be regarded as a new space of state-action pairs for the source and target environments, where the correspondence between state-action pairs is preserved, especially w.r.t.
conﬁdence.
With the common latent feature space, we can easily extend prior works to the setting of different state spaces by conducting the algorithm, that is originally performed on states, on the latent feature.
Our contribution is not only providing a new approach to learning from imperfect demonstrations but also expanding the usage of algorithms from prior work by introducing a new space to operate in.
V. E XPERIMENTAL R
We conduct experiments on two MuJoCo environments, a simulated and a real Franka Panda arm. We release the code here.
We compare our method ( Ours ) with baseline methods ( GAIL [3]), Dynamics Cycle-Consistency ( DCC ) [10], and variants of our method by gradually adding modules: starting from GAIL without conﬁdence weighting, we ﬁrst add featurelevel matching as Ours-Feature .
Then we add the conﬁdencelevel matching as Ours-Conﬁdence .
Adding multi-length partial trajectory matching to Ours-Conﬁdence derives Ours.
Such design can separately demonstrate the efﬁcacy of our each module.
We show the results of using the ground-truth conﬁdence to reweight the target demonstrations ( Oracle ).
We explain how we implement the baselines on our website.
For the Mujoco environments, we evaluate the average return of 100 rollouts.
For both the simulated robot and simto-real environments, we evaluate the average return ofrollouts.
In all the experiments, we compute the results over 10 runs and report the mean and the standard deviation.
We compute the p-values using the student’s t-test.
The details on the process of demonstration generation for all environments are shown on our website.
[Figure]
MuJoCo Environment.
We create 4 different MuJoCo environments (see Fig.
3): 1-joint and 2-joint reacher, 4leg and 5-leg ant.
1-joint reacher and 4-leg ant are the source environments, and the 2-joint reacher and 5-leg ant are the target environments.
The task for the reacher is to reach the red point from its initial conﬁguration.
The task for the ant is to move towards the right horizontally as fast as possible.
We train the optimal policy using the RL algorithm
TRPO [30].
For the Reacher environment, we select the random policy, a partially-trained policy, and the optimal policy to collect demonstrations with mixed conﬁdence.
For the Ant environment, we select the random policy, three partially-trained policies, and the optimal policy to collect demonstrations with mixed conﬁdence.
The expected return for these experiments is shown in
Table I. Ours outperforms GAIL and DCC , and are comparable with the Oracle in both environments.
Also, Ours outperforms Ours-Conﬁdence , which demonstrates the efﬁcacy of multi-step partial trajectory matching.
Ours-Conﬁdence outperforms Ours-Feature and Ours-Feature outperforms
GAIL , which demonstrate the efﬁcacy of conﬁdence-level and feature-level matching respectively.
For 2-joint reacher and 5-leg ant, the highest p-values comparing with baselines (between Ours and DCC ) are 0 : 147 and 0 : 031 (statistically signiﬁcant) respectively. [Table]
[Figure] which is the direction the ant supposes to move.
Figure 4(a) and 4(c) show the learning curves of the algorithms.
We observe that all methods have similar convergence rates.
Figure 4(b) and 4(d) show a rollout generated by each of the policies.
The rollout from the oracle policy reaches the goal in the most effective manner following the shortest trajectory length.
Ours is the next closest to the Oracle , while the GAIL baseline fails at reaching the goal position at times.
[Figure]
[Table]
the simulated robot experiments.
Simulated Robot.
As shown in Fig.
5, we create a task to move the end-effector of a 5 DoF Panda Franka Robot arm toward the center of a plate without colliding with the boundaries.
The reward function consists of the negative L distance to the center; collisions with the wall or failing to reach the table within the time limit; and positive reward when the end-effector reaches the plate.
The source robot arm has 7 DoF and the target has 5 DoF, where the joints marked by red in Fig.
5 are disabled.
We show different kinds of trajectories: (green–optimal trajectory reaching the center of the plate), (blue–suboptimal trajectories reaching the plate but not the center), (red–failure cases colliding with the wall), (violet–another type of failure case failing to reach the plate within the time limit).
We create two settings: (1) (OT) consists of the optimal (green) and out-oftime (violet) trajectories; and (2) (OSC) consists of optimal (green), suboptimal (blue), and collision (red) trajectories.
In the OT setting, we use the joint and end-effector position as the state, which serves as an easy setup since the end-effector position is shared between the 7-DoF and the 5-DoF. In the
OSC setting, we use the joint position, velocity and torque as the state, which is difﬁcult to align.
For both settings, we use the 3D end-effector position difference as the action space.
Expected return and the success rate are shown in Table II.
Ours outperforms GAIL and DCC with a large margin, which indicates that the learned target conﬁdence predictor can assign higher conﬁdence to useful demonstrations than
DCC and GAIL .
The order of the performance from high to low is Ours , Ours-Conﬁdence , Ours-Feature and GAIL , which demonstrates the efﬁcacy of our multi-length partial trajectory matching, conﬁdence-level matching and featurelevel matching respectively.
The highest p-values comparing with baselines (between Ours and DCC ) are 5 : 64 (cid:2) 10 (cid:0)for the success rate and 1 : 25 (cid:2) 10 (cid:0) 13 for the expected return in the OR setting, and 0 : 006 for the success rate and 0 :for the expected return in the OSC setting, demonstrating statistical signiﬁcance.
[Figure] t e d R e t u r n
[Figure]
Sim-to-Real Environment.
In the sim-to-real environment, we use a simulated and a real Franka Panda Arm as the source and the target respectively (both with 7-DoF).
The task is to move the cube to the upper layer of the shelf (see
Fig. 6).
On the shelf, there is a large stack of books on the right and a small stack on the left.
The middle area is empty.
We assign positive rewards to success of placing the cube on the shelf, with rewards 200 , 300 , and 100 for placing on the left, the middle and the right respectively.
We penalize time by giving a (cid:0) 1 reward for each time step.
The average number of steps for all demonstrations is about 300 .
If the arm fails to put the cube within 1000 steps, it receives reward 0 .
We use the joint position, velocity and the end effector position as the state space and joint forces as the action space.
Fig. 6(c) and 6(d) show Ours outperforms baselines DCC and GAIL .
The performance from high to low is: Ours , OursConﬁdence , Ours-Feature , and GAIL , which demonstrates that multi-length partial trajectory matching, feature-level and conﬁdence-level matching are all necessary to learn a common latent space and an accurate target conﬁdence predictor for the real robot.
The p-value between Ours and the highest baseline, DCC , is 0 : 0004 for the expected return and 0 :for the success rate, demonstrating statistical signiﬁcance.
Varying Composition of Demonstrations.
We conduct experiments by varying the composition of source demonstrations but ﬁx the target demonstration set in the two Mujoco environments to test the robustness of conﬁdence predictor under different demonstration setup.
We do not change the target demonstrations since that will also inﬂuence the imitation learning performance and we cannot test the efﬁcacy of the conﬁdence predictor.
On the column ‘Demonstration’, we show the number of demonstrations that are collected from different policies from random to optimal in Reacher (3 policies) and Ant (5 policies).
As shown in Table III, the ﬁrst row shows the demonstration setting of the experiments mentioned in Table I.
From top to bottom, the optimality of source and target demonstrations deviates more and more.
We observe that the performance drops with larger deviation but does not drop too much even on the last row.
This demonstrates that the proposed approach can work stably even with conﬁdence distribution shift between demonstrations.
[Table]
VI. C ONCLUSION
Summary.
We propose an algorithm to learn from imperfect demonstrations, where the demonstrations can be suboptimal or even fail.
We learn a conﬁdence predictor by leveraging conﬁdence labels and demonstrations in a different but correspondent environment.
We show the policy learned by our method outperforms other baselines in various environments.
Limitations and Future Work.
We require the source and target environments are correspondent, which restricts the applications of our method.
In the future, we plan to relax this assumption by deﬁning some correspondence between conﬁdence instead of the strict deﬁnition on dynamics.