Sample-efficient Adversarial Imitation Learning
Abstract
Imitation learning, wherein learning is performed by demonstration, has been studied and advanced for sequential decision-making tasks in which a reward function is not predefined.
However, imitation learning methods still require numerous expert demonstration samples to successfully imitate an expert’s behavior.
To improve sample efficiency, we utilize self-supervised representation learning, which can generate vast training signals from the given data.
In this study, we propose a self-supervised representation-based adversarial imitation learning method to learn state and action representations that are robust to diverse distortions and temporally predictive, on non-image control tasks.
Particularly, in comparison with existing self-supervised learning methods for tabular data, we propose a different corruption method for state and action representations robust to diverse distortions.
The proposed method shows a 39 % relative improvement over the existing adversarial imitation learning methods on MuJoCo in a setting limited to 100 expert state-action pairs.
Moreover, we conduct comprehensive ablations and additional experiments using demonstrations with varying optimality to provide the intuitions of a range of factors.
Introduction
Imitation learning (IL) is widely used in sequential decision-making tasks, where the design of a reward function is complicated or uncertain.
When a reward is sparse [39] or an optimal reward function is unknown, IL finds an optimal policy that relies only on expert demonstrations.
Owing to recent development in deep neural networks, the range of behaviors, which can be imitated, has expanded.
There are two main learning approaches for IL.
The first approach trains a policy by following actions from an expert in a supervised manner called behavioral cloning (BC) [36, 44].
However, error accumulation limits BC because it greedily imitates the demonstrated actions.
The second approach is inverse reinforcement learning (IRL) [3], inferring a cost function based on given expert demonstrations.
The IRL implements adversarial learning [16] to infer the cost function.
Therefore, an agent learns the policy to imitate expert demonstrations, whereas a discriminator learns to differentiate between the expert’s behavior and that of the agent.
The learned discriminator is used as the cost function in the reinforcement learning (RL) phase.
Although IRL has led an advance in IL, it has key challenges.
First, adversarial learning is known to be delicate in practice.
The min-max computational formulation of adversarial imitation learning (AIL) often involves brittle approximation techniques.
Second, the IL requires many demonstration trajectories to recover an expert policy.
Although IRL requires fewer demonstrations than BC, it still requires considerable trajectories.
Recently, many algorithms or techniques have been proposed to 𝑠 𝑡 ,𝑠 𝑡 ′ 𝑠 𝑡+ 𝜋 𝐸 𝑠 𝐸 𝑎 𝐸 Ƹ𝑧 𝑡+1𝑠 𝑉𝑎𝑙𝑢𝑒𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛 (𝑧 𝐸𝑠 , 𝑧 𝐸𝑎 ) 𝑟
𝐀𝐠𝐞𝐧𝐭 𝑆𝑡𝑎𝑡𝑒 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝐴𝑐𝑡𝑖𝑜𝑛𝐸𝑛𝑐𝑜𝑑𝑒𝑟
𝑃𝑜𝑙𝑖𝑐𝑦 𝑎 𝑡 ,𝑎 𝑡′ 𝐹𝑜𝑟𝑤𝑎𝑟𝑑𝐷𝑦𝑛𝑎𝑚𝑖𝑐𝑠𝑀𝑜𝑑𝑒𝑙 𝐷𝑖𝑠𝑐𝑟𝑖𝑚𝑖𝑛𝑎𝑡𝑜𝑟 𝑧 𝐸 𝑠 𝑧 𝑡𝑠 𝑧 𝑡+1𝑠 𝑧 𝐸𝑎 𝑧 𝑡𝑎 (𝑧 𝑡 𝑠 , 𝑧 𝑡 𝑎 ) 𝐅𝐨𝐫𝐰𝐚𝐫𝐝𝐋𝐨𝐬𝐬 (ℒ 𝐹 ) (𝑧 𝑡 𝑠 , 𝑧 𝑡 𝑠 ′ )
𝐂𝐨𝐫𝐫𝐮𝐩𝐭𝐢𝐨𝐧𝐋𝐨𝐬𝐬
(ℒ 𝑆𝐶 ,ℒ 𝐴𝐶 ) (𝑧 𝑡𝑎 , 𝑧 𝑡𝑎 ′ )
𝒩 expert agent (𝑡) reward agent (𝑡 + 1) address the first challenge [10, 15, 35, 38, 39]; however, little work has been done to improve the sample efficiency of expert demonstrations required [5, 55].
Self-supervised representation learning (SSL) has advanced sample efficiency in the image and language domains [11, 17, 31].
It applies various transformations to the data and uses the transformed data itself as supervision.
Thus, it increases the sample efficiency by obtaining training signals from auxiliary tasks or objectives that do not rely on labels.
Specifically, InfoNCE [11, 21] and asymmetric twin-based [12, 17] SSL approaches are known to be effective for learning robust feature representations for different distortions of identical inputs.
Recently, SSL has been utilized in imagebased RL algorithms [41, 43] and has shown significant improvement in performance.
However, transformation techniques applied to image-based RL are not directly adaptable to non-image control benchmarks. This is because these approaches rely upon the semantic/spatial properties of data that may generate either out-of-distribution examples or examples that supply only the same view when directly applied to a continuous control (tabular) domain.
In this study, we propose a sample-efficient AIL method for non-image control benchmarks.
The proposed method leverages auxiliary training signals for learning state and action representations that are temporally predictive and robust to diverse distortions.
Based on the characteristics of each domain and benchmark, an auxiliary task that can learn informative feature representations is different.
For RL, to address sequential decision-making tasks, the feature representation of a state and action should contain temporally predictive information.
To address this, we add an auxiliary task that predicts the next state representation from the given current state and action representations.
Moreover, learning representations that discard information regarding nuisance variables improves generalization and decreases required sample complexity.
Previous transformation techniques for tabular data [4, 52] generate transformed samples far from real samples.
Therefore, we propose a simple, effective corruption method that generates transformed samples showing diverse distortions that are possible in-distribution.
Empirically, we demonstrate that promoting temporally predictive feature representations with robustness against diverse distortions significantly improves sample efficiency.
Method
The core of RL is an agent and environment.
An agent receives a reward from the environment based on the actions determined by a policy.
RL learns the optimal policy of a Markov decision process.
For IL, the agent learns the optimal policy where a pre-defined reward function does not exist by relying only on given expert demonstrations.
We define the IL scenario from a small number of expert demonstrations.
The expert demonstrations D E are sampled from a state-action density of an expert, ρ O , defined as follows:
[Equation]
where N E is the number of state-action pairs from ρ O .
We assume a scenario in which N E is less than the number of one full trajectory.
We denote a state-action pair by x = ( s, a ) where x ∈ X and
X = S × A .
This study comprises the following six networks, as illustrated in Figure 1.
A policy π θ ( · ) parameterized by θ that generates actions a given states s based on a policy.
A value function V ( · ) that evaluates a current policy π θ .
V is trained with rewards r from an estimated cost function.
A state encoder SE ( · ) that extracts a feature representation of raw states s .
z s =
SE ( s ) ∈
Z s . An action encoder AE ( · ) that extracts a feature representation of actions a .
z a = AE ( a ) ∈
Z a .
A
forward dynamics model F · ) that predicts the feature representation of
the
distorted version of next states
ˆ z st
+
from the feature representations of the current state and action, z st and z at , and
Gaussian noises
N
ˆ z st +1 = F ( z st ⊕ z at ⊕ N ) ∈
ˆ Z t +1 , where ⊕
is
concatenation.
A discriminator D ω ( · )
parameterized by ω that discriminates agent demonstrations from expert demonstrations D E .
The input of D ω is z s ⊕ z a .
D ω is
also called
a cost function.
In
the RL phase,
D ω is the estimated cost function ( r = − log (
D
ω ( z s ⊕ z a )
) ∈ R
As shown in Algorithm 1 in the Supplementary S2, the proposed method comprises three major parts.
In Section 2.1, we explain how to train the cost function D ω using expert demonstrations D E (GAIL in Algorithm 1).
In Section 2.2, we describe how to use SSL in a non-image environment (REPR in Algorithm 1).
We implement trust region policy optimization [40] to train the agent policy π θ by following the use in [22] (TRPO in Algorithm 1).
2.1 Generative Adversarial Imitation Learning
The proposed method is based on generative adversarial imitation learning (GAIL) [22].
GAIL finds an optimal policy by matching an occupancy measure between expert E and the agent.
The optimization equation of GAIL can be derived in the form of the Jensen-Shannon divergence, which is equal to the minimax equation of generative adversarial networks [16].
The minimax optimization of GAIL is expressed as follows:
[Equation]
where D π and D E are the corresponding demonstrations from an agent policy π θ and expert policy π E , respectively.
In GAIL, the raw state and action are input to the discriminator.
For the proposed discriminator, state and action representations embedded by state and action encoders, SE ( · ) and
AE ( · ) , are input.
The discriminator loss is expressed as follows:
[Equation]
where z =
z s ⊕ z a .
Here, z s is a state representation embedded by SE ( s ) , and z a is an action representation embedded by AE ( a ) .
2.2 State and Action Representations 2.2.1 Modeling forward Dynamics
Each domain requires different ways of generating self-supervision depending on the properties of the data.
For example, BERT [14] leverages a training signal by predicting future words from previous words.
For RL, the prediction error of a forward dynamics model has been used as an intrinsic reward [34, 41].
In tabular data, in contrast to data from an image, it is difficult to create a distorted version of an original input without losing semantic information.
Therefore, we posit that maximizing the agreement between the distorted and original ones is more suitable for learning meaningful features than maximizing the agreement between the two distorted versions of input in tabular data.
We propose a method that generates a distorted version of the input to learn or discard the desired features and their corresponding loss function for RL.
The proposed method uses the forward dynamics model to predict the distorted version of the next state representation from the given current state and action representations.
First, the forward dynamics model is mathematically expressed as follows:
[Equation]
where z st = SE ( s t ) , z at = AE ( a t ) at a time step t , and N (0 , 1) is the Gaussian noise.
The output ˆ z st +1 represents a distorted version of the observed future representations z t +1 .
The choice of a transformation controls what the representation learns.
Thus, we apply a distortion by concatenating
Gaussian noise rather than using a corrupted state-action pair because corruption cannot guarantee consistency in information with respect to temporality.
We use a contrastive loss for training.
InfoNCE-based unsupervised contrastive learning (UCL) methods learn a feature representation by maximizing the agreement between differently transformed same input while minimizing that of the rest of the input (negative samples).
The learned representation from UCL is invariant in unnecessary details; however, it contains maximal information by maximizing a lower bound on the mutual information between the two views [37, 49].
We utilize the
InfoNCE loss to obtain as many temporally informative features as possible.
The proposed forward dynamics model is trained to maximize the agreement between the distorted and observed next state representations while minimizing that of the rest of the state representations.
This is expressed as follows:
[Equation]
where ( z st , z st +1 , ˆ z st +1 ) ∼ ( Z st , Z st +1 , ˆ Z st +1 ) and cs( u, v ) =
u ⊤ v/
|| u
|| ||
v ||
( j indexes the state or next state representation in the batch, and BS is the batch size).
Following SimCLR [11], we use the other 2( BS − 1) representations in the batch as negative samples.
2.2.2 Corruption Method
Learning representations that can discard nuisance features is preferable to reduce sample complexity.
To help with this, we propose a corruption method that creates a distorted sample showing diverse views that are possible in-distribution.
The proposed method swaps the input features of each stateaction pair with the input features of the same indices of another state-action pair in a batch.
For a batch of state-action pairs sampled from the current policy, X b , we generate a corrupted version x ′ i for each state-action pair x i .
The corrupted versions of state s and action a are generated, respectively.
However, for convenience of rationalization, we explain the method based on the state-action pair x .
First, we make a copy of X b as X cb and permute X cb by changing the order of each state-action pair in the batch at random, perm ( X cb ) .
Second, we sample some indices of the state-action pair without replacement, I ∈ { 0 , ..., dim ( s ⊕ a ) − 1 } q .
q is the number of features to corrupt ( =
⌊ c · dim ( s ⊕ a ) ⌋ ), where c is a corruption rate ( c =
c s + c a ).
Third, we duplicate I as a shape of
N ( X b ) × q .
Subsequently, we generate corrupted state-action pairs X ′ of a given batch as follows:
[Equation]
where X ′ b = ⟨ S ′ b × A ′ b ⟩ .
For convenience, we omit subscripts b hereafter.
We refer to this method as the swapping corruption method.
Empirically, we observed superior performance compared to existing methods on non-image control benchmarks.
Numerous discrepancy measures can quantify the similarity between corrupted and original inputs.
For the state, we maximize the similarity between the representation of the distorted and observed versions by minimizing the mean squared error (MSE) as follows:
[Equation]
where ( z s , z s ′ ) ∼ ( Z s , Z s ′ ) and Z s = SE ( S ) and Z s ′ =
SE ( S ′ ) .
For the state representation, temporally predictive features should be embedded well to minimize L F simultaneously.
Thus, we observed the best performance with MSE compared to other indirect discrepancy measures.
For the action representation, because the gradients from the L F are not sufficient to hinder collapse, we use the Barlow twins loss [53] that is robust against the constant embedding problem.
The Barlow twins loss draws the cross-correlation matrix close to the identity matrix.
This is expressed as follows:
[Equation]
where Z a = AE ( A ) and Z a ′ =
AE ( A ′ ) .
i, j index the vector dimensions of the action representations.
Therefore, the Barlow twins loss prevents collapse by maximizing the similarity between the representation of the distorted and observed versions of action and reducing entanglement between the components of the representations.
Ablations that can give more intuitions about losses are given in Section 5.
Consequently, the total loss for SSL is computed as follows.
[Equation]
where λ F , λ S , and λ
A are hyperparameters for each loss.
Related Work 3.1 Data-efficient Reinforcement Learning
In deep RL, studies have been conducted to improve sample efficiency.
For continuous control, several studies have suggested the use of reconstruction loss [18, 27].
However, most of the suggested methods are RL benchmarks, which have a sparse reward or image state.
Methods using a selfsupervised error as an intrinsic reward have been proposed to improve the sample efficiency in a sparse reward scenario [32, 34, 42].
For the image state, various image augmentation techniques and self-supervised objectives have been applied to reduce environmental interactions [19, 20, 25, 26, 30, 41, 43, 51].
To the best of our knowledge, this is the first IL method using SSL to improve the sample efficiency of expert demonstrations.
3.2 Self-supervised Representation Learning
Currently, SSL is divided into three approaches.
The first is a pretext task [33], which creates a pre-task that can learn useful feature representations and use the learned representations on downstream tasks.
The second is UCL.
Contrastive learning works on a simple push-pull principle, and it can be a sample or cluster level [9].
The contrastive loss contrasts the neighboring instances with non-neighboring ones [11, 21].
The third is asymmetric twin-based SSL.
Unlike UCL, these methods do not use negative samples during training.
Asymmetric twin-based SSL methods learn robust representations in such a way that differently transformed versions of input have the same representation.
As representative methods, BYOL [17] and Simsiam [12] used Siamese networks with weight sharing and stop-gradient techniques to avoid collapse.
Barlow twins [53] utilized a correlation matrix between the representations of a differently transformed same input to maximize the similarity between them while minimizing redundancy.
VICReg [6] is effective for making the two representations similar and reducing the embedding of non-informative factors.
There are two SSL methods for tabular data, VIME [52] and SCARF [4], which can be directly applied to non-image control benchmarks.
VIME uses a random corruption method and SCARF suggests a method that replaces each feature dimension by a random draw from that feature dimension’s empirical marginal distribution.
The difference between our work is the corrupted data of swapping is a mixture of only two state-action pairs because the replaced features are sampled from another single state-action pair.
Meanwhile, the corrupted data of SCARF is a mixture of varying state-action pairs, resulting in possible out-of-distribution data.
In the experiments, the proposed swapping corruption method showed higher performance compared to the other two methods.
We measured the variance and outlier scores of the corrupted samples produced by VIME, SCARF, and the proposed method, to confirm that the proposed method qualitatively generates more realistic and still varied samples.
3.3 Inverse Reinforcement Learning
Although IRL [1, 56] has made significant advances in IL, it encounters some challenges.
First, there is an unstable training issue for adversarial learning; improved algorithms have been proposed to [Table] [Table] overcome this problem.
GAIL [22] is the first study drawing an analogy between IL and generative adversarial networks [16].
AIRL [15] proposed an AIL method that is robust to changes in dynamics.
VAIL [35] improves the stability problem by constraining the information flow in the discriminator.
EAIRL [38] reduces the overfitting problem using empowerment (the information gain on action entropies).
These algorithms have been suggested to improve the stability and scalability of AIL.
Second, studies on the sample efficiency of expert demonstrations have not been sufficiently conducted.
SAILfO [47] covers the necessity of studying the sample efficiency of expert demonstrations, and proposes a simple model-based algorithm.
Recently, f-GAIL [55] showed that finding an appropriate discrepancy measure during training is better than using a predetermined measure to improve sample efficiency.
ASAF [5] is an algorithm in which training the discriminator could perform the role of policy and showed that it helps improve sample efficiency.
However, these methods require at least five full trajectories to recover the expert policy on continuous control benchmarks such as the MuJoCo physics engine.
Unlike those methods, the proposed method successfully imitates the expert’s behavior with less than one full trajectory.
In practice, it is difficult to collect perfectly optimal demonstrations because demonstrations are commonly collected by crowdsourcing [23] or multiple experts [13].
The collected data from external sources are normally imperfect—a mixture of optimal and non-optimal demonstrations.
To address these problems, algorithms for IL from imperfect demonstrations have been proposed [50, 54].
We demonstrate that combining the proposed method with other algorithms for IL from imperfect demonstrations further improves them, thus, verifying the scalability of the proposed method.
Experiments
We assessed the performance of the proposed method on five continuous control benchmarks simulated by MuJoCo [46] (Ant-v2, HalfCheetah-v2, Hopper-v2, Swimmer-v2, and Walker2d-v2) in four distinct settings: using expert demonstrations of less than one full trajectory with the optimality of 25 % , 50 % , 75 % , or 100 % .
We tested the sample efficiency of the proposed method in a scenario where optimal demonstration samples of less than one full trajectory are available ( ≤ 100 ).
Expert demonstrations with optimalities of 25 % , 50 % , and 75 % represent imperfect demonstrations - a mixture of optimal and non-optimal demonstrations.
Imperfect demonstrations D I are sampled from a noisy state-action density ρ , expressed as follows:
D I = { ( s n , a n ) } N I n =1 i.i.d.
∼ ρ ( s, a ) , where N I is the number of state-action pairs from ρ .
The noisy state-action density ρ can be expressed as follows:
[Equation]
where ρ O is the state-action density of an expert, ρ i is the state-action density of a single nonexpert, and n is the number of non-experts.
Furthermore, ψ , satisfying 0 < ψ < 1 , is an unknown mixing coefficient of the optimal and non-optimal state-action densities, and ψ + (cid:80) ni
=1 v i = 1 ;
Optimality of 100 %
First, we evaluated the proposed method with a small number of perfect expert demonstrations.
We compared the method with seven existing IL methods: BC, GAIL, AIRL,
VAIL, EAIRL, SQIL, and ASAF.
Table 1 shows that the proposed method outperforms other IL methods on all three benchmarks.
Particularly, the proposed method succeeded in perfectly imitating the expert policy on HalfCheetah.
Notably, GAIL and VAIL show higher performance than the recently proposed ASAF in 100 state-action pairs.
For f-GAIL, we conducted experiments using the official GitHub; however, it failed to converge in less than one full trajectory.
We surmise that this is because a reasonable number of expert state-action pairs must be guaranteed to automatically learn an appropriate discrepancy measure for the given pairs.
We also tested our method with varying expert data sizes.
As provided in Table S5 in the Supplementary S4, there is a relatively small decrease in the performance up to N E = 20 on all three benchmarks.
However, when N E = 10, the performance is decreased by a large margin.
To make the experimental results stronger, we tested the reliability of the reported average using IQM [2].
We obtained IQM 4555.8 on Ant, IQM 5420.3 on HalfCheetah, and IQM 3527.9 on Walker2d, which are very close to the reported average.
In addition, we tested the proposed method (without L AC ) on two discrete control benchmarks of
OpenAI Gym [8] (BeamRider-ram-v0, and SpaceInvaders-ram-v0).
We observed that the cumulative rewards of the proposed method are superior to those of the GAIL.
For the results on the two discrete control benchmarks, please refer to the Supplementary S8.
Moreover, the average cumulative rewards of the expert policy that we obtained can be found in the Supplementary S1.1.
Corruption method
We verified the performance of the swapping corruption method by comparing it with the existing two corruption methods and additionally, a mean corruption method, which replaces the features with the empirical marginal distribution’s mean, in cumulative rewards, variance, and local outlier score.
Table 2 shows that the proposed method shows a higher performance compared with the three corruption methods.
We observed that the proposed method generated transformed samples that provide more diverse views compared with random and mean methods and comparably diverse views compared with the method, obtaining each replaced feature from varying state-action pairs (Each dim).
We measured its diversity on the corrupted states using variance σ 2 : 1 1000 (cid:80) 1000 i =1 (cid:16)(cid:80) dim ( s )
j = ( s ′ i,j − ¯ s ′
i,j ) 2 (cid:17) where ¯ s ′ denotes the average of { s ′ i } 1000 i =1 .
Also, one potential concern about using corruption as a transformation technique is that the corrupted samples are out-of-distribution, resulting in performance degradation.
To evaluate this, we computed the local outlier factor [7] of the corrupted states.
We computed the percentage of corrupted samples that are local outliers with respect to the observed states.
Table 2 shows that for the random method, the most corrupted states are predicted as outliers, despite the low variance.
For the mean method, the corrupted states are mostly realistic; however, the diversity is lower compared with the swapping method, which limits the effectiveness of data augmentation.
Replacing each feature with a feature from a different combination has diversity, but it also creates more out-of-distribution data.
The proposed swapping method showed high variance and a relatively low percentage of local outliers.
As a result, we empirically confirmed that for control tasks, it is imperative to create meaningful in-distribution data in the corruption process.
Optimality of 25 % , 50 % , or 75 %
We tested the performance of the proposed method in a more practical scenario with imperfect demonstrations.
In this environment, we combined the proposed method with 2IWIL [50].
2IWIL showed stable training compared to other IL algorithms for imperfect demonstrations because they predict the confidence of given demonstrations in a pre-stage. Please
[Table] [Table] refer to the Supplementary S1.2 for detailed explanations about 2IWIL, other comparison methods, and the collected imperfect demonstrations.
The pseudo-code of the combined algorithm can be found in the Supplementary S3.
Figure 2 shows the cumulative rewards on the five continuous control MuJoCo benchmarks with different optimality rates.
The proposed method combined with 2IWIL outperforms the other six comparisons by a large margin in all optimality rates.
Particularly, the relative improvement over 2IWIL and CAIL [54] on Ant is 288 % and 208 % on average, respectively.
However, we observed a decrease in the cumulative rewards when the noise rate was 0.75 on Ant and Walker2d benchmarks.
We surmise that this was caused by a deficiency in the number of locomotion movements from the optimal policy.
The degree of improvement in performance is dependent on a number of given optimal demonstrations to some extent because SSL methods create an auxiliary training signal by leveraging the given data.
Manifold mixup
We applied a widely-used sample efficiency technique, manifold mixup (MM) [48], to the combined method.
In previous studies [28, 29], it is shown that MM in the feature space enriched by SSL is further effective to improve performance.
Through this experiment, we would like to compare the efficiency of 1) MM, 2) the proposed method, and 3) using both as a sample efficiency technique.
MM increases the diversity of expert demonstrations by interpolating the feature space output of the input data pair.
We performed MM on the feature space as follows: (¯ z, ¯ y ) =
( Mix λ ( z i , z j ) , Mix λ ( y i , y j )) , where Mix λ ( a, b ) = λ · a + (1 − λ ) · b .
Here, ( z i , z j ) are the feature representations of ( x i , x j ), and ( y i , y j ) are the estimated confidence by 2IWIL.
Table 3 shows that, when only MM was used, the performance on Walker2d improves; however, the performance on
Ant and HalfCheetah does not show a reasonable improvement.
This indicates that it is difficult to naturally learn representations that are temporally predictive and robust to diverse distortions from training signals generated by synthetic data from MM.
Conversely, when only the proposed method was used, we observed a relatively small increase on Walker2d due to a deficiency in the number of optimal locomotion movements.
Consequently, as shown in Figure 2 and Table 3, we observed near-optimal performance on all the benchmarks with varying optimalities when both ours and MM were used.
Building Intuitions with Ablations
We conducted ablations on the proposed factors to provide an intuition of each role.
Importance of both L F and ( L SC , L AC )
Table 4 shows that L F and ( L SC , L AC ) are complementary to each other.
When only L F is added to GAIL, we observed an increase on HalfCheetah and Walker2d, not on Ant, and the gap in increase is small.
Tian et al.
[45] demonstrated that it is not always good to learn maximal information using contrastive learning.
Rather, it is important to minimize nuisance information as much as possible using a strong transformation to maximize task-specific information.
However, a strong transformation without damaging semantic information is impossible in continuous control data.
Thus, L F mainly plays the role of maximally learning temporally predictive information with a weak, reasonable transformation, and ( L SC , L AC ) helps to suppress the nuisance factors using the corruption method so as to maximize task-relevant features in the proposed method.
Loss function of L F
For the discriminator, D ω , temporally predictive features are important information to distinguish the imitator (agent) from the expert.
However, learning maximal information can lead to learning nuisance information as well.
Transformation techniques should be used to suppress this.
For L F , we tested the concatenation of Gaussian noise and the corrupted state as transformation techniques to generate the distorted version.
The concatenation of Gaussian noise yields better results.
[Table]
Qualitatively, there was a 7.5 % relative improvement when using the concatenation of Gaussian noise.
We surmise that this is because corruption can unavoidably change important input features with respect to temporality.
In addition, to demonstrate the effectiveness of Gaussian noise, we conducted an experiment of not appending noise dimensions and observed a decrease on benchmarks, as provided in the Supplementary S5.
For the loss function of L F , in addition to the InfoNCE loss of SimCLR, we tested MSE, and MSE with the stop-gradient of BYOL, and the Barlow twins loss.
Table 5 presents that SimCLR’s performance is superior to other methods by a significant margin.
Loss function of L SC and L AC
We tested various SSL loss functions for both L SC and L AC .
Notably, for L SC , the MSE loss that is exposed to the collapsing problem shows the highest performance on average.
This is because L F cannot be minimized if the state representation is only the same constant.
For L AC , the Barlow twins and SimSiam losses showed the first- and second-best performance on average, respectively.
Detailed analysis and intuitions of SSL loss functions for a state and action are provided in the Supplementary S6.
Sensitivity to corruption rate
Recently, studies on the importance of the degree of transformation have been proposed in SSL.
Jing et al.
[24] showed that a strong transformation can cause dimensional collapse.
To find an appropriate corruption rate, we studied the sensitivity to the corruption rate of state and action and reported the results in Table S8 in the Supplementary S7.
Conclusion
The sample efficiency of expert demonstrations is desirable in IL because obtaining a large number of expert demonstrations is often costly.
Motivated by successes in SSL, we propose a sample-efficient
IL method that promotes learning feature representations that are temporally predictive and robust against diverse distortions for continuous control.
We evaluated our proposed method in various control tasks with limited expert demonstration settings and showed superior performance compared to existing methods.
In spite of the excellence with limited settings the proposed method has some limitations.
There is an increase in model complexity and additional computational cost during training since three additional networks and SSL losses are added.
Acknowledgments
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by Defense Acquisition Program Administration (DAPA) and Agency for Defense Development (ADD) (UD190031RD) and was supported in part by a research grant from Samsung SDI, Co.,
Ltd.